Cannot copy the versioned files again after rclone copy with `--s3-versions`

Suspected Bug
1

What is the problem you are having with rclone?

I executed rclone copy --s3-version
and then I tried to execute rclone copy again or rclone ls with --s3-versions
but I can't copy or see the versioned file.

e.g.
A object storage has
1.txt (current)
1.txt (version create date = 1/2/2023)
1.txt (version create date = 1/2/2023)

and then those files move to B object storage by rclone copy --s3-versions
B object storage has
1.txt
1-v2023-01-02-0000.txt
1-v2023-01-01-0000.txt

when I try to access 1-v2023-01-02-0000.txt with rclone ls --s3-versions,
but there is nothing to show us.
I can't copy 1-v2023-01-02-0000.txt file to other storage as well.

Run the command 'rclone version' and share the full output of the command.

1.63.1

Which cloud storage system are you using? (eg Google Drive)

custom s3 object storage

Logs

after the versioned files are copied to object storage

rclone ls dev://test2/
      51 4-v2023-07-07-082248-000.txt
      51 4-v2023-07-19-074649-000.txt
      51 4.txt

I copy a versioned file with versions. it's not working

rclone -vv copy dev,versions://test2/4-v2023-07-19-074649-000.txt ./
2023/07/25 19:32:37 INFO  : There was nothing to transfer
2023/07/25 19:32:37 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.0s

I copy a versioned file without versions. it's working

rclone -vv copy dev://test2/4-v2023-07-19-074649-000.txt ./
2023/07/25 19:32:02 INFO  : 4-v2023-07-19-074649-000.txt: Copied (new)
2023/07/25 19:32:02 INFO  :
Transferred:             51 B / 51 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.1s
2

Because these are not versions but 3 different files.

3

I know that those are not versions.
I want to copy as the different files to other storage with versions option like below

rclone -vv copy dev,versions://test2/4.txt ./
2023/07/25 19:35:10 INFO  : 4.txt: Copied (new)
2023/07/25 19:35:10 INFO  :
Transferred:             51 B / 51 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.1s

but 4-v2023-07-07-082248-000.txt and 4-v2023-07-19-074649-000.txt are not working.

4

I think it is misunderstanding.

Rclone can not create versions. Versioning is bucket feature - when ON versions are created every time file changes.

--s3-versions or --s3-versions-at flags allow you to see and access existing versions. They do not allow you to create versions.

You had versions in your storage A

Then using rclone copy --s3-versions you told rclone to show all versions as objects and copy to storage B. Storage B does not have any versions.

5

If you want to replicate versions on your storage B you have to run multiple rclone sync A B --s3-versions-at

6

I know Rclone can not create versions.
Storage A is ON versions in upper case.
Storage B is OFF versions.
After the files of storage A copy to storage B,
the files of storage B copy to the storage C.

I want to copy all files in storage B to storage C.
rclone copy storage_b,versions://test2/ storage_c:/test2/ as the common command whether it's versions or not

7

so what you think this command:

achieves? nothing:)

you want to copy objects from B to C run:

rclone copy B: C:

As per docs:

--s3-versions

Include old versions in directory listings.

but you have no versions

8

I think below case is similiar. but it's not working even my Rclone version is higher.

9

What does

rclone ls dev:test2 --s3-versions

show?

10

Yes it shows all files.

rclone ls dev://test2/ --s3-versions
      51 4-v2023-07-07-082248-000.txt
      51 4-v2023-07-19-074649-000.txt
      51 4.txt
11

Could you post all -vv output?

12

This is the log.

rclone -vv copy dev,versions://test2/4-v2023-07-19-074649-000.txt ./
2023/07/25 19:32:37 DEBUG : rclone: Version "v1.63.1" starting with parameters ["rclone" "-vv" "copy" "dev,versions://test2/4-v2023-07-19-074649-000.txt" "./"]
2023/07/25 19:32:37 DEBUG : Creating backend with remote "dev,versions://test2/4-v2023-07-19-074649-000.txt"
2023/07/25 19:32:37 DEBUG : Using config file from "/home/.config/rclone/rclone.conf"
2023/07/25 19:32:37 DEBUG : dev: detected overridden config - adding "{pO73u}" suffix to name
2023/07/25 19:32:37 DEBUG : name = "dev{pO73u}", root = "//test2/4-v2023-07-19-074649-000.txt", opt = &s3.Options{Provider:"Other", EnvAuth:false, AccessKeyID:"1111", SecretAccessKey:"2222", Region:"", Endpoint:"http://192.168.1.1:80", STSEndpoint:"", LocationConstraint:"", ACL:"", BucketACL:"", RequesterPays:false, ServerSideEncryption:"", SSEKMSKeyID:"", SSECustomerAlgorithm:"", SSECustomerKey:"", SSECustomerKeyBase64:"", SSECustomerKeyMD5:"", StorageClass:"", UploadCutoff:209715200, CopyCutoff:4999341932, ChunkSize:5242880, MaxUploadParts:10000, DisableChecksum:false, SharedCredentialsFile:"", Profile:"", SessionToken:"", UploadConcurrency:4, ForcePathStyle:true, V2Auth:false, UseAccelerateEndpoint:false, LeavePartsOnError:false, ListChunk:1000, ListVersion:0, ListURLEncode:fs.Tristate{Value:false, Valid:false}, NoCheckBucket:false, NoHead:false, NoHeadObject:false, Enc:0x3000002, MemoryPoolFlushTime:60000000000, MemoryPoolUseMmap:false, DisableHTTP2:false, DownloadURL:"", DirectoryMarkers:false, UseMultipartEtag:fs.Tristate{Value:false, Valid:false}, UsePresignedRequest:false, Versions:true, VersionAt:fs.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}, Decompress:false, MightGzip:fs.Tristate{Value:false, Valid:false}, UseAcceptEncodingGzip:fs.Tristate{Value:false, Valid:false}, NoSystemMetadata:false}
2023/07/25 19:32:37 DEBUG : Resolving service "s3" region "us-east-1"
2023/07/25 19:32:37 DEBUG : fs cache: renaming cache item "dev,versions://test2/4-v2023-07-19-074649-000.txt" to be canonical "dev{pO73u}:test2/4-v2023-07-19-074649-000.txt"
2023/07/25 19:32:37 DEBUG : Creating backend with remote "./"
2023/07/25 19:32:37 DEBUG : fs cache: renaming cache item "./" to be canonical "/home/tools/rclonetest"
2023/07/25 19:32:37 DEBUG : Local file system at /home/tools/rclonetest: Waiting for checks to finish
2023/07/25 19:32:37 DEBUG : Local file system at /home/tools/rclonetest: Waiting for transfers to finish
2023/07/25 19:32:37 INFO  : There was nothing to transfer
2023/07/25 19:32:37 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.0s
13

OK there is some weird behaviour of --s3-versions and it is easily replicable:

I created S3 bucket with the same content like yours (there is no versioning enabled, all are regular files):

$ rclone lsf s3:test-bucket/test/

4-v2023-07-07-082248-000.txt
4-v2023-07-19-074649-000.txt
4.txt

and now trying to copy files:

$ rclone copy s3:test-bucket/test/4.txt . --s3-versions -v
2023/07/25 13:56:49 INFO  : 4.txt: Copied (new)
2023/07/25 13:56:49 INFO  :
Transferred:   	          4 B / 4 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         1.3s

this worked as expected.

$ rclone copy s3:test-bucket/test/4-v2023-07-19-074649-000.txt . --s3-versions -v
2023/07/25 13:58:07 INFO  : There was nothing to transfer
2023/07/25 13:58:07 INFO  :
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.8s

this is when things break

but of course when I copy without --s3-versions all works fine.

rclone copy s3:test-bucket/test/4-v2023-07-19-074649-000.txt . -v
2023/07/25 14:09:13 INFO  : 4-v2023-07-19-074649-000.txt: Copied (new)
2023/07/25 14:09:13 INFO  :
Transferred:   	          2 B / 2 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         1.2s

It looks like rclone is not able to distinguish between real versions and files with names looking like versions (at least in rclone versions naming convention). And it makes it confused:)

@ncw - definitely it is a bug. Very edge case IMO - not even sure it is worth fixing.

1 Like
14

And regarding your problem - I agree that this flag behaviour is not perfectly correct when applying to regular files but simply do not use --s3-versions when you do not even have any versions or when trying to copy regular files - at least not when they follow rclone versions naming convention. This is what breaks it.

1 Like
15

Yes this is a potential problem with versions. Rclone is relying on the file name to work out whether the objects are versions or not. There is no easy way to change this because rclone needs to know from looking at the file name whether it is a version or not otherwise it would have to do more transactions which would slow things down.

So its a compromise... Maybe more docs?

16

Good that it is clear why it happens:) Yes some mention of this in docs would be enough. I will add it later.

17

done

1 Like
18

this is BTW let's say not ideal implementation. Obviously when using versions one can expect more API calls. But it is better to document what it is than users bump at weird behaviour.

1 Like
19

@ncw @kapitainsky
FYI.
When I checked the logic which uses filename, Rclone with versions understands as filename+v+date.ext is the versioned file(e.g. text-v2023-07-07-082248-000.txt).
but in my case, I had the same name pattern files. and then rclone showed the error "the specified key does not or exist" even though I have the file (e.g. text-v2023-07-07-082248-000.txt).
because Rclone try to search the filename.ext (e.g. text.txt )

I think that the file which looks like the versioned file(e.g. text-v2023-07-07-082248-000.txt) can be a file, not the versioned file.
If you want to separate the versioned file, I think you can make the other variable like version_creation_time, not the filename.

20

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.