--s3-versions can't retrieve old version

cberni · March 18, 2023, 9:40pm

What is the problem you are having with rclone?

can't copy file version. (--s3-versions)
I list files but when listing one file alone or try to copy it don't work.

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1

os/version: ubuntu 20.04 (64 bit)
os/kernel: 5.4.0-91-generic (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.19.4
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

aws s3

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)


rclone --s3-versions copy remote:2018/11/27/11/30E6B749/7E5385DA/7E9A6FC4-v2022-02-15-183218-000 remote:2018/11/27/11/30E6B749/7E5385DA/7E9A6FC4

The rclone config contents with secrets removed.

type = s3
provider = AWS
access_key_id = 
secret_access_key = 
storage_class = ONEZONE_IA

A log from the command with the `-vv` flag

2023/03/18 18:28:59 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "--s3-versions" "--config" "/etc/rclone.conf" "copy" "remote:bucket_test/2018/11/27/11/30E6B749/7E5385DA/7E9A6FC4-v2022-02-15-183218-000" "remote:bucket_test/2018/11/27/11/30E6B749/7E5385DA/7E9A6FC4" "-vv"]
2023/03/18 18:28:59 DEBUG : Creating backend with remote "remote:bucket_test/2018/11/27/11/30E6B749/7E5385DA/7E9A6FC4-v2022-02-15-183218-000"
2023/03/18 18:28:59 DEBUG : Using config file from "/etc/rclone.conf"
2023/03/18 18:28:59 DEBUG : remote: detected overridden config - adding "{pO73u}" suffix to name
2023/03/18 18:29:00 DEBUG : fs cache: renaming cache item "remote:bucket_test/2018/11/27/11/30E6B749/7E5385DA/7E9A6FC4-v2022-02-15-183218-000" to be canonical "remote{pO73u}:bucket_test/2018/11/27/11/30E6B749/7E5385DA/7E9A6FC4-v2022-02-15-183218-000"
2023/03/18 18:29:00 DEBUG : Creating backend with remote "remote:bucket_test/2018/11/27/11/30E6B749/7E5385DA/7E9A6FC4"
2023/03/18 18:29:00 DEBUG : remote: detected overridden config - adding "{pO73u}" suffix to name
2023/03/18 18:29:00 DEBUG : fs cache: renaming cache item "bucket_test/2018/11/27/11/30E6B749/7E5385DA/7E9A6FC4" to be canonical "remote{pO73u}:bucket_test/2018/11/27/11/30E6B749/7E5385DA/7E9A6FC4"
2023/03/18 18:29:00 DEBUG : S3 bucket bucket_test path 2018/11/27/11/30E6B749/7E5385DA/7E9A6FC4: Waiting for checks to finish
2023/03/18 18:29:00 DEBUG : S3 bucket bucket_test path 2018/11/27/11/30E6B749/7E5385DA/7E9A6FC4: Waiting for transfers to finish
2023/03/18 18:29:00 INFO  : There was nothing to transfer
2023/03/18 18:29:00 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         1.4s

2023/03/18 18:29:00 DEBUG : 6 go routines active

another test:

rclone ls --s3-versions remote:bucket_test/2018/11/29/17/739AA11A/01E4D5C7/
    39596 01C12570-v2022-02-15-183225-000
    38458 01C2828E-v2022-02-15-183225-000
    37814 01CE5435-v2022-02-15-183225-000
    39028 01CFB168-v2022-02-15-183225-000
    39606 01DB830F-v2022-02-15-183225-000

rclone ls --s3-versions remote:bucket_test/2018/11/29/17/739AA11A/01E4D5C7/01C12570-v2022-02-15-183225-000
(show nothing)

ncw · March 20, 2023, 11:30am

Something weird is going on here, but I'm not sure what.

Here is what happens when I try the same test

$ rclone ls --s3-versions s3:rclone-versions/
       11 file1-v2022-07-07-165438-000.txt
        6 file1-v2022-07-07-165433-000.txt
       11 hello
        6 hello-v2023-03-20-112319-000
       25 subdir/file0.txt
        6 subdir/file0-v2022-07-07-172806-000.txt
        6 subdir/file2.txt
$ rclone ls --s3-versions s3:rclone-versions/hello-v2023-03-20-112319-000
        6 hello-v2023-03-20-112319-000

And

$ rclone copy --s3-versions s3:rclone-versions/hello-v2023-03-20-112319-000 s3:rclone-versions/ -vv
2023/03/20 11:25:54 DEBUG : rclone: Version "v1.62.0-beta.6809.2596414b9.pr-6765-pikpak" starting with parameters ["rclone" "copy" "--s3-versions" "s3:rclone-versions/hello-v2023-03-20-112319-000" "s3:rclone-versions/" "-vv"]
2023/03/20 11:25:54 DEBUG : Creating backend with remote "s3:rclone-versions/hello-v2023-03-20-112319-000"
2023/03/20 11:25:54 DEBUG : Using config file from "/home/ncw/.rclone.conf"
2023/03/20 11:25:54 DEBUG : s3: detected overridden config - adding "{pO73u}" suffix to name
2023/03/20 11:25:54 DEBUG : fs cache: adding new entry for parent of "s3:rclone-versions/hello-v2023-03-20-112319-000", "s3{pO73u}:rclone-versions"
2023/03/20 11:25:54 DEBUG : Creating backend with remote "s3:rclone-versions/"
2023/03/20 11:25:54 DEBUG : s3: detected overridden config - adding "{pO73u}" suffix to name
2023/03/20 11:25:54 DEBUG : fs cache: renaming cache item "s3:rclone-versions/" to be canonical "s3{pO73u}:rclone-versions"
2023/03/20 11:25:54 DEBUG : S3 bucket rclone-versions: don't need to copy/move hello-v2023-03-20-112319-000, it is already at target location
2023/03/20 11:25:54 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.1s

2023/03/20 11:25:54 DEBUG : 4 go routines active

So for some reason rclone isn't finding your versions whereas it finds mine just fine.

It might be worth trying 1.62.2 but I don't see anything relevant in the changelog.

Any thoughts? Anything strange in your setup?

cberni · March 21, 2023, 12:41am

rclone v1.62.2
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.4.0-91-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: none

we thought it could be permission but we set full access and the same problem happens...

Attached is a complete log.
client.log (5.9 KB)

ncw · March 21, 2023, 12:55pm

I managed to reproduce this problem and fix it.

Can you give this a go to see if it fixes it for you too?

v1.63.0-beta.6850.542677d80 on branch master (uploaded in 15-30 mins)

(I accidentally committed it to master so let's hope it works properly

cberni · March 21, 2023, 4:57pm

excellent! it worked perfectly. thank you very much

Now I have a new problem. The command kept the versioned file name and threw it inside a folder with the file name.

we just need to revert the files that were deleted, so this copy should keep the previous name.

rclone --s3-versions copy remote:2018/11/27/11/30E6B749/7E5385DA/7E8CCC9F-v2022-02-15-183218-000 remote:2018/11/27/11/30E6B749/7E5385DA/7E8CCC9F

rclone ls --s3-versions remote:bucket_test/2018/11/29/17/739AA11A/01E4D5C7/
    18020 7E8CCC9F-v2022-02-15-183218-000
    18020 7E8CCC9F/7E8CCC9F-v2022-02-15-183218-000
    18482 7E99FB64-v2022-02-15-183218-000
    18820 7E9A6FC4-v2022-02-15-183218-000
    18672 7E9B5883-v2022-02-15-183218-000

ncw · March 21, 2023, 10:57pm

Glad it worked.

Use rclone copyto instead of rclone copy - that should fix the problem.

cberni · March 22, 2023, 3:18am

a very nice feature would be to have a way to recover an entire directory that has been deleted. and do this automatically by copying all files renaming to their original name using multiple threads. today we have to list all deleted files and make a command line to recover each file.

also we would also like to have versioning for google cloud storage.

ncw · March 22, 2023, 10:58am

You can do this by using

  --s3-version-at Time    Show file versions as they were at the specified time (default off)

You'd pick a time at which the directory contents existed - let's say 2023-02-01 and use this as the cutoff. Supplying this parameter in the connection string will stop it applying to both source and destination, so something like this:

rclone copy remote,version_at=2023-02-01:bucket/path/to/dir remote:bucket/path/to/dir

If that works, I should probably put that in the docs as it can be useful to others.

If you use GCS via the S3 interface it should support versions.

If you want versioning via the native GCS interface - maybe you'd like to run a contract with me to add it? I did the S3 versioning under contract so I have a good idea as to how long it will take! If interested PM me or email me nick@craig-wood.com

cberni · March 22, 2023, 2:22pm

As soon as I test it I'll report back.

about versioning is still not widely used. only in case of disaster. good to know that we can use it through the S3 interface. This solves the issue for now.

system · March 25, 2023, 2:23pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.