Linode Object -> Cloudflare R2 : second rclone re-copies all files

Help and Support
1

First off, thank you for all the effort put into rclone. It makes the online storage world so much easier to deal with.

What is the problem you are having with rclone?

rclone is re-copying all the files I transferred even though they exist on the destination server.

I'm copying from a Linode object storage bucket to a Cloudflare R2 bucket. I copied 3TB of files yesterday and was done successfully. However, today I decided to test the syncing and rclone is attempting to copy everything again. I've tried using the flags --update, --use-server-modtime, etc but to no avail. The file mod dates on the source are years in the past so that shouldn't be an issue (though I have attempted to use --modify-window).

Run the command 'rclone version' and share the full output of the command.

rclone v1.60.1
- os/version: centos 7.2.1511 (64 bit)
- os/kernel: 5.17.5-x86_64-linode154 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.3
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Linode object storage and Cloudflare R2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

/usr/bin/rclone copy -vv --update --transfers=8 linode:storagefx/ r2:/

The rclone config contents with secrets removed.

[linode]
type = s3
provider = Other
access_key_id = 
secret_access_key = 
endpoint = 
location_constraint = US
acl = private


[r2]
type = s3
provider = Cloudflare
access_key_id = 
secret_access_key =
endpoint = 
bucket_acl = private

A log from the command with the -vv flag

2022/12/05 07:55:20 DEBUG : rclone: Version "v1.60.1" starting with parameters ["/usr/bin/rclone" "copy" "--update" "-vv" "linode:storagefx/" "r2:/"]
2022/12/05 07:55:20 DEBUG : Creating backend with remote "linode:storagefx/"
2022/12/05 07:55:20 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2022/12/05 07:55:20 DEBUG : fs cache: renaming cache item "linode:storagefx/" to be canonical "linode:storagefx"
2022/12/05 07:55:20 DEBUG : Creating backend with remote "r2:/"
2022/12/05 07:55:20 DEBUG : fs cache: renaming cache item "r2:/" to be canonical "r2:"
2022/12/05 07:55:22 INFO  : S3 root: Bucket "f" created with ACL "private"
2022/12/05 07:55:25 DEBUG : f/fcp213/fcp213-class01.zip: Src hash empty - aborting Dst hash check
2022/12/05 07:55:25 INFO  : f/fcp213/fcp213-class01.zip: Copied (new)
2022/12/05 07:55:26 DEBUG : f/fcp213/fcp213-class02.zip: Src hash empty - aborting Dst hash check
2022/12/05 07:55:26 INFO  : f/fcp213/fcp213-class02.zip: Copied (new)
2022/12/05 07:55:26 DEBUG : f/fcp213/fcp213-class03.zip: Src hash empty - aborting Dst hash check
2022/12/05 07:55:26 INFO  : f/fcp213/fcp213-class03.zip: Copied (new)
2022/12/05 07:55:28 DEBUG : f/fcp213/fcp213-class04.zip: Src hash empty - aborting Dst hash check
2022/12/05 07:55:28 INFO  : f/fcp213/fcp213-class04.zip: Copied (new)
2

I have the same issue with files copied from local hard drive.

rclone copy -P ~/Documents/ cloudflare_backup:/Documents --transfers 100

this copies all files, another run copies all files again.
Same command but with check instead of copy returns error: all files are missing, despite all of them being on destination

3

@johnmontfx

I think what is happening here is that rclone is wondering whether it should update the modtime, so it does a hash check but the source doesn't have a hash, so it goes on to a full upload to be sure.

If you are sure the files really are the same, then this is something --refresh-times will fix I think

  --refresh-times                      Refresh the modtime of remote files

@Pretext4761 please open another topic and fill in the support template and provide a log - thanks!

4

I managed to fix this for me.

Remote config should not include bucket name in endpoint (so only hostname and no path, ex. https://aaaaaaaaa.r2.cloudflarestorage.com)

Add bucket name when running rclone, ex. for "backup" bucket: rclone copy Documents r2:backup/Documents

5

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.