Sync with Cloudflare R2 overrides all files on remote

Help and Support
1

What is the problem you are having with rclone?

Since the 1.59 betas I've been trying to sync web assets from one local folder to a remote Cloudflare R2 bucket and I don't seem to get rclone to actually do partial sync on subsequent runs.

I've mentioned it already on a github thread but reaching out here to not spam over there.

I've revisited it after the stable release and I'm getting the same result: running the same sync command transfers 100% of the files every time.

Previously I've tried different combinations of --fast-list , --checksum and --s3-no-head without success. Now, with the stable release, --s3-no-head throws on me with:

2022/07/30 16:07:36 DEBUG : rclone: Version "v1.59.0" starting with parameters ["rclone" "sync" "-vv" "--s3-no-head" "_build/admin" "r2:admin"]
2022/07/30 16:07:36 DEBUG : Creating backend with remote "_build/admin"
2022/07/30 16:07:36 DEBUG : Using config file from "/home/sam/.config/rclone/rclone.conf"
2022/07/30 16:07:36 DEBUG : fs cache: renaming cache item "_build/admin" to be canonical "/home/sam/Code/masterworks/frontend/_build/admin"
2022/07/30 16:07:36 DEBUG : Creating backend with remote "r2:admin"
2022/07/30 16:07:36 DEBUG : r2: detected overridden config - adding "{NBTUO}" suffix to name
2022/07/30 16:07:36 DEBUG : fs cache: renaming cache item "r2:admin" to be canonical "r2{NBTUO}:admin"
2022/07/30 16:07:37 DEBUG : S3 bucket admin: Waiting for checks to finish
2022/07/30 16:07:37 DEBUG : S3 bucket admin: Waiting for transfers to finish
panic: reflect: Elem of invalid type s3.PutObjectInput

goroutine 56 [running]:
reflect.(*rtype).Elem(0x0?)
	reflect/type.go:965 +0x134
github.com/rclone/rclone/lib/structs.SetFrom({0x1c7ab20?, 0xc00052dc20}, {0x1c9f500?, 0xc000a46000})
	github.com/rclone/rclone/lib/structs/structs.go:22 +0xdf
github.com/rclone/rclone/backend/s3.(*Object).Update(0xc00037c090, {0x2142048, 0xc000634f00}, {0x2130ae0, 0xc000644000}, {0x7fde1a64ad38?, 0xc000300540?}, {0xc000c36060, 0x1, 0x1})
	github.com/rclone/rclone/backend/s3/s3.go:4573 +0x1345
github.com/rclone/rclone/backend/s3.(*Fs).Put(0xc00090fd40, {0x2142048, 0xc000634f00}, {0x2130ae0, 0xc000644000}, {0x7fde1a64ad38?, 0xc000300540?}, {0xc000c36060, 0x1, 0x1})
	github.com/rclone/rclone/backend/s3/s3.go:3085 +0x125
github.com/rclone/rclone/fs/operations.Copy({0x2142048, 0xc000634f00}, {0x2152930, 0xc00090fd40}, {0x0?, 0x0?}, {0xc00011a290, 0xa}, {0x2152000, 0xc000300540})
	github.com/rclone/rclone/fs/operations/operations.go:501 +0x1b24
github.com/rclone/rclone/fs/sync.(*syncCopyMove).pairCopyOrMove(0xc000849d40, {0x2142048, 0xc000634f00}, 0x0?, {0x2152930, 0xc00090fd40}, 0x0?, 0x0?)
	github.com/rclone/rclone/fs/sync/sync.go:416 +0x1bb
created by github.com/rclone/rclone/fs/sync.(*syncCopyMove).startTransfers
	github.com/rclone/rclone/fs/sync/sync.go:443 +0x45

Which didn't happen on the beta releases I tried, but I would say this is unrelated.

I've tried again with a totally new R2 bucket and after deleting my existing rclone config (~/.config/rclone/rclone.conf) and recreating it with the interactive rclone config command (see its resulting config file later on this post). No luck, it transfers 100% of the files every time.

I've checked the md5 hashes of the -vv output and they are all the same on every run.

I've tried with and without trailing slashes in all the permutations for SOURCE and DESTINATION, always getting same result.

I've triple-checked that the files were actually uploaded to the R2 bucket, they were.

Any ideas on what am I doing wrong?

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.0
- os/version: fedora 36 (64 bit)
- os/kernel: 5.18.13-200.fc36.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.18.3
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Cloudflare R2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync -vv _build/admin r2:admin

The rclone config contents with secrets removed.

[r2]
type = s3
provider = Cloudflare
access_key_id = <REDACTED>
secret_access_key = <REDACTED>
region = auto
endpoint = https://<REDACTED>.r2.cloudflarestorage.com/rclone-test

A log from the command with the -vv flag

2

Could you pick one file and run through a test with that?

rclone ls and show the file is there
rclone copy same file

Run those with -vv and share the logs and we can see what's going on.

1 Like
3

Ok, indeed ls doesn't list anything at all. Picking the first file from sync output:

$ rclone -vv ls r2:admin/humans.txt

2022/07/30 17:59:01 DEBUG : rclone: Version "v1.59.0" starting with parameters ["rclone" "-vv" "ls" "r2:admin/humans.txt"]
2022/07/30 17:59:01 DEBUG : Creating backend with remote "r2:admin/humans.txt"
2022/07/30 17:59:01 DEBUG : Using config file from "/home/sam/.config/rclone/rclone.conf"
2022/07/30 17:59:01 DEBUG : fs cache: adding new entry for parent of "r2:admin/humans.txt", "r2:admin"
2022/07/30 17:59:02 DEBUG : 4 go routines active

Trying to list the whole folder:

$ rclone -vv ls r2:admin/

2022/07/30 17:59:55 DEBUG : rclone: Version "v1.59.0" starting with parameters ["rclone" "-vv" "ls" "r2:admin/"]
2022/07/30 17:59:55 DEBUG : Creating backend with remote "r2:admin/"
2022/07/30 17:59:55 DEBUG : Using config file from "/home/sam/.config/rclone/rclone.conf"
2022/07/30 17:59:55 DEBUG : fs cache: renaming cache item "r2:admin/" to be canonical "r2:admin"
2022/07/30 17:59:55 DEBUG : 4 go routines active

EDIT (forgot to paste this one): Trying to list the whole remote:

$ rclone -vv ls r2:

2022/07/30 18:18:39 DEBUG : rclone: Version "v1.59.0" starting with parameters ["rclone" "-vv" "ls" "r2:"]
2022/07/30 18:18:39 DEBUG : Creating backend with remote "r2:"
2022/07/30 18:18:39 DEBUG : Using config file from "/home/sam/.config/rclone/rclone.conf"
2022/07/30 18:18:40 DEBUG : 4 go routines active

No files listed in every case.

I triple-checked at Cloudflare's dashboard and the files are indeed there:

image
image1714×1699 206 KB

Then I deleted every object from the bucket from Cloudflare's dashboard and re-running ls afterwards gives a slightly different output:

$ rclone -vv ls r2:admin/

2022/07/30 18:07:57 DEBUG : rclone: Version "v1.59.0" starting with parameters ["rclone" "-vv" "ls" "r2:admin/"]
2022/07/30 18:07:57 DEBUG : Creating backend with remote "r2:admin/"
2022/07/30 18:07:57 DEBUG : Using config file from "/home/sam/.config/rclone/rclone.conf"
2022/07/30 18:07:57 DEBUG : fs cache: renaming cache item "r2:admin/" to be canonical "r2:admin"
2022/07/30 18:07:58 DEBUG : 4 go routines active
2022/07/30 18:07:58 Failed to ls: directory not found

One thing I noticed that I didn't notice before while looking at Cloudflare's dashboard, is that sync is created an object called admin (at the root, without extension) and this is its content:

<CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><LocationConstraint>auto</LocationConstraint></CreateBucketConfiguration>

No idea if this is expected or related at all with the issue I'm facing.

Seeing the DEBUG messages about cache I figured might be a cache issue, but using --cache-db-purge doesn't seem to help:

rclone -vv --cache-db-purge ls r2:

2022/07/30 18:16:04 DEBUG : rclone: Version "v1.59.0" starting with parameters ["rclone" "-vv" "--cache-db-purge" "ls" "r2:"]
2022/07/30 18:16:04 DEBUG : Creating backend with remote "r2:"
2022/07/30 18:16:04 DEBUG : Using config file from "/home/sam/.config/rclone/rclone.conf"
2022/07/30 18:16:05 DEBUG : 4 go routines active
4

Maybe try without the rclone-test on the end there?

endpoint = https://<REDACTED>.r2.cloudflarestorage.com
1 Like
5

might try endpoint = https://accountid.r2.cloudflarestorage.com
where accountid is the cloudflare r2 account number.

and this is my remote config, change accountid to your r2 account id.

[cloudflare01]
type = s3
provider = Cloudflare
access_key_id = redacted
secret_access_key = redacted
region = auto
endpoint = https://accountid.r2.cloudflarestorage.com
acl = private
2 Likes
6

This is fixed in the beta and the fix will go into 1.59.1 soon

This is most likely the problem solution

1 Like
7

Alright, thank you all here! It was indeed the bucket name at the end of the endpoint.

I guess I misunderstood what a remote means + Cloudflare dashboard gives you the COPY button which includes the bucket + sync actually works in this case when writing (maybe there's some validation/warning that can be added?)

Thank you all again.

8

Hmm, maybe some warning if the endpoint isn't the root. Though I don't think that would work for all S3 clones

Perhaps it could say in the endpoint config that the bucket should not be included - would this have helped?

People make this mistake with aws S3 sometimes too, but put bucket.endpoint.com

Rclone never wants the bucket in the endpoint.

1 Like
9

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.