Surprise bill on Cloudflare R2

hyperknot · May 19, 2025, 11:52am

What is the problem you are having with rclone?

I have a tiny directory, 110 MB in total, 17k files. I run this command every 5 minutes.

rclone copy --transfers=8 --fast-list  /src remote:dest

My conf is the following:

[remote]
type = s3
provider = Cloudflare
access_key_id = x
secret_access_key = x
endpoint = https://x.r2.cloudflarestorage.com
no_check_bucket = true

This gave me a relatively big surprise bill, about $40, showing I am currently doing 150 million Class B requests per month. Which comes out correctly, as listing 17k files every 5 minutes is exactly 150 million requests per month.

But I'm wondering, why does rclone need to make a list request for every single file?

Case in point, I have a much bigger backup, running into Backblaze B2. 7 million (!) files, 500 GB, running once per hour. It's been going on for years, yet I average only 3.8 M requests per month, which is less than what single rclone command would cause if this was on Cloudflare R2!

This is the Backblaze B2 command:

rclone sync \
    --config=/dev/null \
    --b2-account x \
    --b2-key x \
    --crypt-remote :b2:x \
    --crypt-filename-encryption=off \
    --crypt-directory-name-encryption=false \
    --crypt-password x \
    --transfers=8 \
    --multi-thread-streams=8 \
    --fast-list \
    --links \
    --local-no-check-updated \
    /src \
    :crypt:

Can you shine some light on what is happening here? Is this normal with the Cloudflare integration? Basically meaning that Cloudflare cannot be used for a regular sync cronjob?

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.2
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-133-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.6
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Backblaze B2

A log from the command that you were trying to run with the `-vv` flag

I'm happy to provide a log if you tell me exactly what parameters might be interesting here.

asdffdsa · May 19, 2025, 12:24pm

can run rclone selfupdate and test again.

hyperknot · May 19, 2025, 12:29pm

OK, now I'm on

rclone v1.69.2
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-133-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.24.2
- go/linking: static
- go/tags: none

I'll report back as the dashboard updates.

asdffdsa · May 19, 2025, 12:33pm

as some tests:

rclone copy -/src remote:dest try --max-age=5m --no-traverse --transfers=8
rclone copy -/src remote:dest try --max-age=5m --no-traverse --transfers=8 --no-check-dest

hyperknot · May 19, 2025, 12:35pm

OK, I try step by step, first with the updated version now.

Also, what logging would allow me to see the number of requests made?

asdffdsa · May 19, 2025, 12:36pm

afiak, no way to do that, without some scripting.
to log the api calls, --dump=headers

hyperknot · May 19, 2025, 12:36pm

I mean just a grep command on the log file. I'll try dump headers then.

hyperknot · May 19, 2025, 12:45pm

OK, so without changing anything, just after selfupdate. There is a HEAD requests for each file.

2025/05/19 12:40:41 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/05/19 12:40:41 DEBUG : 5m/2025-03-30/21-10.json: Size and modification time the same (differ by 0s, within tolerance 1ns)
2025/05/19 12:40:41 DEBUG : 5m/2025-03-30/21-10.json: Unchanged skipping
2025/05/19 12:40:41 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/05/19 12:40:41 DEBUG : HTTP REQUEST (req 0xc000fa1540)
2025/05/19 12:40:41 DEBUG : HEAD /legter/data/5m/2025-03-30/21-45.json HTTP/1.1
Host: 99fde2e5efdeb199c6910cdeaa276a97.r2.cloudflarestorage.com
User-Agent: rclone/v1.69.2
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: e6ca6f20-3bd6-4c12-b143-1f36f231a463
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20250519T124041Z

Now trying --no-traverse

asdffdsa · May 19, 2025, 12:46pm

need to post the exact command

hyperknot · May 19, 2025, 12:47pm

That was this

rclone copy \
    --verbose=1 \
    --transfers=8 \
    --fast-list \
    --stats-file-name-length=0 \
    --stats-one-line \
    --log-file rclone.log \
    --dump=headers \
    /path/to/activations \
    remote:legter/data

hyperknot · May 19, 2025, 12:48pm

--no-traverse didn't help.

hyperknot · May 19, 2025, 12:51pm

--no-check-dest totally breaks rclone, it tries to copy everything now.

2025/05/19 12:50:12 DEBUG : daily/2025-03-20.json: Need to transfer - File not found at Destination
2025/05/19 12:50:12 DEBUG : daily/2025-03-21.json: Need to transfer - File not found at Destination
2025/05/19 12:50:12 DEBUG : daily/2025-03-22.json: Need to transfer - File not found at Destination
2025/05/19 12:50:12 DEBUG : daily/2025-03-23.json: Need to transfer - File not found at Destination
2025/05/19 12:50:12 DEBUG : daily/2025-03-24.json: Need to transfer - File not found at Destination

hyperknot · May 19, 2025, 12:55pm

Final test with max-age, still doing a HEAD request for every single file:

2025/05/19 12:53:14 DEBUG : HTTP REQUEST (req 0xc000614780)
2025/05/19 12:53:14 DEBUG : HEAD /legter/data/5m/2025-03-19/14-13.json HTTP/1.1

asdffdsa · May 19, 2025, 1:03pm

when rclone asks for a list of files:

modtime is not included.
size is included.
hash is included.

by default, rclone is comparing files by modtime, rclone must HEAD per file to get the modtime.

so, to prevent that HEAD per file, use one of these:

--size-only
--checksum

hyperknot · May 19, 2025, 1:24pm

So far, I tried every single command you've written on their own + the default
--fast-list. Now I'm trying:

            '--transfers=8',
            # '--fast-list',
            # '--no-check-dest',
            '--max-age=5m',
            '--no-traverse',
            '--no-check-dest',

asdffdsa · May 19, 2025, 1:26pm

check my last post

hyperknot · May 19, 2025, 1:30pm

I'm doing that as well, step by step.

So

            '--transfers=8',
            # '--fast-list',
            # '--no-check-dest',
            '--max-age=5m',
            '--no-traverse',
            '--no-check-dest',

results in 17 requests.

hyperknot · May 19, 2025, 1:33pm

            '--transfers=8',
            # '--fast-list',
            # '--no-check-dest',
            # '--max-age=5m',
            '--no-traverse',
            '--no-check-dest',

This is trying to re-upload every single file.

2025/05/19 13:31:46 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/05/19 13:31:46 DEBUG : 5m/2025-03-27/03-45.json: md5 = 50451026a007b1bad1fb597f3239251c OK
2025/05/19 13:31:46 INFO  : 5m/2025-03-27/03-45.json: Copied (new)
2025/05/19 13:31:46 DEBUG : 5m/2025-04-18/12-20.json: Need to transfer - File not found at Destination
2025/05/19 13:31:46 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/05/19 13:31:46 DEBUG : HTTP REQUEST (req 0xc000ff08c0)
2025/05/19 13:31:46 DEBUG : PUT /legter/data/5m/2025-03-27/04-35.json?x-id=PutObject HTTP/1.1

hyperknot · May 19, 2025, 1:43pm

            '--transfers=8',
            '--fast-list',
            '--checksum',
            # '--max-age=5m',
            # '--no-traverse',
            # '--no-check-dest',

=> This results in 19 requests, one per each 1k files, looking perfect!

hyperknot · May 19, 2025, 1:50pm

OK, this seems to be the winner here.

What's really strange is that Cloudflare is not sending modification dates in their GET directory endpoints? So using --checksum is a must?

Surprise bill on Cloudflare R2

What is the problem you are having with rclone?

Run the command 'rclone version' and share the full output of the command.

Which cloud storage system are you using? (eg Google Drive)

A log from the command that you were trying to run with the -vv flag

A log from the command that you were trying to run with the `-vv` flag