Surprise bill on Cloudflare R2

[asdffdsa (jojothehumanmonkey)]

@hyperknot

What's really strange is that Cloudflare is not sending modification dates in their GET directory endpoints? So using --checksum is a must?

that is how all S3 providers work.

can use --size-only or --checksum

[hyperknot (Zsolt Ero)]

Actually, I believe it has to be a bug, as the dump bodies show this. So LastModified is definitely present in the file list response.

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult
	xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
	<Name>legter</Name>
	<Contents>
		<Key>data%2F5m%2F2025-03-19%2F14-13.json</Key>
		<Size>8822</Size>
		<LastModified>2025-05-19T13:31:18.499Z</LastModified>
		<ETag>&quot;7639f9cc211a8e3dc07ef8ce9495bcab&quot;</ETag>
		<StorageClass>STANDARD</StorageClass>
	</Contents>

[asdffdsa (jojothehumanmonkey)]

@hyperknot

Actually, I believe it has to be a bug, as the dump bodies show this. So LastModified is definitely present in the file list response.

that is not a bug, just a confusing quirk.

there are two times.

1. Amz-Meta-Mtime, which is the mod-time of the source file
   set by rclone. to read, need to HEAD file.

2. LastModified, think of it as CreationDate, the time the file was uploaded.
   set by s3 provider. to read, no need to HEAD file. use --use-server-modtime

---

rclone touch file.ext --localtime --timestamp=1970-01-01T00:00:00

ls -al file.ext --full-time
-rw-rw-r-- 1 user01 user01 0 1970-01-01 00:00:00.000000000 -0500 file.ext

rclone copyto file.ext cloudflare:zork/file.ext -v
2025/05/19 10:46:08 INFO  : file.ext: Copied (new)

rclone lsl cloudflare:zork/file.ext
        0 1970-01-01 00:00:00.000000000 file.ext

rclone lsl cloudflare:zork/file.ext --use-server-modtime
        0 2025-05-19 10:46:08.218000000 file.ext

[hyperknot (Zsolt Ero)]

And how would copy/sync with with --use-server-modtime? Does rclone update the time on the local file with the server modtime value it gets back when uploading a file?

Also, when using checksum, does rclone run the full summing algorithm, compatible with remote ETag locally?

[jwink3101 (Justin Winokur)]

@hyperknot

And how would copy/sync with with --use-server-modtime?

You usually will do it with --update. Here are the docs:

Use this flag to disable the extra API call and rely instead on the server's modified time. In cases such as a local to remote sync using --update, knowing the local file is newer than the time it was last uploaded to the remote is sufficient. In those cases, this flag can speed up the process and reduce the number of API calls necessary.

Using this flag on a sync operation without also using --update would cause all files modified at any time other than the last upload time to be uploaded again, which is probably not what you want.

[asdffdsa (jojothehumanmonkey)]

@hyperknot

Also, when using checksum, does rclone run the full summing algorithm, compatible with remote ETag locally?

when rclone uploads a file, it saves the md5 as metadata. that is what rclone uses with rclone check

ETag is used a few ways.

1. for each chunk of a file, uploaded or downloaded, rclone calculates the ETag and compares it to ETag in the s3 response header
2. for file upload. after the entire file is uploaded, rclone compares the ETag for the source file to the ETag in the s3 response header.

---

for the deep dive about ETag, check out my topics

use ETag to verify S3 file transfers

---

In some cases, rclone does not use ETag to verify files

What is the problem you are having with rclone? after file transfer completes, force rclone to compare rclone calculated ETag against provider calculated ETag. from the log, looks like wasabi returns the Etag, yet rclone does not use it? <CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>https://azork.s3.us-east-2.wasabisys.com/file.ext</Location><Bucket>azork</Bucket><Key>file.ext</Key> <ETag>&quot;195c549125ec2fe75e9f3e8444b4a98c-2&quot;</ETag></Complet…

[hyperknot (Zsolt Ero)]

Hehe, this links to my SO answer from back in 2017

What is the algorithm to compute the Amazon-S3 Etag for a file larger than 5GB?

What I'm interested here, is what happens locally? I mean if I have 5 million files / 500 GB of data, then calculating the ETag hashing locally would be quite resource intensive / it might trash a server while just trying to do a backup job.

So what happens locally? Does rclone save into some kind of fancy Linux filesystem metadata the calculated hash, or it actually always reads every single file and runs a full hashing on them? I mean it needs some kind of a database/cache or filesystem metadata storage to skip these calculations, doesn't it?

If that's the case, then I believe the only way to use rclone with R2 is either:

• --use-server-modtime + --update
• --size

I mean if I don't want to trash the server and don't want surprise bills.

---

Just for reference, I run dump=bodies with Backblaze B2. Is src_last_modified_millis the metadata which is added by rclone?

{
  "accountId": "x",
  "action": "upload",
  "bucketId": "x",
  "contentLength": 280251,
  "contentSha1": "5c9c1012a0f493e7ccf097cc581d88fc446cb060",
  "contentType": "application/octet-stream",
  "fileId": "x",
  "fileInfo":
  {
    "src_last_modified_millis": "1692476823207"
  },
  "fileName": "x",
  "size": 280251,
  "uploadTimestamp": 1695293754831
}