Sync from local to Cloudflare R2 results in application/octet-stream content type for some files with .data extension

[chris_xf]

What is the problem you are having with rclone?

I have successfully synced files from a local file system to Cloudflare R2 using rclone.

A bunch of files, because they are served programmatically, have a .data extension. The underlying content type of the files is correct:

file --mime-type -b 99-e0124c7f9f33974f4929c543dfdf5f33.data 
image/jpeg

But in place on Cloudflare R2 the files have a content type of application/octet-stream

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.0-beta.7480.e1b0417c2
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 6.2.0-1012-aws (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.4
- go/linking: static
- go/tags: none

The issue persisted in this beta version, but I was also on the latest stable version previously.

Which cloud storage system are you using? (eg Google Drive)

Cloudflare R2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync . R2:internal/ --verbose --transfers 64 --checkers 64

rclone sync . R2:internal/ --verbose --transfers 64 --checkers 64 --metadata

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[R2]
type = s3
provider = Cloudflare
access_key_id = XXX
secret_access_key = XXX
region = auto
endpoint = https://XXX.r2.cloudflarestorage.com

A log from the command that you were trying to run with the -vv flag

N/A

Worth noting that .data files could essentially be any content type. I'm assuming rclone is doing some sort of file extension coercion to ascertain the content type here.

Ideally it would be utilising whatever content type actually exists in the file metadata but this does not happen.

In an ideal world, I believe that if no content type is set at all, then Cloudflare R2 will automatically infer it itself. That would be a sufficient solution if it was possible to do that with rclone.

Any help would be greatly appreciated. Thanks!

[ncw (Nick Craig-Wood)]

@chris_xf

Worth noting that .data files could essentially be any content type. I'm assuming rclone is doing some sort of file extension coercion to ascertain the content type here

Rclone uses the go runtime for this and on most oses it uses the system way of turning extensions into mime types.

Are all your .data files the same content type?

You can override the content type using an http header flag.

You could also use a --metadata sync and make sure your .data files have a content-type metadata entry.

Maybe rclone should have a flag to do content type sniffing from the local backend.

[chris_xf]

@ncw

Are all your .data files the same content type?

Unfortunately not. Could be images, PDF, basically anything. The backend database knows what they are so the software can serve them with the correct headers.

If there’s no solution I might have to manually update everything via a script.

@ncw

You could also use a --metadata sync and make sure your .data files have a content-type metadata entry.

Not sure if I’ve done anything wrong but I’ve tried using the metadata flag and it didn’t seem to have any effect. Or more specifically I think on R2 there was a few extra pieces of metadata but content type wasn’t one of them.

@ncw

Maybe rclone should have a flag to do content type sniffing from the local backend.

That would certainly solve the problem for me. Seen a few other posts that might suggest it could help there too.

Inferring the mime type from a file extension feels a little fragile to me, particularly with user generated content. Especially if the OS is able to ascertain the correct mime type itself.