Can I disable Etag?

Help and Support
1

What is the problem you are having with rclone?

I can disable MD5 checksum but can't find a way to disable Etag check

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.2-DEV

  • os/version: sles 15-SP3 (64 bit)
  • os/kernel: 5.3.18-150300.59.87_11.0.78-cray_shasta_c (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.16.12
  • go/linking: dynamic
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

In-house Ceph

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy --s3-disable-checksum --no-check-dest --retries 1 --progress --progress-terminal-title --transfers=1 --s3-chunk-size 1024Mi --s3-upload-concurrency 4 -vv --log-file=./rclone-test.log /scratch/pawsey0002/lcampbell/zeros.img courses01:upload-test

The rclone config contents with secrets removed.

[lcampbell]
type = s3
provider = Ceph
endpoint = https://projects.pawsey.org.au
access_key_id = 
secret_access_key = 
[courses01]
type = s3
provider = Ceph
endpoint = https://projects.pawsey.org.au
access_key_id = 
secret_access_key =

A log from the command with the -vv flag

2023/06/14 08:13:35 DEBUG : Creating backend with remote "/scratch/pawsey0002/lcampbell/zeros.img"
2023/06/14 08:13:35 DEBUG : Using config file from "/home/lcampbell/.config/rclone/rclone.conf"
2023/06/14 08:13:35 DEBUG : fs cache: adding new entry for parent of "/scratch/pawsey0002/lcampbell/zeros.img", "/scratch/pawsey0002/lcampbell"
2023/06/14 08:13:35 DEBUG : Creating backend with remote "courses01:upload-test"
2023/06/14 08:13:35 DEBUG : courses01: detected overridden config - adding "{SlLBg}" suffix to name
2023/06/14 08:13:35 DEBUG : fs cache: renaming cache item "courses01:upload-test" to be canonical "courses01{SlLBg}:upload-test"
2023/06/14 08:13:35 DEBUG : zeros.img: Need to transfer - File not found at Destination
2023/06/14 08:13:35 INFO  : S3 bucket upload-test: Bucket "upload-test" created with ACL "private"

023/06/14 08:30:10 DEBUG : zeros.img: Multipart upload Etag: 7b6636397e1a372a0c48b9832210f18b-400 OK
2023/06/14 08:30:10 DEBUG : zeros.img: Dst hash empty - aborting Src hash check
2023/06/14 08:30:10 INFO  : zeros.img: Copied (new)
2

Sorry, version info didn't copy for some reason. Here it is:
rclone v1.59.2-DEV

  • os/version: sles 15-SP3 (64 bit)
  • os/kernel: 5.3.18-150300.59.87_11.0.78-cray_shasta_c (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.16.12
  • go/linking: dynamic
  • go/tags: none

AND I didn't take out the READ ME! :man_facepalming:

3

Update rclone to the latest version - I think some Ceph issues have been fixes since v.1.59

4

This check is really cheap. Rclone has to hash each block as it uploads them anyway as part of the S3 protocol and this final check just hashes all the hashes.

If you want to disable it then use --s3-use-multipart-etag=false

5

Great, thank you, I'll try it :smiley:

6

the MD5 hash is calculated by rclone, and stored as metadata
with multi-part uploads, not usable to verify file transfers.

whereas, with Etag is calculate by both rclone and the S3 provider.
imho, that is a real check

for a deeper look, check my posts at

and
with a multipart, server-side copy, it is no clear what rclone is doing when printing md5 = e9a99fd39b444139598f997510c24076 OK seems that rclone is simply comparing the header X-Amz-Meta-Md5chksum between source and dest. That is correct.

7

Thank you, all good information.

8

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.