I'm perplexed by "rclone hashsum's" output format. I don't understand why it's inconsistent.
Run the command 'rclone version' and share the full output of the command.
rclone v1.67.0
os/version: ubuntu 22.04 (64 bit)
os/kernel: 6.5.0-44-generic (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.22.4
go/linking: static
go/tags: none
Are you on the latest version of rclone? You can validate by checking the version listed here: Rclone downloads
I wasn't, but I installed it for this report. It's giving the same result.
Which cloud storage system are you using? (eg Google Drive)
minio-s3 in a docker container in this case. We use several others though.
The command you were trying to run (eg rclone copy /tmp remote:tmp)
No, my data was not uploaded by rclone. It was uploaded by an internal-only application.
Assuming you mean "--dump headers", I've included the output here:
2024/08/07 09:52:41 NOTICE: Automatically setting -vv as --dump is enabled
2024/08/07 09:52:41 DEBUG : rclone: Version "v1.67.0" starting with parameters ["rclone" "hashsum" "MD5" "dev-env-s3:/bs1/blah_pl1/data/33/06/" "--dump" "headers"]
2024/08/07 09:52:41 DEBUG : Creating backend with remote "dev-env-s3:/bs1/blah_pl1/data/33/06/"
2024/08/07 09:52:41 DEBUG : Using config file from "/data/home/dstromberg/.config/rclone/rclone.conf"
2024/08/07 09:52:41 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/08/07 09:52:41 DEBUG : Resolving service "s3" region "us-east-1"
2024/08/07 09:52:41 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/08/07 09:52:41 DEBUG : fs cache: renaming cache item "dev-env-s3:/bs1/blah_pl1/data/33/06/" to be canonical "dev-env-s3:/bs1/blah_pl1/data/33/06"
2024/08/07 09:52:41 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/08/07 09:52:41 DEBUG : HTTP REQUEST (req 0xc00023f7a0)
2024/08/07 09:52:41 DEBUG : GET /?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=bs1%2Fblah_pl1%2Fdata%2F33%2F06%2F HTTP/1.1
Host: localhost:9000
User-Agent: rclone/v1.67.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240807T165241Z
Accept-Encoding: gzip
2024/08/07 09:52:41 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
3306c357d2cd2e91534b300c9bf4ca05
2024/08/07 09:52:41 DEBUG : 6 go routines active
Do I need to upload an MD5 hash header along with the data in my app, to make it so rclone can verify MD5 hashes more quickly? And is the possible lack of that header likely to be why "rclone hashsum MD5" is giving inconsistent output?
...which is kind of the opposite of what I was (slightly) anticipating. I was thinking maybe long files would have more than one hash because of chunking that isn't needed for short files.
tho, internally, rclone does hash each chunk, as an additional type of file verification. DEBUG : VeeamAgentUser3bef6c4c-360f-11b2-a85c-9070b4e81f40/ABP_EN10_CDRIVE_-_EN10/ABP_EN10_C2024-08-03T105418.vbk: multipart upload wrote chunk 10 with 268435456 bytes and etag "f1f8048a8817b980d29dfe7b6cefd972"