Send X-Amz-Content-Sha256 header with PUT for Ceph S3

I am trying to create some intermediate authentication between rclone and a Ceph S3 service. I started off with minio as my development s3 endpoint.

I have managed to get s3cmd working with the AWS V4 and V2 authentication.

I am struggling with rclone as for a typical rclone copy, the X-Amz-Content-Sha256 header is not set when a file is sent with a PUT request. For my example I tried to upload a 10K file.

You can see from the log below

For the command
rclone -vvv --dump headers --config=conf/rclone.minio.conf copy tests/test_data/10K local_red:/tests

2021/06/19 17:43:02 DEBUG : HTTP REQUEST (req 0xc0003ec100)
2021/06/19 17:43:02 DEBUG : PUT /tests/10K?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minioadmin%2F20210619%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210619T154302Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime&X-Amz-Signature=a3c62c45328034ccaba2fc712de06099acd2dfecbba50288c0402ede0cc6ba00 HTTP/1.1
User-Agent: rclone/v1.55.1
Content-Length: 10241
content-md5: JTNr8mA9kHNPITNgDFfkZg==
content-type: application/octet-stream
x-amz-acl: private
x-amz-meta-mtime: 1624093944.888508793
Accept-Encoding: gzip

So even though there is content (Content-Length is 10241) there is no sha256 of that content in the query paramaters or in the headers.

However if a head is called, the header I need exists.

2021/06/19 17:43:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/06/19 17:43:02 DEBUG : HTTP REQUEST (req 0xc000672c00)
2021/06/19 17:43:02 DEBUG : HEAD /tests/10K HTTP/1.1
User-Agent: rclone/v1.55.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210619T154302Z

2021/06/19 17:43:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

I know rclone is calculating the Sha256, as it is required for the AWS v4 signature.

If I do not get the hash from rsync I will have to calculate the hash myself, which seems like a waste seeing as rclone is doing it and my Ceph backend will have to do it, so I do not want to have to do it in my intermediate authentication step when there is a header that can hold the hash.

Anybody know if this is something I could solve easily without calculating my own hash?

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.