Rclone zcat does not work with a multitenant Ceph backend

What is the problem you are having with rclone?

rclone zcat does not work with ceph RadosGW S3 api when multitenancy is enabled.
It also preventa rclone mount to work with bigger files.

You will see in the debug logs that first, a POST is done to the right bucket path

2024/11/11 14:12:38 DEBUG : POST /b84ceb857d8f40feb7eed5e972ec0e56%3Atest-policies/rcat1?uploads= HTTP/1.1

And all goes well. Then for some reason the upload starts but the bucket has lost its multitennacy prefix:

2024/11/11 14:12:38 DEBUG : PUT /test-policies/rcat1?

resulting in a crash.

Note that I am able to rcat small files without problems. I can also use copy. It seems that mount is also using rcat and it also brak that tool.

It also seems to be a regression. I am able to run rclone rcat on the same system with v1.57.0

Run the command 'rclone version' and share the full output of the command.

$rclone --version
rclone v1.68.1

  • os/version: fedora 40 (64 bit)
  • os/kernel: 6.11.4-201.fc40.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.23.1
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

s3 with Rados gw/ceph backend

The command you were trying to run (eg rclone copy /tmp remote:tmp)

$rclone  rcat --s3-no-check-bucket   < 1M   po-test1:'b84ceb857d8f40feb7eed5e972ec0e56:test-policies/rcat1'
2024/11/11 14:13:32 NOTICE: S3 bucket b84ceb857d8f40feb7eed5e972ec0e56:test-policies: Streaming uploads using chunk size 5Mi will have maximum file size of 48.828Gi
2024/11/11 14:13:32 ERROR : rcat1: Post request rcat error: failed to upload chunk 1 with 1048576 bytes: operation error S3: UploadPart, https response error StatusCode: 404, RequestID: tx000004f597470b6729201-006732575c-30a356c-default, HostID: 30a356c-default-default, api error NoSuchBucket: UnknownError
2024/11/11 14:13:32 NOTICE: Failed to rcat with 2 errors: last error was: failed to upload chunk 1 with 1048576 bytes: operation error S3: UploadPart, https response error StatusCode: 404, RequestID: tx000004f597470b6729201-006732575c-30a356c-default, HostID: 30a356c-default-default, api error NoSuchBucket: UnknownError


The rclone config contents with secrets removed.

[po-test1]
type = s3
provider = Ceph
env_auth = false
access_key_id = XXXXXXXXXXXXXXXXXXx
secret_access_key = XXXXXXXXXXXXXXXXx
endpoint = https://objets.juno.calculquebec.ca
acl = 
region =
server_side_encryption =
storage_class =

A log from the command with the -vv flag

rclone  --dump headers  -vv rcat --s3-no-check-bucket   < 1M   po-test1:'b84ceb857d8f40feb7eed5e972ec0e56:test-policies/rcat1'
2024/11/11 14:12:38 NOTICE: Automatically setting -vv as --dump is enabled
2024/11/11 14:12:38 DEBUG : rclone: Version "v1.68.1" starting with parameters ["rclone" "--dump" "headers" "-vv" "rcat" "--s3-no-check-bucket" "po-test1:b84ceb857d8f40feb7eed5e972ec0e56:test-policies/rcat1"]
2024/11/11 14:12:38 DEBUG : Creating backend with remote "po-test1:b84ceb857d8f40feb7eed5e972ec0e56:test-policies/"
2024/11/11 14:12:38 DEBUG : Using config file from "/home/poq/.config/rclone/rclone.conf"
2024/11/11 14:12:38 DEBUG : po-test1: detected overridden config - adding "{Dn7qA}" suffix to name
2024/11/11 14:12:38 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/11/11 14:12:38 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/11/11 14:12:38 DEBUG : fs cache: renaming cache item "po-test1:b84ceb857d8f40feb7eed5e972ec0e56:test-policies/" to be canonical "po-test1{Dn7qA}:b84ceb857d8f40feb7eed5e972ec0e56:test-policies"
2024/11/11 14:12:38 NOTICE: S3 bucket b84ceb857d8f40feb7eed5e972ec0e56:test-policies: Streaming uploads using chunk size 5Mi will have maximum file size of 48.828Gi
2024/11/11 14:12:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/11 14:12:38 DEBUG : HTTP REQUEST (req 0xc0005acc80)
2024/11/11 14:12:38 DEBUG : POST /b84ceb857d8f40feb7eed5e972ec0e56%3Atest-policies/rcat1?uploads= HTTP/1.1
Host: objets.juno.calculquebec.ca
User-Agent: rclone/v1.68.1
Content-Length: 0
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 282154ce-82c1-4133-bb34-9d9dab11fada
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
Content-Type: application/octet-stream
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20241111T191238Z
X-Amz-Meta-Mtime: 1731352358.127922704

2024/11/11 14:12:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/11 14:12:38 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/11 14:12:38 DEBUG : HTTP RESPONSE (req 0xc0005acc80)
2024/11/11 14:12:38 DEBUG : HTTP/1.1 200 OK
Connection: close
Content-Length: 298
Content-Type: application/xml
Date: Mon, 11 Nov 2024 19:12:38 GMT
X-Amz-Request-Id: tx000003a1fedcba3eadfe3-0067325726-30a356c-default

2024/11/11 14:12:38 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/11 14:12:38 DEBUG : rcat1: open chunk writer: started multipart upload: 2~sgdNIs473gceRQeYIoc1eoC2HJh455u
2024/11/11 14:12:38 DEBUG : rcat1: multipart upload: starting chunk 0 size 1Mi offset 0/off
2024/11/11 14:12:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/11 14:12:38 DEBUG : HTTP REQUEST (req 0xc0009663c0)
2024/11/11 14:12:38 DEBUG : PUT /test-policies/rcat1?partNumber=1&uploadId=2~sgdNIs473gceRQeYIoc1eoC2HJh455u&x-id=UploadPart HTTP/1.1
Host: objets.juno.calculquebec.ca
User-Agent: rclone/v1.68.1
Content-Length: 1048576
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 8845ea1d-0c39-45f3-b106-f7bafed4ec25
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
Content-Md5: 8UwBzYT0sSKDSpYHO/yVdQ==
Content-Type: application/octet-stream
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20241111T191238Z

2024/11/11 14:12:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/11 14:12:38 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/11 14:12:38 DEBUG : HTTP RESPONSE (req 0xc0009663c0)
2024/11/11 14:12:38 DEBUG : HTTP/1.1 404 Not Found
Connection: close
Content-Length: 229
Accept-Ranges: bytes
Content-Type: application/xml
Date: Mon, 11 Nov 2024 19:12:38 GMT
X-Amz-Request-Id: tx0000084ca5c5d6fc4df47-0067325726-30a356c-default

2024/11/11 14:12:38 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/11 14:12:38 DEBUG : rcat1: Cancelling multipart upload
2024/11/11 14:12:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/11 14:12:38 DEBUG : HTTP REQUEST (req 0xc0012e8140)
2024/11/11 14:12:38 DEBUG : DELETE /test-policies/rcat1?uploadId=2~sgdNIs473gceRQeYIoc1eoC2HJh455u&x-id=AbortMultipartUpload HTTP/1.1
Host: objets.juno.calculquebec.ca
User-Agent: rclone/v1.68.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 929dbd5c-a55a-49d4-8382-071c1c1c369e
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20241111T191238Z

2024/11/11 14:12:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/11 14:12:38 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/11 14:12:38 DEBUG : HTTP RESPONSE (req 0xc0012e8140)
2024/11/11 14:12:38 DEBUG : HTTP/1.1 404 Not Found
Connection: close
Content-Length: 229
Accept-Ranges: bytes
Content-Type: application/xml
Date: Mon, 11 Nov 2024 19:12:38 GMT
X-Amz-Request-Id: tx0000052dd6e36c3695c7e-0067325726-30a356c-default

2024/11/11 14:12:38 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/11 14:12:38 DEBUG : rcat1: Failed to cancel multipart upload: failed to abort multipart upload "2~sgdNIs473gceRQeYIoc1eoC2HJh455u": operation error S3: AbortMultipartUpload, https response error StatusCode: 404, RequestID: tx0000052dd6e36c3695c7e-0067325726-30a356c-default, HostID: 30a356c-default-default, api error NoSuchBucket: UnknownError
2024/11/11 14:12:38 ERROR : rcat1: Post request rcat error: failed to upload chunk 1 with 1048576 bytes: operation error S3: UploadPart, https response error StatusCode: 404, RequestID: tx0000084ca5c5d6fc4df47-0067325726-30a356c-default, HostID: 30a356c-default-default, api error NoSuchBucket: UnknownError
2024/11/11 14:12:38 DEBUG : 7 go routines active
2024/11/11 14:12:38 NOTICE: Failed to rcat with 2 errors: last error was: failed to upload chunk 1 with 1048576 bytes: operation error S3: UploadPart, https response error StatusCode: 404, RequestID: tx0000084ca5c5d6fc4df47-0067325726-30a356c-default, HostID: 30a356c-default-default, api error NoSuchBucket: UnknownError

Can you try the latest beta please?

And if that doesn't work make a GitHub issue.

Thanks

Crashing in the same way with:

$./rclone-v1.69.0-beta.8402.f639cd9c7-linux-amd64/rclone --version  
rclone v1.69.0-beta.8402.f639cd9c7
- os/version: fedora 40 (64 bit)
- os/kernel: 6.11.4-201.fc40.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.23.3
- go/linking: static
- go/tags: none

The error is this from your log. Rclone creates the multi part upload but gets that error when it tries to post the first chunk.

Can you run this with an earlier version of rclone and see if you can spot the difference in the http requests between old and new versions which might be causing this

Thanks

It does work with 1.57:

$./rclone-v1.57.0-linux-amd64/rclone --version
rclone v1.57.0
- os/version: fedora 40 (64 bit)
- os/kernel: 6.11.4-201.fc40.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.2
- go/linking: static
- go/tags: none

here is the log, you will see that both the POST and PUT call have the right path, with the mutitenancy related prefix: b84ceb857d8f40feb7eed5e972ec0e56%3Atest-policies

$./rclone-v1.57.0-linux-amd64/rclone  rcat --dump headers  --s3-no-check-bucket < 1M po-test1:'b84ceb857d8f40feb7eed5e972ec0e56:test-policies/rcat1'
2024/11/13 13:04:55 NOTICE: Automatically setting -vv as --dump is enabled
2024/11/13 13:04:55 DEBUG : rclone: Version "v1.57.0" starting with parameters ["./rclone-v1.57.0-linux-amd64/rclone" "rcat" "--dump" "headers" "--s3-no-check-bucket" "po-test1:b84ceb857d8f40feb7eed5e972ec0e56:test-policies/rcat1"]
2024/11/13 13:04:55 DEBUG : Creating backend with remote "po-test1:b84ceb857d8f40feb7eed5e972ec0e56:test-policies/"
2024/11/13 13:04:55 DEBUG : Using config file from "/home/poq/.config/rclone/rclone.conf"
2024/11/13 13:04:55 DEBUG : po-test1: detected overridden config - adding "{Dn7qA}" suffix to name
2024/11/13 13:04:55 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/11/13 13:04:55 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/11/13 13:04:55 DEBUG : fs cache: renaming cache item "po-test1:b84ceb857d8f40feb7eed5e972ec0e56:test-policies/" to be canonical "po-test1{Dn7qA}:b84ceb857d8f40feb7eed5e972ec0e56:test-policies"
2024/11/13 13:04:55 NOTICE: S3 bucket b84ceb857d8f40feb7eed5e972ec0e56:test-policies: Streaming uploads using chunk size 5Mi will have maximum file size of 48.828Gi
2024/11/13 13:04:55 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/13 13:04:55 DEBUG : HTTP REQUEST (req 0xc000906400)
2024/11/13 13:04:55 DEBUG : POST /b84ceb857d8f40feb7eed5e972ec0e56%3Atest-policies/rcat1?uploads= HTTP/1.1
Host: objets.juno.calculquebec.ca
User-Agent: rclone/v1.57.0
Content-Length: 0
Authorization: XXXX
Content-Type: application/octet-stream
X-Amz-Acl: private
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20241113T180455Z
X-Amz-Meta-Mtime: 1731521095.842767544
Accept-Encoding: gzip

2024/11/13 13:04:55 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/13 13:04:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/13 13:04:56 DEBUG : HTTP RESPONSE (req 0xc000906400)
2024/11/13 13:04:56 DEBUG : HTTP/1.1 200 OK
Connection: close
Content-Length: 298
Content-Type: application/xml
Date: Wed, 13 Nov 2024 18:04:56 GMT
X-Amz-Request-Id: tx00000b987e35c2aa2d9bc-006734ea47-30a356c-default

2024/11/13 13:04:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/13 13:04:56 DEBUG : rcat1: multipart upload starting chunk 1 size 1Mi offset 0/off
2024/11/13 13:04:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/13 13:04:56 DEBUG : HTTP REQUEST (req 0xc000906500)
2024/11/13 13:04:56 DEBUG : PUT /b84ceb857d8f40feb7eed5e972ec0e56%3Atest-policies/rcat1?partNumber=1&uploadId=2~wGk_sIMSrrpyFPKBhh3RLCfBeDP3EB4 HTTP/1.1
Host: objets.juno.calculquebec.ca
User-Agent: rclone/v1.57.0
Content-Length: 1048576
Authorization: XXXX
Content-Md5: 8UwBzYT0sSKDSpYHO/yVdQ==
X-Amz-Content-Sha256: 65edf773aba3cb273fe6b98a2f40752831f47fcf8a788fb7445bcbfa37c589b3
X-Amz-Date: 20241113T180456Z
Accept-Encoding: gzip

2024/11/13 13:04:56 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/13 13:04:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/13 13:04:57 DEBUG : HTTP RESPONSE (req 0xc000906500)
2024/11/13 13:04:57 DEBUG : HTTP/1.1 200 OK
Connection: close
Content-Length: 0
Accept-Ranges: bytes
Date: Wed, 13 Nov 2024 18:04:57 GMT
Etag: "f14c01cd84f4b122834a96073bfc9575"
X-Amz-Request-Id: tx000001ac22d85996d17d5-006734ea48-30a356c-default

2024/11/13 13:04:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/13 13:04:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/13 13:04:57 DEBUG : HTTP REQUEST (req 0xc00024b600)
2024/11/13 13:04:57 DEBUG : POST /b84ceb857d8f40feb7eed5e972ec0e56%3Atest-policies/rcat1?uploadId=2~wGk_sIMSrrpyFPKBhh3RLCfBeDP3EB4 HTTP/1.1
Host: objets.juno.calculquebec.ca
User-Agent: rclone/v1.57.0
Content-Length: 193
Authorization: XXXX
X-Amz-Content-Sha256: b83e6d320042cf206a95a45289e2eff2e3b1c5636522b521ed8ba77c36f70114
X-Amz-Date: 20241113T180457Z
Accept-Encoding: gzip

2024/11/13 13:04:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/13 13:04:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/13 13:04:57 DEBUG : HTTP RESPONSE (req 0xc00024b600)
2024/11/13 13:04:57 DEBUG : HTTP/1.1 200 OK
Connection: close
Content-Length: 358
Content-Type: application/xml
Date: Wed, 13 Nov 2024 18:04:57 GMT
X-Amz-Request-Id: tx000008ea166b652397219-006734ea49-30a356c-default

2024/11/13 13:04:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/13 13:04:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/13 13:04:57 DEBUG : HTTP REQUEST (req 0xc000906b00)
2024/11/13 13:04:57 DEBUG : HEAD /b84ceb857d8f40feb7eed5e972ec0e56%3Atest-policies/rcat1 HTTP/1.1
Host: objets.juno.calculquebec.ca
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20241113T180457Z

2024/11/13 13:04:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/13 13:04:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/13 13:04:57 DEBUG : HTTP RESPONSE (req 0xc000906b00)
2024/11/13 13:04:57 DEBUG : HTTP/1.1 200 OK
Connection: close
Content-Length: 1048576
Accept-Ranges: bytes
Content-Type: application/octet-stream
Date: Wed, 13 Nov 2024 18:04:57 GMT
Etag: "aa5f2830bd7c2fca0ee6eda7b41e823c-1"
Last-Modified: Wed, 13 Nov 2024 18:04:57 GMT
X-Amz-Meta-Mtime: 1731521095.842767544
X-Amz-Request-Id: tx0000037cd09fe8d62db99-006734ea49-30a356c-default
X-Rgw-Object-Type: Normal

2024/11/13 13:04:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/13 13:04:57 DEBUG : rcat1: Size and modification time the same (differ by 0s, within tolerance 1ns)
2024/11/13 13:04:57 DEBUG : 6 go routines active

Perhaps the problem here is that : is not legal in bucket names

Bucket names can consist only of lowercase letters, numbers, dots (.), and hyphens (-).

And the new AWS S3 SDK is not dealing well with that.

Can you see if it works without a : in the bucket?

Well, I can't do it witout the : that is how ceph deals with multitenency in its s3 API: RGW Multi-tenancy — Ceph Documentation

My goal it to do file upload with a different user than the one who has created the bucket. In that case the backend requires the :.

But, I can confirm that multipart upload works in the case that the bucket creator is the one doing the upload, because the hash with the colon is not required there.

I just had a look through the code. I think this might be a bug in CEPH

Rclone uses the bucket/key/uploadid as returned by the create multipart upload request

And I guess CEPH isn't returning the tenant:bucket here it is just returning the plain bucket.

We can confirm that if you run your test with -vv --dump responses which will show the XML returned.

I can put a workaround for this in rclone easily enough but it would be good to have it confirmed.

Interesting, I will see If I can open an issue with ceph...

Here is the result with v1.69.0-beta.8402.f639cd9c7-linux-amd64

$./rclone-v1.69.0-beta.8402.f639cd9c7-linux-amd64/rclone    rcat -vv  --dump responses  --s3-no-check-bucket   < 1M po-test1:'b84ceb857d8f40feb7eed5e972ec0e56:test-policies/1M-default'
2024/11/14 09:37:38 NOTICE: Automatically setting -vv as --dump is enabled
2024/11/14 09:37:38 DEBUG : rclone: Version "v1.69.0-beta.8402.f639cd9c7" starting with parameters ["./rclone-v1.69.0-beta.8402.f639cd9c7-linux-amd64/rclone" "rcat" "-vv" "--dump" "responses" "--s3-no-check-bucket" "po-test1:b84ceb857d8f40feb7eed5e972ec0e56:test-policies/1M-default"]
2024/11/14 09:37:38 DEBUG : Creating backend with remote "po-test1:b84ceb857d8f40feb7eed5e972ec0e56:test-policies/"
2024/11/14 09:37:38 DEBUG : Using config file from "/home/poq/.config/rclone/rclone.conf"
2024/11/14 09:37:38 DEBUG : po-test1: detected overridden config - adding "{Dn7qA}" suffix to name
2024/11/14 09:37:38 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/11/14 09:37:38 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/11/14 09:37:38 DEBUG : fs cache: renaming cache item "po-test1:b84ceb857d8f40feb7eed5e972ec0e56:test-policies/" to be canonical "po-test1{Dn7qA}:b84ceb857d8f40feb7eed5e972ec0e56:test-policies"
2024/11/14 09:37:38 NOTICE: S3 bucket b84ceb857d8f40feb7eed5e972ec0e56:test-policies: Streaming uploads using chunk size 5Mi will have maximum file size of 48.828Gi
2024/11/14 09:37:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/14 09:37:38 DEBUG : HTTP REQUEST (req 0xc0004e0500)
2024/11/14 09:37:38 DEBUG : POST /b84ceb857d8f40feb7eed5e972ec0e56%3Atest-policies/1M-default?uploads= HTTP/1.1
Host: objets.juno.calculquebec.ca
User-Agent: rclone/v1.69.0-beta.8402.f639cd9c7
Content-Length: 0
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 512b12fd-9190-4c45-a0cc-8d134bfa634b
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
Content-Type: application/octet-stream
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20241114T143738Z
X-Amz-Meta-Mtime: 1731595058.727976585

2024/11/14 09:37:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/14 09:37:38 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/14 09:37:38 DEBUG : HTTP RESPONSE (req 0xc0004e0500)
2024/11/14 09:37:38 DEBUG : HTTP/1.1 200 OK
Connection: close
Content-Length: 303
Content-Type: application/xml
Date: Thu, 14 Nov 2024 14:37:38 GMT
X-Amz-Request-Id: tx00000733b6f1eb90cba67-0067360b32-30a356c-default

<?xml version="1.0" encoding="UTF-8"?><InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Tenant>b84ceb857d8f40feb7eed5e972ec0e56</Tenant><Bucket>test-policies</Bucket><Key>1M-default</Key><UploadId>2~JQaIbI2NLiJwz5toXnXw4ohzoopGkkz</UploadId></InitiateMultipartUploadResult>
2024/11/14 09:37:38 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/14 09:37:38 DEBUG : 1M-default: open chunk writer: started multipart upload: 2~JQaIbI2NLiJwz5toXnXw4ohzoopGkkz
2024/11/14 09:37:38 DEBUG : 1M-default: multipart upload: starting chunk 0 size 1Mi offset 0/off
2024/11/14 09:37:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/14 09:37:38 DEBUG : HTTP REQUEST (req 0xc0004f63c0)
2024/11/14 09:37:38 DEBUG : PUT /test-policies/1M-default?partNumber=1&uploadId=2~JQaIbI2NLiJwz5toXnXw4ohzoopGkkz&x-id=UploadPart HTTP/1.1
Host: objets.juno.calculquebec.ca
User-Agent: rclone/v1.69.0-beta.8402.f639cd9c7
Content-Length: 1048576
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: e26e462c-662f-42ea-998a-e31f6e62c324
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
Content-Md5: 8UwBzYT0sSKDSpYHO/yVdQ==
Content-Type: application/octet-stream
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20241114T143738Z

2024/11/14 09:37:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/14 09:37:38 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/14 09:37:38 DEBUG : HTTP RESPONSE (req 0xc0004f63c0)
2024/11/14 09:37:38 DEBUG : HTTP/1.1 404 Not Found
Connection: close
Content-Length: 229
Accept-Ranges: bytes
Content-Type: application/xml
Date: Thu, 14 Nov 2024 14:37:38 GMT
X-Amz-Request-Id: tx00000114fa969c4b88a0c-0067360b32-30a356c-default

<?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchBucket</Code><BucketName>test-policies</BucketName><RequestId>tx00000114fa969c4b88a0c-0067360b32-30a356c-default</RequestId><HostId>30a356c-default-default</HostId></Error>
2024/11/14 09:37:38 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/14 09:37:38 DEBUG : 1M-default: Cancelling multipart upload
2024/11/14 09:37:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/14 09:37:38 DEBUG : HTTP REQUEST (req 0xc0004e1cc0)
2024/11/14 09:37:38 DEBUG : DELETE /test-policies/1M-default?uploadId=2~JQaIbI2NLiJwz5toXnXw4ohzoopGkkz&x-id=AbortMultipartUpload HTTP/1.1
Host: objets.juno.calculquebec.ca
User-Agent: rclone/v1.69.0-beta.8402.f639cd9c7
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: c28c01cc-9e9e-43f6-8e91-b84258aff46a
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20241114T143738Z

2024/11/14 09:37:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/14 09:37:38 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/14 09:37:38 DEBUG : HTTP RESPONSE (req 0xc0004e1cc0)
2024/11/14 09:37:38 DEBUG : HTTP/1.1 404 Not Found
Connection: close
Content-Length: 229
Accept-Ranges: bytes
Content-Type: application/xml
Date: Thu, 14 Nov 2024 14:37:38 GMT
X-Amz-Request-Id: tx00000b14c9022ede6562b-0067360b32-30a356c-default

<?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchBucket</Code><BucketName>test-policies</BucketName><RequestId>tx00000b14c9022ede6562b-0067360b32-30a356c-default</RequestId><HostId>30a356c-default-default</HostId></Error>
2024/11/14 09:37:38 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/14 09:37:38 DEBUG : 1M-default: Failed to cancel multipart upload: failed to abort multipart upload "2~JQaIbI2NLiJwz5toXnXw4ohzoopGkkz": operation error S3: AbortMultipartUpload, https response error StatusCode: 404, RequestID: tx00000b14c9022ede6562b-0067360b32-30a356c-default, HostID: 30a356c-default-default, api error NoSuchBucket: UnknownError
2024/11/14 09:37:38 ERROR : 1M-default: Post request rcat error: failed to upload chunk 1 with 1048576 bytes: operation error S3: UploadPart, https response error StatusCode: 404, RequestID: tx00000114fa969c4b88a0c-0067360b32-30a356c-default, HostID: 30a356c-default-default, api error NoSuchBucket: UnknownError
2024/11/14 09:37:38 DEBUG : 7 go routines active
2024/11/14 09:37:38 NOTICE: Failed to rcat with 2 errors: last error was: failed to upload chunk 1 with 1048576 bytes: operation error S3: UploadPart, https response error StatusCode: 404, RequestID: tx00000114fa969c4b88a0c-0067360b32-30a356c-default, HostID: 30a356c-default-default, api error NoSuchBucket: UnknownError

And just for good mesure, I am posting the result of v1.57.0-linux-amd64, because I see the tennant id in the response and it is working properly (our workaround is to use that version in our current use case, but we would like to have some features only avalable in later updates) :

./rclone-v1.57.0-linux-amd64/rclone    rcat -vv  --dump responses  --s3-no-check-bucket   < 1M po-test1:'b84ceb857d8f40feb7eed5e972ec0e56:test-policies/1M-default'
2024/11/14 09:40:40 DEBUG : rclone: Version "v1.57.0" starting with parameters ["./rclone-v1.57.0-linux-amd64/rclone" "rcat" "-vv" "--dump" "responses" "--s3-no-check-bucket" "po-test1:b84ceb857d8f40feb7eed5e972ec0e56:test-policies/1M-default"]
2024/11/14 09:40:40 DEBUG : Creating backend with remote "po-test1:b84ceb857d8f40feb7eed5e972ec0e56:test-policies/"
2024/11/14 09:40:40 DEBUG : Using config file from "/home/poq/.config/rclone/rclone.conf"
2024/11/14 09:40:40 DEBUG : po-test1: detected overridden config - adding "{Dn7qA}" suffix to name
2024/11/14 09:40:40 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/11/14 09:40:40 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/11/14 09:40:40 DEBUG : fs cache: renaming cache item "po-test1:b84ceb857d8f40feb7eed5e972ec0e56:test-policies/" to be canonical "po-test1{Dn7qA}:b84ceb857d8f40feb7eed5e972ec0e56:test-policies"
2024/11/14 09:40:40 NOTICE: S3 bucket b84ceb857d8f40feb7eed5e972ec0e56:test-policies: Streaming uploads using chunk size 5Mi will have maximum file size of 48.828Gi
2024/11/14 09:40:40 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/14 09:40:40 DEBUG : HTTP REQUEST (req 0xc000852400)
2024/11/14 09:40:40 DEBUG : POST /b84ceb857d8f40feb7eed5e972ec0e56%3Atest-policies/1M-default?uploads= HTTP/1.1
Host: objets.juno.calculquebec.ca
User-Agent: rclone/v1.57.0
Content-Length: 0
Authorization: XXXX
Content-Type: application/octet-stream
X-Amz-Acl: private
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20241114T144040Z
X-Amz-Meta-Mtime: 1731595240.497212993
Accept-Encoding: gzip

2024/11/14 09:40:40 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/14 09:40:40 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/14 09:40:40 DEBUG : HTTP RESPONSE (req 0xc000852400)
2024/11/14 09:40:40 DEBUG : HTTP/1.1 200 OK
Connection: close
Content-Length: 303
Content-Type: application/xml
Date: Thu, 14 Nov 2024 14:40:40 GMT
X-Amz-Request-Id: tx00000223f033ea4e091fd-0067360be8-30a356c-default

<?xml version="1.0" encoding="UTF-8"?><InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Tenant>b84ceb857d8f40feb7eed5e972ec0e56</Tenant><Bucket>test-policies</Bucket><Key>1M-default</Key><UploadId>2~HFh7iG78Hu3tc5tnoE7D5R-HWo4h51N</UploadId></InitiateMultipartUploadResult>
2024/11/14 09:40:40 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/14 09:40:40 DEBUG : 1M-default: multipart upload starting chunk 1 size 1Mi offset 0/off
2024/11/14 09:40:40 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/14 09:40:40 DEBUG : HTTP REQUEST (req 0xc0005b5200)
2024/11/14 09:40:40 DEBUG : PUT /b84ceb857d8f40feb7eed5e972ec0e56%3Atest-policies/1M-default?partNumber=1&uploadId=2~HFh7iG78Hu3tc5tnoE7D5R-HWo4h51N HTTP/1.1
Host: objets.juno.calculquebec.ca
User-Agent: rclone/v1.57.0
Content-Length: 1048576
Authorization: XXXX
Content-Md5: 8UwBzYT0sSKDSpYHO/yVdQ==
X-Amz-Content-Sha256: 65edf773aba3cb273fe6b98a2f40752831f47fcf8a788fb7445bcbfa37c589b3
X-Amz-Date: 20241114T144040Z
Accept-Encoding: gzip

2024/11/14 09:40:40 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/14 09:40:40 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/14 09:40:40 DEBUG : HTTP RESPONSE (req 0xc0005b5200)
2024/11/14 09:40:40 DEBUG : HTTP/1.1 200 OK
Connection: close
Content-Length: 0
Accept-Ranges: bytes
Date: Thu, 14 Nov 2024 14:40:40 GMT
Etag: "f14c01cd84f4b122834a96073bfc9575"
X-Amz-Request-Id: tx0000058d69cd4804b74a5-0067360be8-30a356c-default

2024/11/14 09:40:40 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/14 09:40:40 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/14 09:40:40 DEBUG : HTTP REQUEST (req 0xc000852600)
2024/11/14 09:40:40 DEBUG : POST /b84ceb857d8f40feb7eed5e972ec0e56%3Atest-policies/1M-default?uploadId=2~HFh7iG78Hu3tc5tnoE7D5R-HWo4h51N HTTP/1.1
Host: objets.juno.calculquebec.ca
User-Agent: rclone/v1.57.0
Content-Length: 193
Authorization: XXXX
X-Amz-Content-Sha256: 280ccc4a39684b7714a24a5f6691e1e3c5e8a34bdae037416d60f4ca081bf810
X-Amz-Date: 20241114T144040Z
Accept-Encoding: gzip

2024/11/14 09:40:40 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/14 09:40:41 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/14 09:40:41 DEBUG : HTTP RESPONSE (req 0xc000852600)
2024/11/14 09:40:41 DEBUG : HTTP/1.1 200 OK
Connection: close
Content-Length: 368
Content-Type: application/xml
Date: Thu, 14 Nov 2024 14:40:41 GMT
X-Amz-Request-Id: tx000000ee549fea324effe-0067360be8-30a356c-default

<?xml version="1.0" encoding="UTF-8"?><CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>objets.juno.calculquebec.ca/b84ceb857d8f40feb7eed5e972ec0e56:test-policies/1M-default</Location><Tenant>b84ceb857d8f40feb7eed5e972ec0e56</Tenant><Bucket>test-policies</Bucket><Key>1M-default</Key><ETag></ETag></CompleteMultipartUploadResult>
2024/11/14 09:40:41 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/14 09:40:41 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/14 09:40:41 DEBUG : HTTP REQUEST (req 0xc00059f700)
2024/11/14 09:40:41 DEBUG : HEAD /b84ceb857d8f40feb7eed5e972ec0e56%3Atest-policies/1M-default HTTP/1.1
Host: objets.juno.calculquebec.ca
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20241114T144041Z

2024/11/14 09:40:41 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/14 09:40:41 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/14 09:40:41 DEBUG : HTTP RESPONSE (req 0xc00059f700)
2024/11/14 09:40:41 DEBUG : HTTP/1.1 200 OK
Connection: close
Content-Length: 1048576
Accept-Ranges: bytes
Content-Type: application/octet-stream
Date: Thu, 14 Nov 2024 14:40:41 GMT
Etag: "aa5f2830bd7c2fca0ee6eda7b41e823c-1"
Last-Modified: Thu, 14 Nov 2024 14:40:40 GMT
X-Amz-Meta-Mtime: 1731595240.497212993
X-Amz-Request-Id: tx00000a7660c8e1bef03f1-0067360be9-30a356c-default
X-Rgw-Object-Type: Normal

2024/11/14 09:40:41 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/14 09:40:41 DEBUG : 1M-default: Size and modification time the same (differ by 0s, within tolerance 1ns)
2024/11/14 09:40:41 DEBUG : 6 go routines active

Thanks for testing.

I think this should fix the problem :crossed_fingers:

v1.69.0-beta.8403.cf37d848a.fix-ceph-multi-tenant on branch fix-ceph-multi-tenant (uploaded in 15-30 mins)

It uses the same value for bucket and key that it used to create the request in the upload part code. I don't know why I read it from the response using the original values seems like a better choice.

Let me know if that works and I'll sneak it into 1.68.2

1 Like

Wow, thanks a lot!
I tested the fix and todays release and it works like a charm.

1 Like