What is the problem you are having with rclone?
rclone zcat does not work with ceph RadosGW S3 api when multitenancy is enabled.
It also preventa rclone mount to work with bigger files.
You will see in the debug logs that first, a POST is done to the right bucket path
2024/11/11 14:12:38 DEBUG : POST /b84ceb857d8f40feb7eed5e972ec0e56%3Atest-policies/rcat1?uploads= HTTP/1.1
And all goes well. Then for some reason the upload starts but the bucket has lost its multitennacy prefix:
2024/11/11 14:12:38 DEBUG : PUT /test-policies/rcat1?
resulting in a crash.
Note that I am able to rcat small files without problems. I can also use copy. It seems that mount is also using rcat and it also brak that tool.
It also seems to be a regression. I am able to run rclone rcat on the same system with v1.57.0
Run the command 'rclone version' and share the full output of the command.
$rclone --version
rclone v1.68.1
- os/version: fedora 40 (64 bit)
- os/kernel: 6.11.4-201.fc40.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.23.1
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
s3 with Rados gw/ceph backend
The command you were trying to run (eg rclone copy /tmp remote:tmp)
$rclone rcat --s3-no-check-bucket < 1M po-test1:'b84ceb857d8f40feb7eed5e972ec0e56:test-policies/rcat1'
2024/11/11 14:13:32 NOTICE: S3 bucket b84ceb857d8f40feb7eed5e972ec0e56:test-policies: Streaming uploads using chunk size 5Mi will have maximum file size of 48.828Gi
2024/11/11 14:13:32 ERROR : rcat1: Post request rcat error: failed to upload chunk 1 with 1048576 bytes: operation error S3: UploadPart, https response error StatusCode: 404, RequestID: tx000004f597470b6729201-006732575c-30a356c-default, HostID: 30a356c-default-default, api error NoSuchBucket: UnknownError
2024/11/11 14:13:32 NOTICE: Failed to rcat with 2 errors: last error was: failed to upload chunk 1 with 1048576 bytes: operation error S3: UploadPart, https response error StatusCode: 404, RequestID: tx000004f597470b6729201-006732575c-30a356c-default, HostID: 30a356c-default-default, api error NoSuchBucket: UnknownError
The rclone config contents with secrets removed.
[po-test1]
type = s3
provider = Ceph
env_auth = false
access_key_id = XXXXXXXXXXXXXXXXXXx
secret_access_key = XXXXXXXXXXXXXXXXx
endpoint = https://objets.juno.calculquebec.ca
acl =
region =
server_side_encryption =
storage_class =
A log from the command with the -vv flag
rclone --dump headers -vv rcat --s3-no-check-bucket < 1M po-test1:'b84ceb857d8f40feb7eed5e972ec0e56:test-policies/rcat1'
2024/11/11 14:12:38 NOTICE: Automatically setting -vv as --dump is enabled
2024/11/11 14:12:38 DEBUG : rclone: Version "v1.68.1" starting with parameters ["rclone" "--dump" "headers" "-vv" "rcat" "--s3-no-check-bucket" "po-test1:b84ceb857d8f40feb7eed5e972ec0e56:test-policies/rcat1"]
2024/11/11 14:12:38 DEBUG : Creating backend with remote "po-test1:b84ceb857d8f40feb7eed5e972ec0e56:test-policies/"
2024/11/11 14:12:38 DEBUG : Using config file from "/home/poq/.config/rclone/rclone.conf"
2024/11/11 14:12:38 DEBUG : po-test1: detected overridden config - adding "{Dn7qA}" suffix to name
2024/11/11 14:12:38 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/11/11 14:12:38 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/11/11 14:12:38 DEBUG : fs cache: renaming cache item "po-test1:b84ceb857d8f40feb7eed5e972ec0e56:test-policies/" to be canonical "po-test1{Dn7qA}:b84ceb857d8f40feb7eed5e972ec0e56:test-policies"
2024/11/11 14:12:38 NOTICE: S3 bucket b84ceb857d8f40feb7eed5e972ec0e56:test-policies: Streaming uploads using chunk size 5Mi will have maximum file size of 48.828Gi
2024/11/11 14:12:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/11 14:12:38 DEBUG : HTTP REQUEST (req 0xc0005acc80)
2024/11/11 14:12:38 DEBUG : POST /b84ceb857d8f40feb7eed5e972ec0e56%3Atest-policies/rcat1?uploads= HTTP/1.1
Host: objets.juno.calculquebec.ca
User-Agent: rclone/v1.68.1
Content-Length: 0
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 282154ce-82c1-4133-bb34-9d9dab11fada
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
Content-Type: application/octet-stream
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20241111T191238Z
X-Amz-Meta-Mtime: 1731352358.127922704
2024/11/11 14:12:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/11 14:12:38 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/11 14:12:38 DEBUG : HTTP RESPONSE (req 0xc0005acc80)
2024/11/11 14:12:38 DEBUG : HTTP/1.1 200 OK
Connection: close
Content-Length: 298
Content-Type: application/xml
Date: Mon, 11 Nov 2024 19:12:38 GMT
X-Amz-Request-Id: tx000003a1fedcba3eadfe3-0067325726-30a356c-default
2024/11/11 14:12:38 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/11 14:12:38 DEBUG : rcat1: open chunk writer: started multipart upload: 2~sgdNIs473gceRQeYIoc1eoC2HJh455u
2024/11/11 14:12:38 DEBUG : rcat1: multipart upload: starting chunk 0 size 1Mi offset 0/off
2024/11/11 14:12:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/11 14:12:38 DEBUG : HTTP REQUEST (req 0xc0009663c0)
2024/11/11 14:12:38 DEBUG : PUT /test-policies/rcat1?partNumber=1&uploadId=2~sgdNIs473gceRQeYIoc1eoC2HJh455u&x-id=UploadPart HTTP/1.1
Host: objets.juno.calculquebec.ca
User-Agent: rclone/v1.68.1
Content-Length: 1048576
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 8845ea1d-0c39-45f3-b106-f7bafed4ec25
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
Content-Md5: 8UwBzYT0sSKDSpYHO/yVdQ==
Content-Type: application/octet-stream
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20241111T191238Z
2024/11/11 14:12:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/11 14:12:38 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/11 14:12:38 DEBUG : HTTP RESPONSE (req 0xc0009663c0)
2024/11/11 14:12:38 DEBUG : HTTP/1.1 404 Not Found
Connection: close
Content-Length: 229
Accept-Ranges: bytes
Content-Type: application/xml
Date: Mon, 11 Nov 2024 19:12:38 GMT
X-Amz-Request-Id: tx0000084ca5c5d6fc4df47-0067325726-30a356c-default
2024/11/11 14:12:38 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/11 14:12:38 DEBUG : rcat1: Cancelling multipart upload
2024/11/11 14:12:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/11 14:12:38 DEBUG : HTTP REQUEST (req 0xc0012e8140)
2024/11/11 14:12:38 DEBUG : DELETE /test-policies/rcat1?uploadId=2~sgdNIs473gceRQeYIoc1eoC2HJh455u&x-id=AbortMultipartUpload HTTP/1.1
Host: objets.juno.calculquebec.ca
User-Agent: rclone/v1.68.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 929dbd5c-a55a-49d4-8382-071c1c1c369e
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20241111T191238Z
2024/11/11 14:12:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/11/11 14:12:38 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/11 14:12:38 DEBUG : HTTP RESPONSE (req 0xc0012e8140)
2024/11/11 14:12:38 DEBUG : HTTP/1.1 404 Not Found
Connection: close
Content-Length: 229
Accept-Ranges: bytes
Content-Type: application/xml
Date: Mon, 11 Nov 2024 19:12:38 GMT
X-Amz-Request-Id: tx0000052dd6e36c3695c7e-0067325726-30a356c-default
2024/11/11 14:12:38 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/11/11 14:12:38 DEBUG : rcat1: Failed to cancel multipart upload: failed to abort multipart upload "2~sgdNIs473gceRQeYIoc1eoC2HJh455u": operation error S3: AbortMultipartUpload, https response error StatusCode: 404, RequestID: tx0000052dd6e36c3695c7e-0067325726-30a356c-default, HostID: 30a356c-default-default, api error NoSuchBucket: UnknownError
2024/11/11 14:12:38 ERROR : rcat1: Post request rcat error: failed to upload chunk 1 with 1048576 bytes: operation error S3: UploadPart, https response error StatusCode: 404, RequestID: tx0000084ca5c5d6fc4df47-0067325726-30a356c-default, HostID: 30a356c-default-default, api error NoSuchBucket: UnknownError
2024/11/11 14:12:38 DEBUG : 7 go routines active
2024/11/11 14:12:38 NOTICE: Failed to rcat with 2 errors: last error was: failed to upload chunk 1 with 1048576 bytes: operation error S3: UploadPart, https response error StatusCode: 404, RequestID: tx0000084ca5c5d6fc4df47-0067325726-30a356c-default, HostID: 30a356c-default-default, api error NoSuchBucket: UnknownError
