Multipart copy: failed to open source: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided

Help and Support
1

What is the problem you are having with rclone?

When i try to GET large objects from a bucket i get the error: SignatureDoesNotMatch.

For PUT objects rclone works fine.

I have a proxy behind swift and it's log shows: 200 for PUT & HEAD and 403 for GET. When i use the internal endpoint without proxy GET works fine.

I've seen some threads about this topic but I haven't found the solution, could anyone help me, pls?

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.2

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 5.4.0-128-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.18.6
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Private Cloud= Openstack + Swift (S3 API)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy migracao:migracao-dados/marvin.20221007.gz /dados -vv

The rclone config contents with secrets removed.

[migracao]
type = s3
provider = Other
env_auth = false
max_upload_parts = 999
access_key_id = <removed>
secret_access_key = <removed>
region = us-east-1
endpoint = <removed>

A log from the command with the -vv flag

rclone copy migracao:migracao-dados/marvin.20221007.gz /dados -vv
2022/10/20 06:34:54 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rclone" "copy" "migracao:migracao-dados/marvin.20221007.gz" "/dados" "-vv"]
2022/10/20 06:34:54 DEBUG : Creating backend with remote "migracao:migracao-dados/marvin.20221007.gz"
2022/10/20 06:34:54 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2022/10/20 06:34:54 DEBUG : fs cache: adding new entry for parent of "migracao:migracao-dados/marvin.20221007.gz", "migracao:migracao-dados"
2022/10/20 06:34:54 DEBUG : Creating backend with remote "/dados"
2022/10/20 06:34:54 DEBUG : marvin.20221007.gz: Sizes differ (src 152026426505 vs dst 0)
2022/10/20 06:34:54 DEBUG : marvin.20221007.gz: Starting multi-thread copy with 4 parts of size 35.396Gi
2022/10/20 06:34:54 DEBUG : marvin.20221007.gz: multi-thread copy: stream 4/4 (114019860480-152026426505) size 35.396Gi starting
2022/10/20 06:34:54 DEBUG : marvin.20221007.gz: multi-thread copy: stream 2/4 (38006620160-76013240320) size 35.396Gi starting
2022/10/20 06:34:54 DEBUG : marvin.20221007.gz: multi-thread copy: stream 1/4 (0-38006620160) size 35.396Gi starting
2022/10/20 06:34:54 DEBUG : marvin.20221007.gz: multi-thread copy: stream 3/4 (76013240320-114019860480) size 35.396Gi starting
2022/10/20 06:34:54 DEBUG : marvin.20221007.gz: multi-thread copy: stream 4/4 failed: multipart copy: failed to open source: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
	status code: 403, request id: txa85ea49c38324d969e491-006351163e, host id: txa85ea49c38324d969e491-006351163e
2022/10/20 06:34:54 DEBUG : marvin.20221007.gz: multi-thread copy: stream 2/4 failed: multipart copy: failed to open source: RequestCanceled: request context canceled
caused by: context canceled
2022/10/20 06:34:54 DEBUG : marvin.20221007.gz: multi-thread copy: stream 3/4 failed: multipart copy: failed to open source: RequestCanceled: request context canceled
caused by: context canceled
2022/10/20 06:34:54 DEBUG : marvin.20221007.gz: multi-thread copy: stream 1/4 failed: multipart copy: failed to open source: RequestCanceled: request context canceled
caused by: context canceled
2022/10/20 06:34:54 ERROR : marvin.20221007.gz: Failed to copy: multipart copy: failed to open source: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
	status code: 403, request id: txa85ea49c38324d969e491-006351163e, host id: txa85ea49c38324d969e491-006351163e
2022/10/20 06:34:54 ERROR : Attempt 1/3 failed with 1 errors and: multipart copy: failed to open source: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
	status code: 403, request id: txa85ea49c38324d969e491-006351163e, host id: txa85ea49c38324d969e491-006351163e
2022/10/20 06:34:55 DEBUG : marvin.20221007.gz: Sizes differ (src 152026426505 vs dst 0)
2022/10/20 06:34:55 DEBUG : marvin.20221007.gz: Starting multi-thread copy with 4 parts of size 35.396Gi
2022/10/20 06:34:55 DEBUG : marvin.20221007.gz: multi-thread copy: stream 4/4 (114019860480-152026426505) size 35.396Gi starting
2022/10/20 06:34:55 DEBUG : marvin.20221007.gz: multi-thread copy: stream 1/4 (0-38006620160) size 35.396Gi starting
2022/10/20 06:34:55 DEBUG : marvin.20221007.gz: multi-thread copy: stream 2/4 (38006620160-76013240320) size 35.396Gi starting
2022/10/20 06:34:55 DEBUG : marvin.20221007.gz: multi-thread copy: stream 3/4 (76013240320-114019860480) size 35.396Gi starting
2022/10/20 06:34:55 DEBUG : marvin.20221007.gz: multi-thread copy: stream 1/4 failed: multipart copy: failed to open source: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
	status code: 403, request id: tx9834130b3cba4391ac924-006351163f, host id: tx9834130b3cba4391ac924-006351163f
2022/10/20 06:34:55 DEBUG : marvin.20221007.gz: multi-thread copy: stream 4/4 failed: multipart copy: failed to open source: RequestCanceled: request context canceled
caused by: context canceled
2022/10/20 06:34:55 DEBUG : marvin.20221007.gz: multi-thread copy: stream 2/4 failed: multipart copy: failed to open source: RequestCanceled: request context canceled
caused by: context canceled
2022/10/20 06:34:55 DEBUG : marvin.20221007.gz: multi-thread copy: stream 3/4 failed: multipart copy: failed to open source: RequestCanceled: request context canceled
caused by: context canceled
2022/10/20 06:34:55 ERROR : marvin.20221007.gz: Failed to copy: multipart copy: failed to open source: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
	status code: 403, request id: tx9834130b3cba4391ac924-006351163f, host id: tx9834130b3cba4391ac924-006351163f
2022/10/20 06:34:55 ERROR : Attempt 2/3 failed with 1 errors and: multipart copy: failed to open source: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
	status code: 403, request id: tx9834130b3cba4391ac924-006351163f, host id: tx9834130b3cba4391ac924-006351163f
2022/10/20 06:34:55 DEBUG : marvin.20221007.gz: Sizes differ (src 152026426505 vs dst 0)
2022/10/20 06:34:55 DEBUG : marvin.20221007.gz: Starting multi-thread copy with 4 parts of size 35.396Gi
2022/10/20 06:34:55 DEBUG : marvin.20221007.gz: multi-thread copy: stream 4/4 (114019860480-152026426505) size 35.396Gi starting
2022/10/20 06:34:55 DEBUG : marvin.20221007.gz: multi-thread copy: stream 1/4 (0-38006620160) size 35.396Gi starting
2022/10/20 06:34:55 DEBUG : marvin.20221007.gz: multi-thread copy: stream 3/4 (76013240320-114019860480) size 35.396Gi starting
2022/10/20 06:34:55 DEBUG : marvin.20221007.gz: multi-thread copy: stream 2/4 (38006620160-76013240320) size 35.396Gi starting
2022/10/20 06:34:55 DEBUG : marvin.20221007.gz: multi-thread copy: stream 1/4 failed: multipart copy: failed to open source: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
	status code: 403, request id: txc06ce68edb8c4213b3593-006351163f, host id: txc06ce68edb8c4213b3593-006351163f
2022/10/20 06:34:55 DEBUG : marvin.20221007.gz: multi-thread copy: stream 2/4 failed: multipart copy: failed to open source: RequestCanceled: request context canceled
caused by: context canceled
2022/10/20 06:34:55 DEBUG : marvin.20221007.gz: multi-thread copy: stream 3/4 failed: multipart copy: failed to open source: RequestCanceled: request context canceled
caused by: context canceled
2022/10/20 06:34:55 DEBUG : marvin.20221007.gz: multi-thread copy: stream 4/4 failed: multipart copy: failed to open source: RequestCanceled: request context canceled
caused by: context canceled
2022/10/20 06:34:55 ERROR : marvin.20221007.gz: Failed to copy: multipart copy: failed to open source: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
	status code: 403, request id: txc06ce68edb8c4213b3593-006351163f, host id: txc06ce68edb8c4213b3593-006351163f
2022/10/20 06:34:55 ERROR : Attempt 3/3 failed with 1 errors and: multipart copy: failed to open source: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
	status code: 403, request id: txc06ce68edb8c4213b3593-006351163f, host id: txc06ce68edb8c4213b3593-006351163f
2022/10/20 06:34:55 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.7s

2022/10/20 06:34:55 DEBUG : 5 go routines active
2022/10/20 06:34:55 Failed to copy: multipart copy: failed to open source: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
	status code: 403, request id: txc06ce68edb8c4213b3593-006351163f, host id: txc06ce68edb8c4213b3593-006351163f
2

Not sure what is causing this. Rclone uses the official S3 SDK so is probably doing its auth correctly. I'd suspect a bug in Openstack + Swift (S3 API)...

Try disabling multithread streams --multi-thread-streams 0 and see if that helps.

  --multi-thread-streams int         Max number of streams to use for multi-thread downloads (default 4)
1 Like
3

Hi Nick!

yes! it's worked! i used the command:

rclone copy migracao:migracao-dados/marvin.20221007.gz /dados -vv --multi-thread-streams 0

Thank you very much!

1 Like
4

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.