S3 copy with SSE (Bucket Key) fail if not us-east-1

Suspected Bug
1

What is the problem you are having with rclone?

I am using RClone as part of seldon, and am hitting an issue whereby I can't seem to download files from an S3 bucket that is no in us-east-1 region due to the KMS key unable to be read by the client.

I have been able to correctly set the Region and remove (which remove the initially warning about switching regions, but am still getting error AccessDenied: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0
- os/version: alpine 3.14.2 (64 bit)
- os/kernel: 5.10.76-linuxkit (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -v copy s3:{bucket_name}/{bucket_prefix} .

The rclone config contents with secrets removed.

I have set the following config variables including the KMS_KEY_ID as an ARN:

  RCLONE_CONFIG_S3_TYPE: "s3"
  RCLONE_CONFIG_S3_ACCESS_KEY_ID: "{aws_access_key_id}"
  RCLONE_CONFIG_S3_SECRET_ACCESS_KEY: "{aws_secret_access_key}"
  RCLONE_CONFIG_S3_REGION: "{region_name}"
  RCLONE_CONFIG_S3_SSE_KMS_KEY_ID: "{bucket_sse_kms_key_id}"
  RCLONE_CONFIG_S3_SERVER_SIDE_ENCRYPTION: "{'aws:kms' if bucket_sse_kms_key_id else ''}"

A log from the command with the -vv flag

I am getting errors for all the files I am attempting to copy on the prefix

2022/07/05 00:13:09 NOTICE: Config file "/config/rclone/rclone.conf" not found - using defaults
2022/07/05 00:13:11 ERROR : <<<S3_PATH>>>: Failed to copy: failed to open source object: AccessDenied: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.
	status code: 403, request id: 61KTSAR2A89AKGBD, host id: fUOl0c9CMOn3jaHO/lABvzCuEnPzKEyOkQa6ZEvyQ1kCn/C9ZQS+fwvvtQcnY1/1S6+g4bsvp+s=
2

Are you sure the master key does exist in that region?

According to FAQs | AWS Key Management Service (KMS) | Amazon Web Services (AWS)

Q: What geographic region are my keys stored in?
A single-Region KMS key generated by AWS KMS is stored and used only in the Region in which it was created. With AWS KMS multi-Region keys you can choose to replicate a multi-Region primary key into multiple Regions within the same AWS partition.

So unless your key is multi-region you'll need one in the other region too.

3

I am using the default amazon managed S3 Bucket Key which was created when the bucket was created, so yes the bucket and key are in the us-west-2 region. If I attempt to read from a bucket in us-east-1 region that also has a S3 Bucket Key, then I have no issue. The framework seems to "default" to us-east-1 which I override by setting the RCLONE_CONFIG_S3_REGION option, however my suspicion is that this is not being propagated when attempting to copy.

4

Try running with "-vv --dump bodies" and you can see what rclone is sending and see if it matches your expectations?

I think it unlikely the region gets lost but the above will tell you for sure!

6

Hi Nick,

I have been able to confirm that the env vars are correctly being pass down and set in the config (as per logs at end of the post).

I have also been able to reproduce this locally for a us-west-2 bucket, where by I attempt to use rclone to copy and get error, but can use the aws cli and have it succeed eg the following fails:

export RCLONE_CONFIG_S3_ACCESS_KEY_ID="MY_KEY"
export RCLONE_CONFIG_S3_SECRET_ACCESS_KEY="MY_SECRET"
export RCLONE_CONFIG_S3_REGION="us-west-2"
export RCLONE_CONFIG_S3_SSE_KMS_KEY_ID="arn:aws:kms:us-west-2:MY_ACCOUNT:key/f6fbd5c7-ab1b-4eb6-a769-ea789c24125f"
export RCLONE_CONFIG_S3_SERVER_SIDE_ENCRYPTION="aws:kms"

rclone -vv copy s3:MY_PREFIX-us-west-2-474375891613/model_repository/titanic_ensemble/config.pbtxt test.txt

But the following exceeds

aws s3 cp s3://MY_PREFIX-us-west-2-474375891613/model_repository/titanic_ensemble/config.pbtxt test.txt

I have obfuscated the prefix and account name for this bucket, but everything else should be easy to repo.

Below are the logs if they shed any more light.

2022/07/06 02:30:21 DEBUG : rclone: Version "v1.57.0" starting with parameters ["/usr/local/bin/rclone" "-vv" "--dump" "bodies" "copy" "s3:MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository" "."]
2022/07/06 02:30:21 DEBUG : Creating backend with remote "s3:MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository"
2022/07/06 02:30:21 DEBUG : Setting type="s3" for "s3" from environment variable RCLONE_CONFIG_S3_TYPE
2022/07/06 02:30:21 DEBUG : Setting access_key_id="MY_ACCESS_KEY" for "s3" from environment variable RCLONE_CONFIG_S3_ACCESS_KEY_ID
2022/07/06 02:30:21 DEBUG : Setting secret_access_key="MY_SECRET_KEY" for "s3" from environment variable RCLONE_CONFIG_S3_SECRET_ACCESS_KEY
2022/07/06 02:30:21 DEBUG : Setting region="us-west-2" for "s3" from environment variable RCLONE_CONFIG_S3_REGION
2022/07/06 02:30:21 DEBUG : Setting region="us-west-2" for "s3" from environment variable RCLONE_CONFIG_S3_REGION
2022/07/06 02:30:21 DEBUG : Setting region="us-west-2" for "s3" from environment variable RCLONE_CONFIG_S3_REGION
2022/07/06 02:30:21 DEBUG : Setting server_side_encryption="aws:kms" for "s3" from environment variable RCLONE_CONFIG_S3_SERVER_SIDE_ENCRYPTION
2022/07/06 02:30:21 DEBUG : Setting sse_kms_key_id="arn:aws:kms:us-west-2:MY_ACCOUNT:key/f6fbd5c7-ab1b-4eb6-a769-ea789c24125f" for "s3" from environment variable RCLONE_CONFIG_S3_SSE_KMS_KEY_ID
2022/07/06 02:30:21 DEBUG : s3: detected overridden config - adding "{MmKCF}" suffix to name
2022/07/06 02:30:21 NOTICE: Config file "/config/rclone/rclone.conf" not found - using defaults
2022/07/06 02:30:21 DEBUG : Setting access_key_id="MY_ACCESS_KEY" for "s3" from environment variable RCLONE_CONFIG_S3_ACCESS_KEY_ID
2022/07/06 02:30:21 DEBUG : Setting secret_access_key="MY_SECRET_KEY" for "s3" from environment variable RCLONE_CONFIG_S3_SECRET_ACCESS_KEY
2022/07/06 02:30:21 DEBUG : Setting region="us-west-2" for "s3" from environment variable RCLONE_CONFIG_S3_REGION
2022/07/06 02:30:21 DEBUG : Setting server_side_encryption="aws:kms" for "s3" from environment variable RCLONE_CONFIG_S3_SERVER_SIDE_ENCRYPTION
2022/07/06 02:30:21 DEBUG : Setting sse_kms_key_id="arn:aws:kms:us-west-2:MY_ACCOUNT:key/f6fbd5c7-ab1b-4eb6-a769-ea789c24125f" for "s3" from environment variable RCLONE_CONFIG_S3_SSE_KMS_KEY_ID
2022/07/06 02:30:21 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/07/06 02:30:21 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/07/06 02:30:21 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:21 DEBUG : HTTP REQUEST (req 0xc0004ea000)
2022/07/06 02:30:21 DEBUG : HEAD /MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository HTTP/1.1
Host: s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220706T023021Z

2022/07/06 02:30:21 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:22 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/06 02:30:22 DEBUG : HTTP RESPONSE (req 0xc0004ea000)
2022/07/06 02:30:22 DEBUG : HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Wed, 06 Jul 2022 02:30:22 GMT
Server: AmazonS3
X-Amz-Id-2: 0h1/4TS3BucLRn6Mfy7czlwyMFSQ96BlKUtDP5XEZuLeQCqWLyYvjpsSZp9Hkm5cE3fpu2CU+98=
X-Amz-Request-Id: MNHKYQ6SH5H8G52B

2022/07/06 02:30:22 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/06 02:30:22 DEBUG : fs cache: renaming cache item "s3:MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository" to be canonical "s3{MmKCF}:MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository"
2022/07/06 02:30:22 DEBUG : Creating backend with remote "."
2022/07/06 02:30:22 DEBUG : fs cache: renaming cache item "." to be canonical "/data"
2022/07/06 02:30:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:22 DEBUG : HTTP REQUEST (req 0xc0006e2900)
2022/07/06 02:30:22 DEBUG : GET /MY_PREFIX-us-west-2-MY_ACCOUNT?delimiter=%2F&max-keys=1000&prefix=model_repository%2F HTTP/1.1
Host: s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220706T023022Z
Accept-Encoding: gzip

2022/07/06 02:30:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:22 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/06 02:30:22 DEBUG : HTTP RESPONSE (req 0xc0006e2900)
2022/07/06 02:30:22 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Wed, 06 Jul 2022 02:30:23 GMT
Server: AmazonS3
X-Amz-Bucket-Region: us-west-2
X-Amz-Id-2: +qHR9+ZgLnMuyj1rxydgHQ0HDr15cwV6paYM8t3NtXrSsiX2046n06PvvVYSguLdmv+ZiwbiJGs=
X-Amz-Request-Id: MNHN4EC7FBQ10TY9

4a6
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>MY_PREFIX-us-west-2-MY_ACCOUNT</Name><Prefix>model_repository/</Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><CommonPrefixes><Prefix>model_repository/ensemble_entry/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>model_repository/ensemble_postprocess/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>model_repository/ensemble_predict/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>model_repository/ensemble_preprocess/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>model_repository/pytorch_processed/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>model_repository/pytorch_raw/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>model_repository/titanic_ensemble/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>model_repository/titanic_ensemble_postprocess/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>model_repository/titanic_ensemble_predict/</Prefix></CommonPrefixes><CommonPrefixes><Prefix>model_repository/titanic_ensemble_preprocess/</Prefix></CommonPrefixes></ListBucketResult>
0

2022/07/06 02:30:22 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/06 02:30:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:22 DEBUG : HTTP REQUEST (req 0xc0006e2e00)
2022/07/06 02:30:22 DEBUG : GET /MY_PREFIX-us-west-2-MY_ACCOUNT?delimiter=%2F&max-keys=1000&prefix=model_repository%2Fensemble_preprocess%2F HTTP/1.1
Host: s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220706T023022Z
Accept-Encoding: gzip

2022/07/06 02:30:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:22 DEBUG : HTTP REQUEST (req 0xc0004ea600)
2022/07/06 02:30:22 DEBUG : GET /MY_PREFIX-us-west-2-MY_ACCOUNT?delimiter=%2F&max-keys=1000&prefix=model_repository%2Fensemble_postprocess%2F HTTP/1.1
Host: s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220706T023022Z
Accept-Encoding: gzip

2022/07/06 02:30:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:22 DEBUG : HTTP REQUEST (req 0xc0005cda00)
2022/07/06 02:30:22 DEBUG : GET /MY_PREFIX-us-west-2-MY_ACCOUNT?delimiter=%2F&max-keys=1000&prefix=model_repository%2Fpytorch_processed%2F HTTP/1.1
Host: s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220706T023022Z
Accept-Encoding: gzip

2022/07/06 02:30:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:22 DEBUG : HTTP REQUEST (req 0xc0005cde00)
2022/07/06 02:30:22 DEBUG : GET /MY_PREFIX-us-west-2-MY_ACCOUNT?delimiter=%2F&max-keys=1000&prefix=model_repository%2Fensemble_entry%2F HTTP/1.1
Host: s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220706T023022Z
Accept-Encoding: gzip

2022/07/06 02:30:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:22 DEBUG : HTTP REQUEST (req 0xc000655500)
2022/07/06 02:30:22 DEBUG : GET /MY_PREFIX-us-west-2-MY_ACCOUNT?delimiter=%2F&max-keys=1000&prefix=model_repository%2Ftitanic_ensemble_postprocess%2F HTTP/1.1
Host: s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220706T023022Z
Accept-Encoding: gzip

2022/07/06 02:30:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:22 DEBUG : HTTP REQUEST (req 0xc0005cd600)
2022/07/06 02:30:22 DEBUG : GET /MY_PREFIX-us-west-2-MY_ACCOUNT?delimiter=%2F&max-keys=1000&prefix=model_repository%2Ftitanic_ensemble%2F HTTP/1.1
Host: s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220706T023022Z
Accept-Encoding: gzip

2022/07/06 02:30:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:22 DEBUG : HTTP REQUEST (req 0xc000560400)
2022/07/06 02:30:22 DEBUG : GET /MY_PREFIX-us-west-2-MY_ACCOUNT?delimiter=%2F&max-keys=1000&prefix=model_repository%2Fensemble_predict%2F HTTP/1.1
Host: s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220706T023022Z
Accept-Encoding: gzip

2022/07/06 02:30:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:22 DEBUG : HTTP REQUEST (req 0xc00019f900)
2022/07/06 02:30:22 DEBUG : GET /MY_PREFIX-us-west-2-MY_ACCOUNT?delimiter=%2F&max-keys=1000&prefix=model_repository%2Fpytorch_raw%2F HTTP/1.1
Host: s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220706T023022Z
Accept-Encoding: gzip

2022/07/06 02:30:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/06 02:30:23 DEBUG : HTTP RESPONSE (req 0xc0006e2e00)
2022/07/06 02:30:23 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Wed, 06 Jul 2022 02:30:24 GMT
Server: AmazonS3
X-Amz-Bucket-Region: us-west-2
X-Amz-Id-2: OsVRhFMSgfdWmY1hTwlwsAs3AgG4to+qDy/ErAkDFpBBkIgZCEvJ4+mzE/GWLc/WcBBf7oZ5++Y=
X-Amz-Request-Id: RPVJJTNNRWC3Q26Q

319
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>MY_PREFIX-us-west-2-MY_ACCOUNT</Name><Prefix>model_repository/ensemble_preprocess/</Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Contents><Key>model_repository/ensemble_preprocess/config.pbtxt</Key><LastModified>2022-07-05T00:40:57.000Z</LastModified><ETag>&quot;cab20010c4c679f8d34af5a6d67b5849&quot;</ETag><Size>1369</Size><Owner><ID>046a99cdee8130f606cf0579b9bba1b7d2018637882065c31e9acb5c20875fb4</ID><DisplayName>travis</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><CommonPrefixes><Prefix>model_repository/ensemble_preprocess/1/</Prefix></CommonPrefixes></ListBucketResult>
0

2022/07/06 02:30:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/06 02:30:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:23 DEBUG : HTTP REQUEST (req 0xc00074e300)
2022/07/06 02:30:23 DEBUG : GET /MY_PREFIX-us-west-2-MY_ACCOUNT?delimiter=%2F&max-keys=1000&prefix=model_repository%2Ftitanic_ensemble_predict%2F HTTP/1.1
Host: s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220706T023023Z
Accept-Encoding: gzip

2022/07/06 02:30:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:23 DEBUG : HTTP REQUEST (req 0xc0004eab00)
2022/07/06 02:30:23 DEBUG : GET /MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository/ensemble_preprocess/config.pbtxt HTTP/1.1
Host: s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220706T023023Z
Accept-Encoding: gzip

2022/07/06 02:30:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/06 02:30:23 DEBUG : HTTP RESPONSE (req 0xc0005cda00)
2022/07/06 02:30:23 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Wed, 06 Jul 2022 02:30:24 GMT
Server: AmazonS3
X-Amz-Bucket-Region: us-west-2
X-Amz-Id-2: b+uevwHYzInJ66OwHhNtN7xRDcNl07cxT2GGygcFO+ulOd2vJo2bqJO8zESV9QCYPWTF+vi0WME=
X-Amz-Request-Id: RPVZ1EXH1XT8VFBE

312
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>MY_PREFIX-us-west-2-MY_ACCOUNT</Name><Prefix>model_repository/pytorch_processed/</Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Contents><Key>model_repository/pytorch_processed/config.pbtxt</Key><LastModified>2022-07-06T02:22:21.000Z</LastModified><ETag>&quot;591a651680d6bb1f9a287223677e5455&quot;</ETag><Size>936</Size><Owner><ID>046a99cdee8130f606cf0579b9bba1b7d2018637882065c31e9acb5c20875fb4</ID><DisplayName>travis</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><CommonPrefixes><Prefix>model_repository/pytorch_processed/1/</Prefix></CommonPrefixes></ListBucketResult>
0

2022/07/06 02:30:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/06 02:30:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:23 DEBUG : HTTP REQUEST (req 0xc0006e3b00)
2022/07/06 02:30:23 DEBUG : GET /MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository/pytorch_processed/config.pbtxt HTTP/1.1
Host: s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220706T023023Z
Accept-Encoding: gzip

2022/07/06 02:30:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:23 DEBUG : HTTP REQUEST (req 0xc000560b00)
2022/07/06 02:30:23 DEBUG : GET /MY_PREFIX-us-west-2-MY_ACCOUNT?delimiter=%2F&max-keys=1000&prefix=model_repository%2Ftitanic_ensemble_preprocess%2F HTTP/1.1
Host: s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220706T023023Z
Accept-Encoding: gzip

2022/07/06 02:30:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/06 02:30:23 DEBUG : HTTP RESPONSE (req 0xc0005cde00)
2022/07/06 02:30:23 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Wed, 06 Jul 2022 02:30:24 GMT
Server: AmazonS3
X-Amz-Bucket-Region: us-west-2
X-Amz-Id-2: doD6oKRSFfDAuq5nu/SLtStctIBsQ2/kkAh/8w5Oqg3031TzlArBj842hTOPdd+zvvNU+Z64QYs=
X-Amz-Request-Id: RPVMZEW2ECT7XW9A

30a
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>MY_PREFIX-us-west-2-MY_ACCOUNT</Name><Prefix>model_repository/ensemble_entry/</Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Contents><Key>model_repository/ensemble_entry/config.pbtxt</Key><LastModified>2022-07-05T00:40:52.000Z</LastModified><ETag>&quot;9a90bf656b03053115bcbcc8e73bc5c2&quot;</ETag><Size>3580</Size><Owner><ID>046a99cdee8130f606cf0579b9bba1b7d2018637882065c31e9acb5c20875fb4</ID><DisplayName>travis</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><CommonPrefixes><Prefix>model_repository/ensemble_entry/1/</Prefix></CommonPrefixes></ListBucketResult>
0

2022/07/06 02:30:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/06 02:30:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:23 DEBUG : HTTP REQUEST (req 0xc0006e3f00)
2022/07/06 02:30:23 DEBUG : GET /MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository/ensemble_entry/config.pbtxt HTTP/1.1
Host: s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220706T023023Z
Accept-Encoding: gzip

2022/07/06 02:30:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:23 DEBUG : HTTP REQUEST (req 0xc0007e4500)
2022/07/06 02:30:23 DEBUG : GET /MY_PREFIX-us-west-2-MY_ACCOUNT?delimiter=%2F&max-keys=1000&prefix=model_repository%2Fensemble_preprocess%2F1%2F HTTP/1.1
Host: s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220706T023023Z
Accept-Encoding: gzip

2022/07/06 02:30:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/06 02:30:23 DEBUG : HTTP RESPONSE (req 0xc0005cd600)
2022/07/06 02:30:23 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Wed, 06 Jul 2022 02:30:24 GMT
Server: AmazonS3
X-Amz-Bucket-Region: us-west-2
X-Amz-Id-2: +/+0f6El5Kuks5FIkz9AfsO8tNE9DQymDEXrDsUnCLrogHePdk7quijXNpeSSusWYP7y+zYCgdE=
X-Amz-Request-Id: RPVGXY44JJYRPHAQ

310
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>MY_PREFIX-us-west-2-MY_ACCOUNT</Name><Prefix>model_repository/titanic_ensemble/</Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Contents><Key>model_repository/titanic_ensemble/config.pbtxt</Key><LastModified>2022-07-06T02:22:26.000Z</LastModified><ETag>&quot;cbf35447b8e32dda8db30ceb05292937&quot;</ETag><Size>3606</Size><Owner><ID>046a99cdee8130f606cf0579b9bba1b7d2018637882065c31e9acb5c20875fb4</ID><DisplayName>travis</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><CommonPrefixes><Prefix>model_repository/titanic_ensemble/1/</Prefix></CommonPrefixes></ListBucketResult>
0

2022/07/06 02:30:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/06 02:30:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/06 02:30:23 DEBUG : HTTP RESPONSE (req 0xc0004ea600)
2022/07/06 02:30:23 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Wed, 06 Jul 2022 02:30:24 GMT
Server: AmazonS3
X-Amz-Bucket-Region: us-west-2
X-Amz-Id-2: 9tvxjawSPFfP9TpD5sgZjRfavJn7XGSmE5y3wmxefElVbmjrHHh05kVTKlmUhLnzASyrVQU3sOM=
X-Amz-Request-Id: RPVQ9Z8B2G8XNP9X

31b
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>MY_PREFIX-us-west-2-MY_ACCOUNT</Name><Prefix>model_repository/ensemble_postprocess/</Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Contents><Key>model_repository/ensemble_postprocess/config.pbtxt</Key><LastModified>2022-07-05T00:40:55.000Z</LastModified><ETag>&quot;e18309c517ea7d5d59298b3cf406ea39&quot;</ETag><Size>718</Size><Owner><ID>046a99cdee8130f606cf0579b9bba1b7d2018637882065c31e9acb5c20875fb4</ID><DisplayName>travis</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><CommonPrefixes><Prefix>model_repository/ensemble_postprocess/1/</Prefix></CommonPrefixes></ListBucketResult>
0

2022/07/06 02:30:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/06 02:30:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:23 DEBUG : HTTP REQUEST (req 0xc0007e4a00)
2022/07/06 02:30:23 DEBUG : GET /MY_PREFIX-us-west-2-MY_ACCOUNT?delimiter=%2F&max-keys=1000&prefix=model_repository%2Fpytorch_processed%2F1%2F HTTP/1.1
Host: s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220706T023023Z
Accept-Encoding: gzip

2022/07/06 02:30:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:23 DEBUG : HTTP REQUEST (req 0xc0004eb900)
2022/07/06 02:30:23 DEBUG : GET /MY_PREFIX-us-west-2-MY_ACCOUNT?delimiter=%2F&max-keys=1000&prefix=model_repository%2Fensemble_entry%2F1%2F HTTP/1.1
Host: s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220706T023023Z
Accept-Encoding: gzip

2022/07/06 02:30:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/06 02:30:23 DEBUG : HTTP RESPONSE (req 0xc000655500)
2022/07/06 02:30:23 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Wed, 06 Jul 2022 02:30:24 GMT
Server: AmazonS3
X-Amz-Bucket-Region: us-west-2
X-Amz-Id-2: IoD5d4JymaOaKpbUX9vuFqkhodKpLUfj+b/zG+OHECAkC1cWQgRQ9j5fF8muVwYofyqcjO2i5cw=
X-Amz-Request-Id: RPVTFCMNRNN3KHQG

333
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>MY_PREFIX-us-west-2-MY_ACCOUNT</Name><Prefix>model_repository/titanic_ensemble_postprocess/</Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated><Contents><Key>model_repository/titanic_ensemble_postprocess/config.pbtxt</Key><LastModified>2022-07-06T02:22:30.000Z</LastModified><ETag>&quot;b1b416147af1ba8c6183211e58b6e669&quot;</ETag><Size>726</Size><Owner><ID>046a99cdee8130f606cf0579b9bba1b7d2018637882065c31e9acb5c20875fb4</ID><DisplayName>travis</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><CommonPrefixes><Prefix>model_repository/titanic_ensemble_postprocess/1/</Prefix></CommonPrefixes></ListBucketResult>
0

2022/07/06 02:30:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/06 02:30:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:23 DEBUG : HTTP REQUEST (req 0xc00074e700)
2022/07/06 02:30:23 DEBUG : GET /MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository/titanic_ensemble/config.pbtxt HTTP/1.1
Host: s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220706T023023Z
Accept-Encoding: gzip

2022/07/06 02:30:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/06 02:30:23 DEBUG : HTTP RESPONSE (req 0xc0004eab00)
2022/07/06 02:30:23 DEBUG : HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Wed, 06 Jul 2022 02:30:23 GMT
Server: AmazonS3
X-Amz-Id-2: +/GUZQ4QgmIIIVRzmZ9SGzqeZkhO0upFDiyNkX0uxWyS2VHa+D0Sn2ZpJfQ4IbdJ9G5WkpSBByo=
X-Amz-Request-Id: RPVQ7E3GYA2WZT2T

16a
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.</Message><RequestId>RPVQ7E3GYA2WZT2T</RequestId><HostId>+/GUZQ4QgmIIIVRzmZ9SGzqeZkhO0upFDiyNkX0uxWyS2VHa+D0Sn2ZpJfQ4IbdJ9G5WkpSBByo=</HostId></Error>
0

2022/07/06 02:30:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/06 02:30:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/06 02:30:23 DEBUG : HTTP RESPONSE (req 0xc0006e3b00)
2022/07/06 02:30:23 DEBUG : HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Wed, 06 Jul 2022 02:30:23 GMT
Server: AmazonS3
X-Amz-Id-2: GAyhfoKaprh7E08ag5arWdpQDHV6iLM52OZpyoi6g6kXXpiBYXwqX5FhxhBJfskf6whDwpaiwt4=
X-Amz-Request-Id: RPVSC03BPDG74H66

16a
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.</Message><RequestId>RPVSC03BPDG74H66</RequestId><HostId>GAyhfoKaprh7E08ag5arWdpQDHV6iLM52OZpyoi6g6kXXpiBYXwqX5FhxhBJfskf6whDwpaiwt4=</HostId></Error>
0

2022/07/06 02:30:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/06 02:30:23 ERROR : pytorch_processed/config.pbtxt: Failed to copy: failed to open source object: AccessDenied: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.
	status code: 403, request id: RPVSC03BPDG74H66, host id: GAyhfoKaprh7E08ag5arWdpQDHV6iLM52OZpyoi6g6kXXpiBYXwqX5FhxhBJfskf6whDwpaiwt4=
2022/07/06 02:30:23 ERROR : ensemble_preprocess/config.pbtxt: Failed to copy: failed to open source object: AccessDenied: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.
	status code: 403, request id: RPVQ7E3GYA2WZT2T, host id: +/GUZQ4QgmIIIVRzmZ9SGzqeZkhO0upFDiyNkX0uxWyS2VHa+D0Sn2ZpJfQ4IbdJ9G5WkpSBByo=
2022/07/06 02:30:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/06 02:30:23 DEBUG : HTTP REQUEST (req 0xc000b04600)
2022/07/06 02:30:23 DEBUG : GET /MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository/ensemble_postprocess/config.pbtxt HTTP/1.1
Host: s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.57.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220706T023023Z
Accept-Encoding: gzip
7

This is the failing request. It appears to be going to the correct endpoint which is good. It doesn't have an x-amz-server-side-encryption-customer-algorithm header though, but according to the docs

Encryption request headers, like x-amz-server-side-encryption , should not be sent for GET requests if your object uses server-side encryption with KMS keys (SSE-KMS)

I'm not sure what is wrong with the request.

Can you try your aws s3 cp command getting it to dump the HTTP requests and responses?

You can do this by adding the --debug flag and you'll get a line like this. Make sure you redact the auth from it.

2022-07-06 12:07:20,949 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://rclone.s3.eu-west-2.amazonaws.com/?list-type=2&prefix=&delimiter=%2F&encoding-type=url, headers={'User-Agent': b'aws-cli/2.2.44 Python/3.8.8 Linux/5.15.0-37-generic exe/x86_64.ubuntu.22 prompt/off command/s3.ls', 'X-Amz-Date': b'20220706T110720Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': XXX, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=085e0b6f2f6a663260a9f396290b0622c125c9cc40e2bf97a76cb54dbbaef4b1'}>

Then we can see what the difference between the two tools is.

Can you also try this with rclone v1.58.1 and the latest beta just to check to see if it was an SDK bug it might have been fixed?

8

Running aws cp with --debug yeilds the following output.

It looks like its doing an initial HEAD request, possibly to get back metadata so that it can then pass the following header to the GET request 'x-amz-server-side-encryption-bucket-key-enabled': 'true'

2022-07-07 09:08:08,637 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=https://MY_PREFIX-us-west-2-MY_ACCOUNT.s3.ap-southeast-2.amazonaws.com/model_repository/titanic_ensemble/config.pbtxt, headers={'User-Agent': b'aws-cli/2.4.27 Python/3.8.8 Darwin/21.4.0 exe/x86_64 prompt/off command/s3.cp', 'X-Amz-Date': b'20220706T230808Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAW44YBVKOQKIQBFYF/20220706/ap-southeast-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=40df0bd1e72e56bdddcc5857ea642be1fd7b2518299aa93885f765eb01069b52'}>
2022-07-07 09:08:08,638 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/awscli/botocore/cacert.pem
2022-07-07 09:08:08,638 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): MY_PREFIX-us-west-2-MY_ACCOUNT.s3.ap-southeast-2.amazonaws.com:443
2022-07-07 09:08:08,733 - MainThread - urllib3.connectionpool - DEBUG - https://MY_PREFIX-us-west-2-MY_ACCOUNT.s3.ap-southeast-2.amazonaws.com:443 "HEAD /model_repository/titanic_ensemble/config.pbtxt HTTP/1.1" 301 0
2022-07-07 09:08:08,734 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-bucket-region': 'us-west-2', 'x-amz-request-id': 'PET4V4E7TYJGRF3Y', 'x-amz-id-2': 'YwM0GbSgQEz+WkDohVBklkbDPX9VRGAcvlCe6iag/BgwxWaqWmUGrpaThWOGS95ynz1ahvOeAWU=', 'Content-Type': 'application/xml', 'Date': 'Wed, 06 Jul 2022 23:08:08 GMT', 'Server': 'AmazonS3'}
2022-07-07 09:08:08,734 - MainThread - botocore.parsers - DEBUG - Response body:
b''
2022-07-07 09:08:08,735 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.HeadObject: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fd5c0c445e0>>
2022-07-07 09:08:08,736 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2022-07-07 09:08:08,736 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fd5c0c44640>>
2022-07-07 09:08:08,736 - MainThread - botocore.utils - DEBUG - S3 client configured for region ap-southeast-2 but the bucket MY_PREFIX-us-west-2-MY_ACCOUNT is in region us-west-2; Please configure the proper region to avoid multiple unnecessary redirects and signing attempts.
2022-07-07 09:08:08,736 - MainThread - botocore.utils - DEBUG - Updating URI from https://s3.ap-southeast-2.amazonaws.com/MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository/titanic_ensemble/config.pbtxt to https://s3.us-west-2.amazonaws.com/MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository/titanic_ensemble/config.pbtxt
2022-07-07 09:08:08,736 - MainThread - botocore.endpoint - DEBUG - Response received to retry, sleeping for 0 seconds
2022-07-07 09:08:08,736 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.HeadObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fd5a0554a90>>
2022-07-07 09:08:08,736 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.HeadObject: calling handler <bound method S3EndpointSetter.set_signer of <botocore.utils.S3EndpointSetter object at 0x7fd5c0c44790>>
2022-07-07 09:08:08,736 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.HeadObject: calling handler <function set_operation_specific_signer at 0x7fd5a019b940>
2022-07-07 09:08:08,736 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.HeadObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7fd5c0c44790>>
2022-07-07 09:08:08,736 - MainThread - botocore.utils - DEBUG - Checking for DNS compatible bucket for: https://s3.us-west-2.amazonaws.com/MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository/titanic_ensemble/config.pbtxt
2022-07-07 09:08:08,736 - MainThread - botocore.utils - DEBUG - URI updated to: https://MY_PREFIX-us-west-2-MY_ACCOUNT.s3.us-west-2.amazonaws.com/model_repository/titanic_ensemble/config.pbtxt
2022-07-07 09:08:08,736 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2022-07-07 09:08:08,736 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
HEAD
/model_repository/titanic_ensemble/config.pbtxt

host:MY_PREFIX-us-west-2-MY_ACCOUNT.s3.us-west-2.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220706T230808Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2022-07-07 09:08:08,736 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20220706T230808Z
20220706/us-west-2/s3/aws4_request
cfb3aa4f1ad58518d11bf351e30059845b4e7fb666e886e2acf2e13234c25940
2022-07-07 09:08:08,737 - MainThread - botocore.auth - DEBUG - Signature:
47d8235f1b50083780b17a1f4b0886782178a5fc9b2b54110dc427c51faad5fb
2022-07-07 09:08:08,737 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=https://MY_PREFIX-us-west-2-MY_ACCOUNT.s3.us-west-2.amazonaws.com/model_repository/titanic_ensemble/config.pbtxt, headers={'User-Agent': b'aws-cli/2.4.27 Python/3.8.8 Darwin/21.4.0 exe/x86_64 prompt/off command/s3.cp', 'X-Amz-Date': b'20220706T230808Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAW44YBVKOQKIQBFYF/20220706/us-west-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=47d8235f1b50083780b17a1f4b0886782178a5fc9b2b54110dc427c51faad5fb'}>
2022-07-07 09:08:08,737 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/awscli/botocore/cacert.pem
2022-07-07 09:08:08,737 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): MY_PREFIX-us-west-2-MY_ACCOUNT.s3.us-west-2.amazonaws.com:443
2022-07-07 09:08:09,608 - MainThread - urllib3.connectionpool - DEBUG - https://MY_PREFIX-us-west-2-MY_ACCOUNT.s3.us-west-2.amazonaws.com:443 "HEAD /model_repository/titanic_ensemble/config.pbtxt HTTP/1.1" 200 0
2022-07-07 09:08:09,609 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': '6LA8592CjGgNwn1MrEp+zftcZN7peifi7SXZP8h4lkh8eIa8dpxK64WKThSOoVfiUn0C2Zcjhy4=', 'x-amz-request-id': '07WEEPAZ4DBJ2AJD', 'Date': 'Wed, 06 Jul 2022 23:08:10 GMT', 'Last-Modified': 'Wed, 06 Jul 2022 02:22:26 GMT', 'ETag': '"cbf35447b8e32dda8db30ceb05292937"', 'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'arn:aws:kms:us-west-2:MY_ACCOUNT:key/f6fbd5c7-ab1b-4eb6-a769-ea789c24125f', 'x-amz-server-side-encryption-bucket-key-enabled': 'true', 'x-amz-version-id': 'zmWsxJtj5EEQmrb8xr2bzzC.IRT4OgkR', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'Server': 'AmazonS3', 'Content-Length': '3606'}
2022-07-07 09:08:09,609 - MainThread - botocore.parsers - DEBUG - Response body:
b''
2022-07-07 09:08:09,611 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.HeadObject: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fd5c0c445e0>>
2022-07-07 09:08:09,611 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2022-07-07 09:08:09,611 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fd5c0c44640>>
2022-07-07 09:08:09,611 - MainThread - botocore.utils - DEBUG - S3 request was previously redirected, not redirecting.
2022-07-07 09:08:09,611 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.HeadObject: calling handler <function enhance_error_msg at 0x7fd5c0a188b0>
2022-07-07 09:08:09,612 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.HeadObject: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7fd5c0c44130>>
2022-07-07 09:08:09,613 - MainThread - s3transfer.utils - DEBUG - Acquiring 0
2022-07-07 09:08:09,613 - ThreadPoolExecutor-1_0 - s3transfer.tasks - DEBUG - DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7fd5a05ff910>}) about to wait for the following futures []
2022-07-07 09:08:09,613 - ThreadPoolExecutor-1_0 - s3transfer.tasks - DEBUG - DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7fd5a05ff910>}) done waiting for dependent futures
2022-07-07 09:08:09,613 - ThreadPoolExecutor-1_0 - s3transfer.tasks - DEBUG - Executing task DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7fd5a05ff910>}) with kwargs {'client': <botocore.client.S3 object at 0x7fd5a0570490>, 'config': <s3transfer.manager.TransferConfig object at 0x7fd5a05ae220>, 'osutil': <s3transfer.utils.OSUtils object at 0x7fd5a05ae2b0>, 'request_executor': <s3transfer.futures.BoundedExecutor object at 0x7fd5a05ae4f0>, 'transfer_future': <s3transfer.futures.TransferFuture object at 0x7fd5a05ff910>, 'io_executor': <s3transfer.futures.BoundedExecutor object at 0x7fd5a05aeb50>}
2022-07-07 09:08:09,613 - ThreadPoolExecutor-1_0 - s3transfer.futures - DEBUG - Submitting task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'MY_PREFIX-us-west-2-MY_ACCOUNT', 'key': 'model_repository/titanic_ensemble/config.pbtxt', 'extra_args': {}}) to executor <s3transfer.futures.BoundedExecutor object at 0x7fd5a05ae4f0> for transfer request: 0.
2022-07-07 09:08:09,614 - ThreadPoolExecutor-1_0 - s3transfer.utils - DEBUG - Acquiring 0
2022-07-07 09:08:09,614 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'MY_PREFIX-us-west-2-MY_ACCOUNT', 'key': 'model_repository/titanic_ensemble/config.pbtxt', 'extra_args': {}}) about to wait for the following futures []
2022-07-07 09:08:09,614 - ThreadPoolExecutor-1_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2022-07-07 09:08:09,614 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'MY_PREFIX-us-west-2-MY_ACCOUNT', 'key': 'model_repository/titanic_ensemble/config.pbtxt', 'extra_args': {}}) done waiting for dependent futures
2022-07-07 09:08:09,614 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - Executing task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'MY_PREFIX-us-west-2-MY_ACCOUNT', 'key': 'model_repository/titanic_ensemble/config.pbtxt', 'extra_args': {}}) with kwargs {'client': <botocore.client.S3 object at 0x7fd5a0570490>, 'bucket': 'MY_PREFIX-us-west-2-MY_ACCOUNT', 'key': 'model_repository/titanic_ensemble/config.pbtxt', 'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7fd5a05ffbe0>, 'extra_args': {}, 'callbacks': [functools.partial(<bound method BaseSubscriber.on_progress of <awscli.customizations.s3.subscribers.ProvideSizeSubscriber object at 0x7fd5a05cbfa0>>, future=<s3transfer.futures.TransferFuture object at 0x7fd5a05ff910>), functools.partial(<bound method BaseSubscriber.on_progress of <awscli.customizations.s3.results.QueuedResultSubscriber object at 0x7fd5a05cb970>>, future=<s3transfer.futures.TransferFuture object at 0x7fd5a05ff910>), functools.partial(<bound method BaseSubscriber.on_progress of <awscli.customizations.s3.subscribers.DirectoryCreatorSubscriber object at 0x7fd5a05cbbe0>>, future=<s3transfer.futures.TransferFuture object at 0x7fd5a05ff910>), functools.partial(<bound method BaseSubscriber.on_progress of <awscli.customizations.s3.subscribers.ProvideLastModifiedTimeSubscriber object at 0x7fd5a05cb760>>, future=<s3transfer.futures.TransferFuture object at 0x7fd5a05ff910>), functools.partial(<bound method ProgressResultSubscriber.on_progress of <awscli.customizations.s3.results.ProgressResultSubscriber object at 0x7fd5a05cbfd0>>, future=<s3transfer.futures.TransferFuture object at 0x7fd5a05ff910>), functools.partial(<bound method BaseSubscriber.on_progress of <awscli.customizations.s3.results.DoneResultSubscriber object at 0x7fd5a05cbeb0>>, future=<s3transfer.futures.TransferFuture object at 0x7fd5a05ff910>)], 'max_attempts': 5, 'download_output_manager': <s3transfer.download.DownloadFilenameOutputManager object at 0x7fd5a05ffc10>, 'io_chunksize': 262144, 'bandwidth_limiter': None}
2022-07-07 09:08:09,614 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event provide-client-params.s3.GetObject: calling handler <function base64_decode_input_blobs at 0x7fd5c0a8b940>
2022-07-07 09:08:09,614 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-parameter-build.s3.GetObject: calling handler <function sse_md5 at 0x7fd5a019bca0>
2022-07-07 09:08:09,614 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-parameter-build.s3.GetObject: calling handler <function validate_bucket_name at 0x7fd5a019bc10>
2022-07-07 09:08:09,614 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-parameter-build.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7fd5a05ae070>>
2022-07-07 09:08:09,614 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-parameter-build.s3.GetObject: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x7fd5a05ae130>>
2022-07-07 09:08:09,614 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-parameter-build.s3.GetObject: calling handler <function generate_idempotent_uuid at 0x7fd5a019ba60>
2022-07-07 09:08:09,615 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-call.s3.GetObject: calling handler <function add_expect_header at 0x7fd5a019bf70>
2022-07-07 09:08:09,615 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-call.s3.GetObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7fd5a05ae070>>
2022-07-07 09:08:09,615 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-call.s3.GetObject: calling handler <function inject_api_version_header_if_needed at 0x7fd5a01a1310>
2022-07-07 09:08:09,615 - ThreadPoolExecutor-0_0 - botocore.endpoint - DEBUG - Making request for OperationModel(name=GetObject) with params: {'url_path': '/MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository/titanic_ensemble/config.pbtxt', 'query_string': {}, 'method': 'GET', 'headers': {'User-Agent': 'aws-cli/2.4.27 Python/3.8.8 Darwin/21.4.0 exe/x86_64 prompt/off command/s3.cp'}, 'body': b'', 'url': 'https://s3.ap-southeast-2.amazonaws.com/MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository/titanic_ensemble/config.pbtxt', 'context': {'client_region': 'ap-southeast-2', 'client_config': <botocore.config.Config object at 0x7fd5a0570550>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'MY_PREFIX-us-west-2-MY_ACCOUNT'}}}
2022-07-07 09:08:09,615 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event request-created.s3.GetObject: calling handler <function signal_not_transferring at 0x7fd5c07c2040>
2022-07-07 09:08:09,615 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event request-created.s3.GetObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fd5a05704f0>>
2022-07-07 09:08:09,615 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event choose-signer.s3.GetObject: calling handler <bound method S3EndpointSetter.set_signer of <botocore.utils.S3EndpointSetter object at 0x7fd5a05ae1c0>>
2022-07-07 09:08:09,615 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event choose-signer.s3.GetObject: calling handler <function set_operation_specific_signer at 0x7fd5a019b940>
2022-07-07 09:08:09,615 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-sign.s3.GetObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7fd5a05ae1c0>>
2022-07-07 09:08:09,615 - ThreadPoolExecutor-0_0 - botocore.utils - DEBUG - Defaulting to S3 virtual host style addressing with path style addressing fallback.
2022-07-07 09:08:09,615 - ThreadPoolExecutor-0_0 - botocore.utils - DEBUG - Checking for DNS compatible bucket for: https://s3.ap-southeast-2.amazonaws.com/MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository/titanic_ensemble/config.pbtxt
2022-07-07 09:08:09,615 - ThreadPoolExecutor-0_0 - botocore.utils - DEBUG - URI updated to: https://MY_PREFIX-us-west-2-MY_ACCOUNT.s3.ap-southeast-2.amazonaws.com/model_repository/titanic_ensemble/config.pbtxt
2022-07-07 09:08:09,616 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - Calculating signature using v4 auth.
2022-07-07 09:08:09,616 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - CanonicalRequest:
GET
/model_repository/titanic_ensemble/config.pbtxt

host:MY_PREFIX-us-west-2-MY_ACCOUNT.s3.ap-southeast-2.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220706T230809Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2022-07-07 09:08:09,616 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20220706T230809Z
20220706/ap-southeast-2/s3/aws4_request
f7f8edb844f1643060e430ee64539e89d8abcd74ac791b5cb869167242b731ec
2022-07-07 09:08:09,616 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - Signature:
f6856f437d21e2b4a7440d811f653b469fe31190f21f9f66024e7116d2d77013
2022-07-07 09:08:09,616 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event request-created.s3.GetObject: calling handler <function signal_transferring at 0x7fd5c07c20d0>
2022-07-07 09:08:09,616 - ThreadPoolExecutor-0_0 - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=True, method=GET, url=https://MY_PREFIX-us-west-2-MY_ACCOUNT.s3.ap-southeast-2.amazonaws.com/model_repository/titanic_ensemble/config.pbtxt, headers={'User-Agent': b'aws-cli/2.4.27 Python/3.8.8 Darwin/21.4.0 exe/x86_64 prompt/off command/s3.cp', 'X-Amz-Date': b'20220706T230809Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAW44YBVKOQKIQBFYF/20220706/ap-southeast-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=f6856f437d21e2b4a7440d811f653b469fe31190f21f9f66024e7116d2d77013'}>
2022-07-07 09:08:09,616 - ThreadPoolExecutor-0_0 - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/awscli/botocore/cacert.pem
2022-07-07 09:08:09,616 - ThreadPoolExecutor-0_0 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): MY_PREFIX-us-west-2-MY_ACCOUNT.s3.ap-southeast-2.amazonaws.com:443
2022-07-07 09:08:09,709 - ThreadPoolExecutor-0_0 - urllib3.connectionpool - DEBUG - https://MY_PREFIX-us-west-2-MY_ACCOUNT.s3.ap-southeast-2.amazonaws.com:443 "GET /model_repository/titanic_ensemble/config.pbtxt HTTP/1.1" 301 None
2022-07-07 09:08:09,710 - ThreadPoolExecutor-0_0 - botocore.parsers - DEBUG - Response headers: {'x-amz-bucket-region': 'us-west-2', 'x-amz-request-id': '07WCTBJFKMZ49VR3', 'x-amz-id-2': 'joU1uuWInUK7A38M6GZuw9LqYelPjar/mZdc9RTRNzzOb7L0jjyqCfUSNLPsjZD2RyowqRXXbos=', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Wed, 06 Jul 2022 23:08:09 GMT', 'Server': 'AmazonS3'}
2022-07-07 09:08:09,710 - ThreadPoolExecutor-0_0 - botocore.parsers - DEBUG - Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><Endpoint>MY_PREFIX-us-west-2-MY_ACCOUNT.s3-us-west-2.amazonaws.com</Endpoint><Bucket>MY_PREFIX-us-west-2-MY_ACCOUNT</Bucket><RequestId>07WCTBJFKMZ49VR3</RequestId><HostId>joU1uuWInUK7A38M6GZuw9LqYelPjar/mZdc9RTRNzzOb7L0jjyqCfUSNLPsjZD2RyowqRXXbos=</HostId></Error>'
2022-07-07 09:08:09,714 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event needs-retry.s3.GetObject: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fd5a0570fd0>>
2022-07-07 09:08:09,715 - ThreadPoolExecutor-0_0 - botocore.retries.standard - DEBUG - Not retrying request.
2022-07-07 09:08:09,715 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event needs-retry.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fd5a05ae070>>
2022-07-07 09:08:09,715 - ThreadPoolExecutor-0_0 - botocore.utils - DEBUG - S3 client configured for region ap-southeast-2 but the bucket MY_PREFIX-us-west-2-MY_ACCOUNT is in region us-west-2; Please configure the proper region to avoid multiple unnecessary redirects and signing attempts.
2022-07-07 09:08:09,715 - ThreadPoolExecutor-0_0 - botocore.utils - DEBUG - Updating URI from https://s3.ap-southeast-2.amazonaws.com/MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository/titanic_ensemble/config.pbtxt to https://s3.us-west-2.amazonaws.com/MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository/titanic_ensemble/config.pbtxt
2022-07-07 09:08:09,715 - ThreadPoolExecutor-0_0 - botocore.endpoint - DEBUG - Response received to retry, sleeping for 0 seconds
2022-07-07 09:08:09,715 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event request-created.s3.GetObject: calling handler <function signal_not_transferring at 0x7fd5c07c2040>
2022-07-07 09:08:09,715 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event request-created.s3.GetObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fd5a05704f0>>
2022-07-07 09:08:09,715 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event choose-signer.s3.GetObject: calling handler <bound method S3EndpointSetter.set_signer of <botocore.utils.S3EndpointSetter object at 0x7fd5a05ae1c0>>
2022-07-07 09:08:09,715 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event choose-signer.s3.GetObject: calling handler <function set_operation_specific_signer at 0x7fd5a019b940>
2022-07-07 09:08:09,715 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-sign.s3.GetObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7fd5a05ae1c0>>
2022-07-07 09:08:09,715 - ThreadPoolExecutor-0_0 - botocore.utils - DEBUG - Checking for DNS compatible bucket for: https://s3.us-west-2.amazonaws.com/MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository/titanic_ensemble/config.pbtxt
2022-07-07 09:08:09,715 - ThreadPoolExecutor-0_0 - botocore.utils - DEBUG - URI updated to: https://MY_PREFIX-us-west-2-MY_ACCOUNT.s3.us-west-2.amazonaws.com/model_repository/titanic_ensemble/config.pbtxt
2022-07-07 09:08:09,715 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - Calculating signature using v4 auth.
2022-07-07 09:08:09,715 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - CanonicalRequest:
GET
/model_repository/titanic_ensemble/config.pbtxt

host:MY_PREFIX-us-west-2-MY_ACCOUNT.s3.us-west-2.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220706T230809Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2022-07-07 09:08:09,715 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20220706T230809Z
20220706/us-west-2/s3/aws4_request
7115821a0bf033fa78a814043e4ab3bbad46d5ee0c316066d423e50f33648fbd
2022-07-07 09:08:09,715 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - Signature:
a20027b0d0e0bf823845a2a5eec75a45e0507d587a12acc063eee21cc960fc6e
2022-07-07 09:08:09,715 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event request-created.s3.GetObject: calling handler <function signal_transferring at 0x7fd5c07c20d0>
2022-07-07 09:08:09,715 - ThreadPoolExecutor-0_0 - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=True, method=GET, url=https://MY_PREFIX-us-west-2-MY_ACCOUNT.s3.us-west-2.amazonaws.com/model_repository/titanic_ensemble/config.pbtxt, headers={'User-Agent': b'aws-cli/2.4.27 Python/3.8.8 Darwin/21.4.0 exe/x86_64 prompt/off command/s3.cp', 'X-Amz-Date': b'20220706T230809Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAW44YBVKOQKIQBFYF/20220706/us-west-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=a20027b0d0e0bf823845a2a5eec75a45e0507d587a12acc063eee21cc960fc6e'}>
2022-07-07 09:08:09,716 - ThreadPoolExecutor-0_0 - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/awscli/botocore/cacert.pem
2022-07-07 09:08:09,716 - ThreadPoolExecutor-0_0 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): MY_PREFIX-us-west-2-MY_ACCOUNT.s3.us-west-2.amazonaws.com:443
2022-07-07 09:08:10,567 - ThreadPoolExecutor-0_0 - urllib3.connectionpool - DEBUG - https://MY_PREFIX-us-west-2-MY_ACCOUNT.s3.us-west-2.amazonaws.com:443 "GET /model_repository/titanic_ensemble/config.pbtxt HTTP/1.1" 200 3606
2022-07-07 09:08:10,568 - ThreadPoolExecutor-0_0 - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': 'x6u4qIJFpJW9xi+FdhAYv6ICvZapESuDM7AEIxa5LDAkXHDp1cGdrVXNIN2wWankNSsj4M/BwUo=', 'x-amz-request-id': 'CPDYP2DV495GW90F', 'Date': 'Wed, 06 Jul 2022 23:08:11 GMT', 'Last-Modified': 'Wed, 06 Jul 2022 02:22:26 GMT', 'ETag': '"cbf35447b8e32dda8db30ceb05292937"', 'x-amz-server-side-encryption': 'aws:kms', 'x-amz-server-side-encryption-aws-kms-key-id': 'arn:aws:kms:us-west-2:MY_ACCOUNT:key/f6fbd5c7-ab1b-4eb6-a769-ea789c24125f', 'x-amz-server-side-encryption-bucket-key-enabled': 'true', 'x-amz-version-id': 'zmWsxJtj5EEQmrb8xr2bzzC.IRT4OgkR', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'Server': 'AmazonS3', 'Content-Length': '3606'}
2022-07-07 09:08:10,569 - ThreadPoolExecutor-0_0 - botocore.parsers - DEBUG - Response body:
<botocore.response.StreamingBody object at 0x7fd5b0888040>
2022-07-07 09:08:10,570 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event needs-retry.s3.GetObject: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fd5a0570fd0>>
2022-07-07 09:08:10,570 - ThreadPoolExecutor-0_0 - botocore.retries.standard - DEBUG - Not retrying request.
2022-07-07 09:08:10,570 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event needs-retry.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fd5a05ae070>>
2022-07-07 09:08:10,571 - ThreadPoolExecutor-0_0 - botocore.utils - DEBUG - S3 request was previously redirected, not redirecting.
2022-07-07 09:08:10,571 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event after-call.s3.GetObject: calling handler <function enhance_error_msg at 0x7fd5c0a188b0>
2022-07-07 09:08:10,571 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event after-call.s3.GetObject: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7fd5a0570b20>>
2022-07-07 09:08:10,571 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 0}) about to wait for the following futures []
2022-07-07 09:08:10,571 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 0}) done waiting for dependent futures
2022-07-07 09:08:10,572 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - Executing task IOWriteTask(transfer_id=0, {'offset': 0}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7fd5a05ffbe0>, 'offset': 0}
2022-07-07 09:08:10,572 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - IORenameFileTask(transfer_id=0, {'final_filename': '/private/tmp/test.txt'}) about to wait for the following futures []
2022-07-07 09:08:10,572 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - IORenameFileTask(transfer_id=0, {'final_filename': '/private/tmp/test.txt'}) done waiting for dependent futures
2022-07-07 09:08:10,572 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - Executing task IORenameFileTask(transfer_id=0, {'final_filename': '/private/tmp/test.txt'}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7fd5a05ffbe0>, 'final_filename': '/private/tmp/test.txt', 'osutil': <s3transfer.utils.OSUtils object at 0x7fd5a05ae2b0>}
2022-07-07 09:08:10,573 - ThreadPoolExecutor-0_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
download: s3://MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository/titanic_ensemble/config.pbtxt to ./test.txt
2022-07-07 09:08:10,573 - Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.

I tested with latest rclone v1.58.1 locally and still got the same error

9

I'm not sure that header is being sent by the aws tool though...

Here is the final GET which is the successful one

2022-07-07 09:08:09,715 - ThreadPoolExecutor-0_0 - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=True, method=GET, url=https://MY_PREFIX-us-west-2-MY_ACCOUNT.s3.us-west-2.amazonaws.com/model_repository/titanic_ensemble/config.pbtxt, headers={'User-Agent': b'aws-cli/2.4.27 Python/3.8.8 Darwin/21.4.0 exe/x86_64 prompt/off command/s3.cp', 'X-Amz-Date': b'20220706T230809Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAW44YBVKOQKIQBFYF/20220706/us-west-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=a20027b0d0e0bf823845a2a5eec75a45e0507d587a12acc063eee21cc960fc6e'}>

With a few newlines it looks like this, so no unusual headers at all.

2022-07-07 09:08:09,715 - ThreadPoolExecutor-0_0 - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=True,
 method=GET,
 url=https://MY_PREFIX-us-west-2-MY_ACCOUNT.s3.us-west-2.amazonaws.com/model_repository/titanic_ensemble/config.pbtxt,
 headers={
    'User-Agent': b'aws-cli/2.4.27 Python/3.8.8 Darwin/21.4.0 exe/x86_64 prompt/off command/s3.cp',
    'X-Amz-Date': b'20220706T230809Z',
    'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855',
    'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAW44YBVKOQKIQBFYF/20220706/us-west-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=a20027b0d0e0bf823845a2a5eec75a45e0507d587a12acc063eee21cc960fc6e'
}>

The only real difference beween that and the rclone one is the fact that it is using the bucket host directly and that is because you are missing RCLONE_CONFIG_S3_PROVIDER = AWS from your config.

Unless you tell rclone it is using AWS it will fall back to a maximally compatible subset of the S3 protocol and maybe this is an edge case where it doesn't work.

Give that a go and see if it helps.

10

Hi Nick, sorry to say but that didn't seem to fix it. I can see that setting is being set, but still the error

$ rclone -vv --dump bodies  copy s3:MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository/titanic_ensemble/config.pbtxt .
2022/07/07 20:11:46 DEBUG : Setting default for s3-endpoint="s3.us-west-2.amazonaws.com" from environment variable RCLONE_S3_ENDPOINT
2022/07/07 20:11:46 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "-vv" "--dump" "bodies" "copy" "s3:MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository/titanic_ensemble/config.pbtxt" "."]
2022/07/07 20:11:46 DEBUG : Creating backend with remote "s3:MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository/titanic_ensemble/config.pbtxt"
2022/07/07 20:11:46 DEBUG : Setting type="s3" for "s3" from environment variable RCLONE_CONFIG_S3_TYPE
2022/07/07 20:11:46 DEBUG : Setting provider="AWS" for "s3" from environment variable RCLONE_CONFIG_S3_PROVIDER
2022/07/07 20:11:46 DEBUG : Setting access_key_id="MY_ACCES_KEY" for "s3" from environment variable RCLONE_CONFIG_S3_ACCESS_KEY_ID
2022/07/07 20:11:46 DEBUG : Setting secret_access_key="MY_SECRET_KEY" for "s3" from environment variable RCLONE_CONFIG_S3_SECRET_ACCESS_KEY
2022/07/07 20:11:46 DEBUG : Setting region="us-west-2" for "s3" from environment variable RCLONE_CONFIG_S3_REGION
2022/07/07 20:11:46 DEBUG : Setting region="us-west-2" for "s3" from environment variable RCLONE_CONFIG_S3_REGION
2022/07/07 20:11:46 DEBUG : Setting region="us-west-2" for "s3" from environment variable RCLONE_CONFIG_S3_REGION
2022/07/07 20:11:46 DEBUG : Setting region="us-west-2" for "s3" from environment variable RCLONE_CONFIG_S3_REGION
2022/07/07 20:11:46 DEBUG : Setting s3_endpoint="s3.us-west-2.amazonaws.com" from environment variable RCLONE_S3_ENDPOINT
2022/07/07 20:11:46 DEBUG : Setting s3_endpoint="s3.us-west-2.amazonaws.com" from environment variable RCLONE_S3_ENDPOINT
2022/07/07 20:11:46 DEBUG : Setting s3_endpoint="s3.us-west-2.amazonaws.com" from environment variable RCLONE_S3_ENDPOINT
2022/07/07 20:11:46 DEBUG : Setting s3_endpoint="s3.us-west-2.amazonaws.com" from environment variable RCLONE_S3_ENDPOINT
2022/07/07 20:11:46 DEBUG : Setting s3_endpoint="s3.us-west-2.amazonaws.com" from environment variable RCLONE_S3_ENDPOINT
2022/07/07 20:11:46 DEBUG : Setting s3_endpoint="s3.us-west-2.amazonaws.com" from environment variable RCLONE_S3_ENDPOINT
2022/07/07 20:11:46 DEBUG : Setting s3_endpoint="s3.us-west-2.amazonaws.com" from environment variable RCLONE_S3_ENDPOINT
2022/07/07 20:11:46 DEBUG : Setting s3_endpoint="s3.us-west-2.amazonaws.com" from environment variable RCLONE_S3_ENDPOINT
2022/07/07 20:11:46 DEBUG : Setting s3_endpoint="s3.us-west-2.amazonaws.com" from environment variable RCLONE_S3_ENDPOINT
2022/07/07 20:11:46 DEBUG : Setting s3_endpoint="s3.us-west-2.amazonaws.com" from environment variable RCLONE_S3_ENDPOINT
2022/07/07 20:11:46 DEBUG : s3: detected overridden config - adding "{D8fIO}" suffix to name
2022/07/07 20:11:46 DEBUG : Setting provider="AWS" for "s3" from environment variable RCLONE_CONFIG_S3_PROVIDER
2022/07/07 20:11:46 NOTICE: Config file "/Users/julian/.config/rclone/rclone.conf" not found - using defaults
2022/07/07 20:11:46 DEBUG : Setting access_key_id="MY_ACCES_KEY" for "s3" from environment variable RCLONE_CONFIG_S3_ACCESS_KEY_ID
2022/07/07 20:11:46 DEBUG : Setting secret_access_key="MY_SECRET_KEY" for "s3" from environment variable RCLONE_CONFIG_S3_SECRET_ACCESS_KEY
2022/07/07 20:11:46 DEBUG : Setting region="us-west-2" for "s3" from environment variable RCLONE_CONFIG_S3_REGION
2022/07/07 20:11:46 DEBUG : Setting s3_endpoint="s3.us-west-2.amazonaws.com" from environment variable RCLONE_S3_ENDPOINT
2022/07/07 20:11:46 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/07/07 20:11:46 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/07/07 20:11:46 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/07 20:11:46 DEBUG : HTTP REQUEST (req 0x14000531f00)
2022/07/07 20:11:46 DEBUG : HEAD /model_repository/titanic_ensemble/config.pbtxt HTTP/1.1
Host: MY_PREFIX-us-west-2-MY_ACCOUNT.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.58.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220707T101146Z

2022/07/07 20:11:46 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/07 20:11:47 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/07 20:11:47 DEBUG : HTTP RESPONSE (req 0x14000531f00)
2022/07/07 20:11:47 DEBUG : HTTP/1.1 200 OK
Content-Length: 3606
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Date: Thu, 07 Jul 2022 10:11:48 GMT
Etag: "cbf35447b8e32dda8db30ceb05292937"
Last-Modified: Wed, 06 Jul 2022 02:22:26 GMT
Server: AmazonS3
X-Amz-Id-2: bXvxU2frlaYDd2XyZqsiXkeTGUGgCHFUbA75LzB4obdnw65/AQX4vuAFe+NZaIeC4qjhv537AoM=
X-Amz-Request-Id: D6F373B8ZAB9GA9Z
X-Amz-Server-Side-Encryption: aws:kms
X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id: arn:aws:kms:us-west-2:MY_ACCOUNT:key/f6fbd5c7-ab1b-4eb6-a769-ea789c24125f
X-Amz-Server-Side-Encryption-Bucket-Key-Enabled: true
X-Amz-Version-Id: zmWsxJtj5EEQmrb8xr2bzzC.IRT4OgkR

2022/07/07 20:11:47 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/07 20:11:47 DEBUG : fs cache: adding new entry for parent of "s3:MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository/titanic_ensemble/config.pbtxt", "s3{D8fIO}:MY_PREFIX-us-west-2-MY_ACCOUNT/model_repository/titanic_ensemble"
2022/07/07 20:11:47 DEBUG : Creating backend with remote "."
2022/07/07 20:11:47 DEBUG : fs cache: renaming cache item "." to be canonical "/tmp"
2022/07/07 20:11:47 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/07 20:11:47 DEBUG : HTTP REQUEST (req 0x14000152200)
2022/07/07 20:11:47 DEBUG : HEAD /model_repository/titanic_ensemble/config.pbtxt HTTP/1.1
Host: MY_PREFIX-us-west-2-MY_ACCOUNT.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.58.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220707T101147Z

2022/07/07 20:11:47 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/07 20:11:47 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/07 20:11:47 DEBUG : HTTP RESPONSE (req 0x14000152200)
2022/07/07 20:11:47 DEBUG : HTTP/1.1 200 OK
Content-Length: 3606
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Date: Thu, 07 Jul 2022 10:11:48 GMT
Etag: "cbf35447b8e32dda8db30ceb05292937"
Last-Modified: Wed, 06 Jul 2022 02:22:26 GMT
Server: AmazonS3
X-Amz-Id-2: n4akjWLibpVoOdc8y8mDjZaveaTB2op4zH387LY/m6FpIlZWH8s/Nfag2AKN01gHEjjxEt++4DQ=
X-Amz-Request-Id: D6FBWKCG8H2J877K
X-Amz-Server-Side-Encryption: aws:kms
X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id: arn:aws:kms:us-west-2:MY_ACCOUNT:key/f6fbd5c7-ab1b-4eb6-a769-ea789c24125f
X-Amz-Server-Side-Encryption-Bucket-Key-Enabled: true
X-Amz-Version-Id: zmWsxJtj5EEQmrb8xr2bzzC.IRT4OgkR

2022/07/07 20:11:47 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/07 20:11:47 DEBUG : config.pbtxt: Need to transfer - File not found at Destination
2022/07/07 20:11:47 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/07 20:11:47 DEBUG : HTTP REQUEST (req 0x1400088e200)
2022/07/07 20:11:47 DEBUG : GET /model_repository/titanic_ensemble/config.pbtxt HTTP/1.1
Host: MY_PREFIX-us-west-2-MY_ACCOUNT.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.58.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220707T101147Z
Accept-Encoding: gzip

2022/07/07 20:11:47 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/07 20:11:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/07 20:11:48 DEBUG : HTTP RESPONSE (req 0x1400088e200)
2022/07/07 20:11:48 DEBUG : HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 07 Jul 2022 10:11:47 GMT
Server: AmazonS3
X-Amz-Id-2: cQ/4wFWq8/ZWPzX64SWuAwGIx/i+92OKRa7Tk53WGt14jKwn0OImRvJuZ/c5SaVA/JbZDRhtUI8=
X-Amz-Request-Id: 80GE547K73RMG2EH

16a
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.</Message><RequestId>80GE547K73RMG2EH</RequestId><HostId>cQ/4wFWq8/ZWPzX64SWuAwGIx/i+92OKRa7Tk53WGt14jKwn0OImRvJuZ/c5SaVA/JbZDRhtUI8=</HostId></Error>
0

2022/07/07 20:11:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/07 20:11:48 ERROR : config.pbtxt: Failed to copy: failed to open source object: AccessDenied: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.

Also including my version so you know it the latest code:

rclone v1.58.1
- os/version: darwin 12.3.1 (64 bit)
- os/kernel: 21.4.0 (arm64)
- os/type: darwin
- os/arch: arm64
- go/version: go1.18.1
- go/linking: dynamic
- go/tags: none
11

I just wanted to share the us-east-1 region rclone call that does work:

$RCLONE_CONFIG_S3_REGION=us-east-1  rclone -vv --dump bodies  copy s3:MY_PREFIX-us-east-1/model_repository/titanic_ensemble/config.pbtxt test.txt

2022/07/07 20:36:03 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "-vv" "--dump" "bodies" "copy" "s3:MY_PREFIX-us-east-1/model_repository/titanic_ensemble/config.pbtxt" "test.txt"]
2022/07/07 20:36:03 DEBUG : Creating backend with remote "s3:MY_PREFIX-us-east-1/model_repository/titanic_ensemble/config.pbtxt"
2022/07/07 20:36:03 DEBUG : Setting type="s3" for "s3" from environment variable RCLONE_CONFIG_S3_TYPE
2022/07/07 20:36:03 DEBUG : Setting provider="AWS" for "s3" from environment variable RCLONE_CONFIG_S3_PROVIDER
2022/07/07 20:36:03 DEBUG : Setting access_key_id="MY_ACCES_KEY" for "s3" from environment variable RCLONE_CONFIG_S3_ACCESS_KEY_ID
2022/07/07 20:36:03 DEBUG : Setting secret_access_key="MY_SECRET_KEY" for "s3" from environment variable RCLONE_CONFIG_S3_SECRET_ACCESS_KEY
2022/07/07 20:36:03 DEBUG : Setting region="us-east-1" for "s3" from environment variable RCLONE_CONFIG_S3_REGION
2022/07/07 20:36:03 DEBUG : Setting region="us-east-1" for "s3" from environment variable RCLONE_CONFIG_S3_REGION
2022/07/07 20:36:03 DEBUG : Setting region="us-east-1" for "s3" from environment variable RCLONE_CONFIG_S3_REGION
2022/07/07 20:36:03 DEBUG : Setting region="us-east-1" for "s3" from environment variable RCLONE_CONFIG_S3_REGION
2022/07/07 20:36:03 DEBUG : s3: detected overridden config - adding "{452F6}" suffix to name
2022/07/07 20:36:03 DEBUG : Setting provider="AWS" for "s3" from environment variable RCLONE_CONFIG_S3_PROVIDER
2022/07/07 20:36:03 NOTICE: Config file "/Users/julian/.config/rclone/rclone.conf" not found - using defaults
2022/07/07 20:36:03 DEBUG : Setting access_key_id="MY_ACCES_KEY" for "s3" from environment variable RCLONE_CONFIG_S3_ACCESS_KEY_ID
2022/07/07 20:36:03 DEBUG : Setting secret_access_key="MY_SECRET_KEY" for "s3" from environment variable RCLONE_CONFIG_S3_SECRET_ACCESS_KEY
2022/07/07 20:36:03 DEBUG : Setting region="us-east-1" for "s3" from environment variable RCLONE_CONFIG_S3_REGION
2022/07/07 20:36:03 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/07/07 20:36:03 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/07/07 20:36:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/07 20:36:03 DEBUG : HTTP REQUEST (req 0x14000421b00)
2022/07/07 20:36:03 DEBUG : HEAD /model_repository/titanic_ensemble/config.pbtxt HTTP/1.1
Host: MY_PREFIX-us-east-1.s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.58.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220707T103603Z

2022/07/07 20:36:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/07 20:36:04 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/07 20:36:04 DEBUG : HTTP RESPONSE (req 0x14000421b00)
2022/07/07 20:36:04 DEBUG : HTTP/1.1 200 OK
Content-Length: 3606
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Date: Thu, 07 Jul 2022 10:36:05 GMT
Etag: "89c06428a2f312caaabfba5a308516b3"
Last-Modified: Tue, 05 Jul 2022 03:30:59 GMT
Server: AmazonS3
X-Amz-Id-2: YRMYomObrgFeS9cFs4UZD7B3mLT1ikrD65VUQgberj3Z1McqVigDTCHqkdn97iOsJ/j0o/RTlVI=
X-Amz-Request-Id: QAXX2PDQRVMBQ361

2022/07/07 20:36:04 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/07 20:36:04 DEBUG : fs cache: adding new entry for parent of "s3:MY_PREFIX-us-east-1/model_repository/titanic_ensemble/config.pbtxt", "s3{452F6}:MY_PREFIX-us-east-1/model_repository/titanic_ensemble"
2022/07/07 20:36:04 DEBUG : Creating backend with remote "test.txt"
2022/07/07 20:36:04 DEBUG : fs cache: renaming cache item "test.txt" to be canonical "/tmp/test.txt"
2022/07/07 20:36:04 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/07 20:36:04 DEBUG : HTTP REQUEST (req 0x14000162500)
2022/07/07 20:36:04 DEBUG : HEAD /model_repository/titanic_ensemble/config.pbtxt HTTP/1.1
Host: MY_PREFIX-us-east-1.s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.58.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220707T103604Z

2022/07/07 20:36:04 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/07/07 20:36:04 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/07 20:36:04 DEBUG : HTTP RESPONSE (req 0x14000162500)
2022/07/07 20:36:04 DEBUG : HTTP/1.1 200 OK
Content-Length: 3606
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Date: Thu, 07 Jul 2022 10:36:05 GMT
Etag: "89c06428a2f312caaabfba5a308516b3"
Last-Modified: Tue, 05 Jul 2022 03:30:59 GMT
Server: AmazonS3
X-Amz-Id-2: vT1R8V0jkAdmVKT2xozFPj6pPnSLfOyTujSLVp5UPphYBMJ6LUJ0983/syTsPvEFL/sIOlWO2xc=
X-Amz-Request-Id: QAXZ39RM1APQ8KM3

2022/07/07 20:36:04 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/07/07 20:36:04 DEBUG : config.pbtxt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022/07/07 20:36:04 DEBUG : config.pbtxt: Unchanged skipping
2022/07/07 20:36:04 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         1.1s

2022/07/07 20:36:04 DEBUG : 6 go routines active
12

And the equivalent aws body

$ aws s3 cp --region us-east-1 --debug s3://MY_PREFIX-us-east-1/model_repository/titanic_ensemble/config.pbtxt test.txt                          

...
2022-07-07 20:37:59,991 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=HeadObject) with params: {'url_path': '/MY_PREFIX-us-east-1/model_repository/titanic_ensemble/config.pbtxt', 'query_string': {}, 'method': 'HEAD', 'headers': {'User-Agent': 'aws-cli/2.4.27 Python/3.8.8 Darwin/21.4.0 exe/x86_64 prompt/off command/s3.cp'}, 'body': b'', 'url': 'https://s3.us-east-1.amazonaws.com/MY_PREFIX-us-east-1/model_repository/titanic_ensemble/config.pbtxt', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fce801ccac0>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'MY_PREFIX-us-east-1'}}}
2022-07-07 20:37:59,992 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.HeadObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fce801cca60>>
2022-07-07 20:37:59,992 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.HeadObject: calling handler <bound method S3EndpointSetter.set_signer of <botocore.utils.S3EndpointSetter object at 0x7fcea898c760>>
2022-07-07 20:37:59,992 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.HeadObject: calling handler <function set_operation_specific_signer at 0x7fce784a39d0>
2022-07-07 20:37:59,992 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.HeadObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7fcea898c760>>
2022-07-07 20:37:59,992 - MainThread - botocore.utils - DEBUG - Defaulting to S3 virtual host style addressing with path style addressing fallback.
2022-07-07 20:37:59,992 - MainThread - botocore.utils - DEBUG - Checking for DNS compatible bucket for: https://s3.us-east-1.amazonaws.com/MY_PREFIX-us-east-1/model_repository/titanic_ensemble/config.pbtxt
2022-07-07 20:37:59,992 - MainThread - botocore.utils - DEBUG - URI updated to: https://MY_PREFIX-us-east-1.s3.us-east-1.amazonaws.com/model_repository/titanic_ensemble/config.pbtxt
2022-07-07 20:37:59,992 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2022-07-07 20:37:59,992 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
HEAD
/model_repository/titanic_ensemble/config.pbtxt

host:MY_PREFIX-us-east-1.s3.us-east-1.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220707T103759Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2022-07-07 20:37:59,993 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20220707T103759Z
20220707/us-east-1/s3/aws4_request
4bf42c2b85afd51b41359a58fc87d10cf3873d9db2c4487270cacc6fb9aedae2
2022-07-07 20:37:59,993 - MainThread - botocore.auth - DEBUG - Signature:
d63afa449552917cd521fc828fb74df56f7592304abc82b212a08bd085b01fa9
2022-07-07 20:37:59,993 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=HEAD, url=https://MY_PREFIX-us-east-1.s3.us-east-1.amazonaws.com/model_repository/titanic_ensemble/config.pbtxt, headers={'User-Agent': b'aws-cli/2.4.27 Python/3.8.8 Darwin/21.4.0 exe/x86_64 prompt/off command/s3.cp', 'X-Amz-Date': b'20220707T103759Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAW44YBVKOQKIQBFYF/20220707/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=d63afa449552917cd521fc828fb74df56f7592304abc82b212a08bd085b01fa9'}>
2022-07-07 20:37:59,994 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/awscli/botocore/cacert.pem
2022-07-07 20:37:59,994 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): MY_PREFIX-us-east-1.s3.us-east-1.amazonaws.com:443
2022-07-07 20:38:00,963 - MainThread - urllib3.connectionpool - DEBUG - https://MY_PREFIX-us-east-1.s3.us-east-1.amazonaws.com:443 "HEAD /model_repository/titanic_ensemble/config.pbtxt HTTP/1.1" 200 0
2022-07-07 20:38:00,966 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': 'KX28EUxnGbKkqR4yANI0gATpNoe6EsbGzJNU07kYKS6hKzRNh63JHbEiwhzJRC7wjmCRQxNwCYU=', 'x-amz-request-id': 'Y4H5R7X7CXEFSBDP', 'Date': 'Thu, 07 Jul 2022 10:38:01 GMT', 'Last-Modified': 'Tue, 05 Jul 2022 03:30:59 GMT', 'ETag': '"89c06428a2f312caaabfba5a308516b3"', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'Server': 'AmazonS3', 'Content-Length': '3606'}
2022-07-07 20:38:00,966 - MainThread - botocore.parsers - DEBUG - Response body:
b''
2022-07-07 20:38:00,968 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.HeadObject: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fcea898c5b0>>
2022-07-07 20:38:00,968 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2022-07-07 20:38:00,968 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fcea898c610>>
2022-07-07 20:38:00,968 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.HeadObject: calling handler <function enhance_error_msg at 0x7fce88530940>
2022-07-07 20:38:00,968 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.HeadObject: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7fcea898c100>>
2022-07-07 20:38:00,969 - MainThread - s3transfer.utils - DEBUG - Acquiring 0
2022-07-07 20:38:00,969 - ThreadPoolExecutor-1_0 - s3transfer.tasks - DEBUG - DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7fcea8a03f10>}) about to wait for the following futures []
2022-07-07 20:38:00,969 - ThreadPoolExecutor-1_0 - s3transfer.tasks - DEBUG - DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7fcea8a03f10>}) done waiting for dependent futures
2022-07-07 20:38:00,970 - ThreadPoolExecutor-1_0 - s3transfer.tasks - DEBUG - Executing task DownloadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7fcea8a03f10>}) with kwargs {'client': <botocore.client.S3 object at 0x7fcea89aa460>, 'config': <s3transfer.manager.TransferConfig object at 0x7fcea89e61f0>, 'osutil': <s3transfer.utils.OSUtils object at 0x7fcea89e6280>, 'request_executor': <s3transfer.futures.BoundedExecutor object at 0x7fcea89e64c0>, 'transfer_future': <s3transfer.futures.TransferFuture object at 0x7fcea8a03f10>, 'io_executor': <s3transfer.futures.BoundedExecutor object at 0x7fcea89e6b20>}
2022-07-07 20:38:00,970 - ThreadPoolExecutor-1_0 - s3transfer.futures - DEBUG - Submitting task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'MY_PREFIX-us-east-1', 'key': 'model_repository/titanic_ensemble/config.pbtxt', 'extra_args': {}}) to executor <s3transfer.futures.BoundedExecutor object at 0x7fcea89e64c0> for transfer request: 0.
2022-07-07 20:38:00,970 - ThreadPoolExecutor-1_0 - s3transfer.utils - DEBUG - Acquiring 0
2022-07-07 20:38:00,970 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'MY_PREFIX-us-east-1', 'key': 'model_repository/titanic_ensemble/config.pbtxt', 'extra_args': {}}) about to wait for the following futures []
2022-07-07 20:38:00,970 - ThreadPoolExecutor-1_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2022-07-07 20:38:00,970 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'MY_PREFIX-us-east-1', 'key': 'model_repository/titanic_ensemble/config.pbtxt', 'extra_args': {}}) done waiting for dependent futures
2022-07-07 20:38:00,970 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - Executing task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'MY_PREFIX-us-east-1', 'key': 'model_repository/titanic_ensemble/config.pbtxt', 'extra_args': {}}) with kwargs {'client': <botocore.client.S3 object at 0x7fcea89aa460>, 'bucket': 'MY_PREFIX-us-east-1', 'key': 'model_repository/titanic_ensemble/config.pbtxt', 'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7fcea8a2d730>, 'extra_args': {}, 'callbacks': [functools.partial(<bound method BaseSubscriber.on_progress of <awscli.customizations.s3.subscribers.ProvideSizeSubscriber object at 0x7fcea8a03970>>, future=<s3transfer.futures.TransferFuture object at 0x7fcea8a03f10>), functools.partial(<bound method BaseSubscriber.on_progress of <awscli.customizations.s3.results.QueuedResultSubscriber object at 0x7fcea8a03d60>>, future=<s3transfer.futures.TransferFuture object at 0x7fcea8a03f10>), functools.partial(<bound method BaseSubscriber.on_progress of <awscli.customizations.s3.subscribers.DirectoryCreatorSubscriber object at 0x7fcea8a03df0>>, future=<s3transfer.futures.TransferFuture object at 0x7fcea8a03f10>), functools.partial(<bound method BaseSubscriber.on_progress of <awscli.customizations.s3.subscribers.ProvideLastModifiedTimeSubscriber object at 0x7fcea8a03e50>>, future=<s3transfer.futures.TransferFuture object at 0x7fcea8a03f10>), functools.partial(<bound method ProgressResultSubscriber.on_progress of <awscli.customizations.s3.results.ProgressResultSubscriber object at 0x7fcea8a03a00>>, future=<s3transfer.futures.TransferFuture object at 0x7fcea8a03f10>), functools.partial(<bound method BaseSubscriber.on_progress of <awscli.customizations.s3.results.DoneResultSubscriber object at 0x7fcea8a03a60>>, future=<s3transfer.futures.TransferFuture object at 0x7fcea8a03f10>)], 'max_attempts': 5, 'download_output_manager': <s3transfer.download.DownloadFilenameOutputManager object at 0x7fcea8a2d790>, 'io_chunksize': 262144, 'bandwidth_limiter': None}
2022-07-07 20:38:00,970 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event provide-client-params.s3.GetObject: calling handler <function base64_decode_input_blobs at 0x7fce885a39d0>
2022-07-07 20:38:00,970 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-parameter-build.s3.GetObject: calling handler <function sse_md5 at 0x7fce784a3d30>
2022-07-07 20:38:00,970 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-parameter-build.s3.GetObject: calling handler <function validate_bucket_name at 0x7fce784a3ca0>
2022-07-07 20:38:00,970 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-parameter-build.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7fcea89e6040>>
2022-07-07 20:38:00,970 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-parameter-build.s3.GetObject: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x7fcea89e6100>>
2022-07-07 20:38:00,970 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-parameter-build.s3.GetObject: calling handler <function generate_idempotent_uuid at 0x7fce784a3af0>
2022-07-07 20:38:00,971 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-call.s3.GetObject: calling handler <function add_expect_header at 0x7fce784a8040>
2022-07-07 20:38:00,971 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-call.s3.GetObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7fcea89e6040>>
2022-07-07 20:38:00,971 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-call.s3.GetObject: calling handler <function inject_api_version_header_if_needed at 0x7fce784a93a0>
2022-07-07 20:38:00,971 - ThreadPoolExecutor-0_0 - botocore.endpoint - DEBUG - Making request for OperationModel(name=GetObject) with params: {'url_path': '/MY_PREFIX-us-east-1/model_repository/titanic_ensemble/config.pbtxt', 'query_string': {}, 'method': 'GET', 'headers': {'User-Agent': 'aws-cli/2.4.27 Python/3.8.8 Darwin/21.4.0 exe/x86_64 prompt/off command/s3.cp'}, 'body': b'', 'url': 'https://s3.us-east-1.amazonaws.com/MY_PREFIX-us-east-1/model_repository/titanic_ensemble/config.pbtxt', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fcea89aa520>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'MY_PREFIX-us-east-1'}}}
2022-07-07 20:38:00,971 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event request-created.s3.GetObject: calling handler <function signal_not_transferring at 0x7fce789330d0>
2022-07-07 20:38:00,971 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event request-created.s3.GetObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fcea89aa4c0>>
2022-07-07 20:38:00,971 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event choose-signer.s3.GetObject: calling handler <bound method S3EndpointSetter.set_signer of <botocore.utils.S3EndpointSetter object at 0x7fcea89e6190>>
2022-07-07 20:38:00,971 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event choose-signer.s3.GetObject: calling handler <function set_operation_specific_signer at 0x7fce784a39d0>
2022-07-07 20:38:00,971 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event before-sign.s3.GetObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7fcea89e6190>>
2022-07-07 20:38:00,971 - ThreadPoolExecutor-0_0 - botocore.utils - DEBUG - Defaulting to S3 virtual host style addressing with path style addressing fallback.
2022-07-07 20:38:00,971 - ThreadPoolExecutor-0_0 - botocore.utils - DEBUG - Checking for DNS compatible bucket for: https://s3.us-east-1.amazonaws.com/MY_PREFIX-us-east-1/model_repository/titanic_ensemble/config.pbtxt
2022-07-07 20:38:00,971 - ThreadPoolExecutor-0_0 - botocore.utils - DEBUG - URI updated to: https://MY_PREFIX-us-east-1.s3.us-east-1.amazonaws.com/model_repository/titanic_ensemble/config.pbtxt
2022-07-07 20:38:00,971 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - Calculating signature using v4 auth.
2022-07-07 20:38:00,971 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - CanonicalRequest:
GET
/model_repository/titanic_ensemble/config.pbtxt

host:MY_PREFIX-us-east-1.s3.us-east-1.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220707T103800Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2022-07-07 20:38:00,971 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20220707T103800Z
20220707/us-east-1/s3/aws4_request
8f4766017c9015abc773069bdb90168e2952a594eccab446b9896d83d2536391
2022-07-07 20:38:00,971 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - Signature:
9df2d3fdbafb38a8baefe46f3c3d79c2d827092550fb153bb1d3ca5517142a18
2022-07-07 20:38:00,971 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event request-created.s3.GetObject: calling handler <function signal_transferring at 0x7fce78933160>
2022-07-07 20:38:00,972 - ThreadPoolExecutor-0_0 - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=True, method=GET, url=https://MY_PREFIX-us-east-1.s3.us-east-1.amazonaws.com/model_repository/titanic_ensemble/config.pbtxt, headers={'User-Agent': b'aws-cli/2.4.27 Python/3.8.8 Darwin/21.4.0 exe/x86_64 prompt/off command/s3.cp', 'X-Amz-Date': b'20220707T103800Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAW44YBVKOQKIQBFYF/20220707/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=9df2d3fdbafb38a8baefe46f3c3d79c2d827092550fb153bb1d3ca5517142a18'}>
2022-07-07 20:38:00,972 - ThreadPoolExecutor-0_0 - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/awscli/botocore/cacert.pem
2022-07-07 20:38:00,972 - ThreadPoolExecutor-0_0 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): MY_PREFIX-us-east-1.s3.us-east-1.amazonaws.com:443
2022-07-07 20:38:01,936 - ThreadPoolExecutor-0_0 - urllib3.connectionpool - DEBUG - https://MY_PREFIX-us-east-1.s3.us-east-1.amazonaws.com:443 "GET /model_repository/titanic_ensemble/config.pbtxt HTTP/1.1" 200 3606
2022-07-07 20:38:01,937 - ThreadPoolExecutor-0_0 - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': 'hnGZAIG9+mVhOZ66bJtxZSPMpzVZC+1CV9kze6x8aHP+XaZfbgbR/E7t9ECo91QfVZv6zViL7Uo=', 'x-amz-request-id': 'M774Y8KQEATF8CDX', 'Date': 'Thu, 07 Jul 2022 10:38:02 GMT', 'Last-Modified': 'Tue, 05 Jul 2022 03:30:59 GMT', 'ETag': '"89c06428a2f312caaabfba5a308516b3"', 'Accept-Ranges': 'bytes', 'Content-Type': 'binary/octet-stream', 'Server': 'AmazonS3', 'Content-Length': '3606'}
2022-07-07 20:38:01,937 - ThreadPoolExecutor-0_0 - botocore.parsers - DEBUG - Response body:
<botocore.response.StreamingBody object at 0x7fce886291f0>
2022-07-07 20:38:01,939 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event needs-retry.s3.GetObject: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fcea89aafa0>>
2022-07-07 20:38:01,939 - ThreadPoolExecutor-0_0 - botocore.retries.standard - DEBUG - Not retrying request.
2022-07-07 20:38:01,940 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event needs-retry.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fcea89e6040>>
2022-07-07 20:38:01,940 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event after-call.s3.GetObject: calling handler <function enhance_error_msg at 0x7fce88530940>
2022-07-07 20:38:01,940 - ThreadPoolExecutor-0_0 - botocore.hooks - DEBUG - Event after-call.s3.GetObject: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7fcea89aaaf0>>
2022-07-07 20:38:01,941 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 0}) about to wait for the following futures []
2022-07-07 20:38:01,941 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - IOWriteTask(transfer_id=0, {'offset': 0}) done waiting for dependent futures
2022-07-07 20:38:01,941 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - Executing task IOWriteTask(transfer_id=0, {'offset': 0}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7fcea8a2d730>, 'offset': 0}
2022-07-07 20:38:01,941 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - IORenameFileTask(transfer_id=0, {'final_filename': '/private/tmp/test.txt/config.pbtxt'}) about to wait for the following futures []
2022-07-07 20:38:01,941 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - IORenameFileTask(transfer_id=0, {'final_filename': '/private/tmp/test.txt/config.pbtxt'}) done waiting for dependent futures
2022-07-07 20:38:01,942 - ThreadPoolExecutor-0_0 - s3transfer.tasks - DEBUG - Executing task IORenameFileTask(transfer_id=0, {'final_filename': '/private/tmp/test.txt/config.pbtxt'}) with kwargs {'fileobj': <s3transfer.utils.DeferredOpenFile object at 0x7fcea8a2d730>, 'final_filename': '/private/tmp/test.txt/config.pbtxt', 'osutil': <s3transfer.utils.OSUtils object at 0x7fcea89e6280>}
2022-07-07 20:38:01,942 - ThreadPoolExecutor-0_0 - s3transfer.utils - DEBUG - Releasing acquire 0/None
download: s3://MY_PREFIX-us-east-1/model_repository/titanic_ensemble/config.pbtxt to test.txt/config.pbtxt
2022-07-07 20:38:01,942 - Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.
13

Thats a shame that didn't work! I can see by the requests that it is using the provider.

Perhaps it is time to dig into the auth header

Here is what the aws cli tool sent

You can get rclone to show the auth with -vv --dump bodies,auth

Does that look significantly different to the aws cli one?

If you want to try adding this header to the GET requests you can do so with

--header-download 'x-amz-server-side-encryption-bucket-key-enabled: true'

Same with any other headers you think might help.

14

Headers are:

GET /model_repository/titanic_ensemble/config.pbtxt HTTP/1.1
Host: MY_PREFIX-us-west-2-MY_ACCOUNT.s3.us-west-2.amazonaws.com
User-Agent: rclone/v1.58.1
Authorization: AWS4-HMAC-SHA256 Credential=AKIAW44YBVKOQDVV6C4J/20220707/us-west-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=7b5f5180fab4c75ef4cdc30fce1990b1e01399a458203bf129a0a120b8102762
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220707T112553Z
Accept-Encoding: gzip

The only difference I could see besides the User-Agent is the Signature as a result of the aws cli using SignedHeaders=host;x-amz-content-sha256;x-amz-date and the date is slightly different in each case.

The x-amz-server-side-encryption-bucket-key-enabled was on the response not request, so I don't think that matters.

I also wanted to double check that the go sdk works for this use case. And it did with this code.

package main

import (
	"fmt"
	"log"
	"os"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3"
	"github.com/aws/aws-sdk-go/service/s3/s3manager"
)

func download(filename, bucket, key string) error {
	// Create a file to write the S3 Object contents to.

	// The session the S3 Downloader will use
	sess := session.Must(session.NewSession(
		aws.NewConfig().WithRegion("us-west-2").WithLogLevel(aws.LogDebug),
	))

	// Create a downloader with the session and default options
	downloader := s3manager.NewDownloader(sess)

	f, err := os.Create(filename)
	if err != nil {
		return fmt.Errorf("failed to create file %q, %v", filename, err)
	}

	// Write the contents of S3 Object to the file
	n, err := downloader.Download(f, &s3.GetObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(key),
	})
	if err != nil {
		return fmt.Errorf("failed to download file, %v", err)
	}

	fmt.Printf("file downloaded, %d bytes\n", n)
	return nil
}

func main() {
	filename := "text.txt"
	bucket := "MY_PREFIX-us-west-2-MY_ACCOUNT"
	key := "model_repository/titanic_ensemble/config.pbtxt"

	if err := download(filename, bucket, key); err != nil {
		log.Panic(err)
	}
}

And in terms of logging, this is what I see:

---[ REQUEST POST-SIGN ]-----------------------------
GET /model_repository/titanic_ensemble/config.pbtxt HTTP/1.1
Host: MY_PREFIX-us-west-2-MY_ACCOUNT.s3.us-west-2.amazonaws.com
User-Agent: aws-sdk-go/1.44.49 (go1.18.3; darwin; arm64) S3Manager
Authorization: AWS4-HMAC-SHA256 Credential=AKIAW44YBVKOQKIQBFYF/20220707/us-west-2/s3/aws4_request, SignedHeaders=host;range;x-amz-content-sha256;x-amz-date, Signature=0baf9913890ffe00eb8a25b4c05cb50c6ddfa49e90e27a3d7aa50529cce250f4
Range: bytes=0-5242879
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220707T120057Z


-----------------------------------------------------
2022/07/07 22:00:58 DEBUG: Response s3/GetObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 206 Partial Content
Content-Length: 3606
Accept-Ranges: bytes
Content-Range: bytes 0-3605/3606
Content-Type: binary/octet-stream
Date: Thu, 07 Jul 2022 12:00:59 GMT
Etag: "cbf35447b8e32dda8db30ceb05292937"
Last-Modified: Wed, 06 Jul 2022 02:22:26 GMT
Server: AmazonS3
X-Amz-Id-2: GMklrHf9uUL7Hp4V87XLAt/EzMnjBwY9YhePcE4f4gSN+PViAQSowvqrPeHGX1v7OByGtniOvD4=
X-Amz-Request-Id: W6VJ1C1X8SMT04HP
X-Amz-Server-Side-Encryption: aws:kms
X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id: arn:aws:kms:us-west-2:MY_ACCOUNT:key/f6fbd5c7-ab1b-4eb6-a769-ea789c24125f
X-Amz-Server-Side-Encryption-Bucket-Key-Enabled: true
X-Amz-Version-Id: zmWsxJtj5EEQmrb8xr2bzzC.IRT4OgkR


-----------------------------------------------------

I'm not sure if RClone is using the s3Manager to download, but perhaps it could, or else you may need to dig into the code further.

15

As far as I can see with your testing, rclone is using the correct request, so I think there is no need to investigate that further.

Very interesting you made it work with the go sdk. Rclone doesn't use the download manager as it needs to integrate with the rest of rclone.

Rclone uses GetObject basically - the details are in the Open function. You could try that in your Go code and then io.Copy(f, resp.Body) to read the data into the file.

I suspect you are going to get the same result that it works.

I suspect then something is up with rclone's auth. I'm not sure how you are passing the auth to the Go test code - is it with a credentials file? Whichever way you are passing the auth, you should be able to get rclone to use it by setting env_auth = true.

Can you try using the remote :s3,env_auth: with the same credentials you are using for your Go test program?

16

Hi,
So it turns out that there was another profile being used by the aws cli, that had less limited permissions then the one I was testing. When I added the specific credentials to the golang code I got the same error. So the fix was to add by kms:GeneratedDataKey and kms:Decrypt - which isn't covered in the S3FullAccess managed policy surpassingly

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt",
                "kms:GenerateDataKey"
            ],
            "Resource": "MY_KEY_ARN"
        }
    ]
}

Once I added this, I could read this data, so will close the issue. The reason this wasn't required in the us-east-1 region was that default encryption was disabled.
Thanks this issue can be resolved now.

17

Great - glad you've got to the bottom of it. I'm relieved it wasn't some really complicated bit of auth rclone was missing!

18

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.