S3 operation fails when region specified in environment variable

Suspected Bug
1

What is the problem you are having with rclone?

rclone operation fails when S3 region is specified via an environment variable. Same configuration works when specified in ~/.rclone.confg so this seems to be a bug with env var configuration.

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0
- os/version: darwin 12.1 (64 bit)
- os/kernel: 21.2.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.17.2
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

env RCLONE_CONFIG_MYS3_TYPE=s3 RCLONE_CONFIG_MYS3_ACCESS_KEY_ID=AKIAJV3VAEW6OFKXWMLQ RCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY=X RCLONE_CONFIG_MYS3_ENDPOINT=s3.amazonaws.com RCLONE_CONFIG_MYS3_REGION=us-east-2 rclone ls -vv mys3:s3backup20210924

The rclone config contents with secrets removed.

[mys3]
type = s3
access_key_id = AKIAJV3VAEW6OFKXWMLQ
secret_access_key = X
region=us-east-2

A log from the command with the -vv flag

% env RCLONE_CONFIG_MYS3_TYPE=s3 RCLONE_CONFIG_MYS3_ACCESS_KEY_ID=AKIAJV3VAEW6OFKXWMLQ RCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY=X RCLONE_CONFIG_MYS3_ENDPOINT=s3.amazonaws.com RCLONE_CONFIG_MYS3_REGION=us-east-2 rclone ls -vv mys3:s3backup20210924
2022/01/11 12:29:30 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "ls" "-vv" "mys3:s3backup20210924"]
2022/01/11 12:29:30 DEBUG : Creating backend with remote "mys3:s3backup20210924"
2022/01/11 12:29:30 DEBUG : Setting type="s3" for "mys3" from environment variable RCLONE_CONFIG_MYS3_TYPE
2022/01/11 12:29:30 DEBUG : Setting access_key_id="AKIAJV3VAEW6OFKXWMLQ" for "mys3" from environment variable RCLONE_CONFIG_MYS3_ACCESS_KEY_ID
2022/01/11 12:29:30 DEBUG : Setting secret_access_key="X" for "mys3" from environment variable RCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY
2022/01/11 12:29:30 DEBUG : Setting region="us-east-2" for "mys3" from environment variable RCLONE_CONFIG_MYS3_REGION
2022/01/11 12:29:30 DEBUG : Setting region="us-east-2" for "mys3" from environment variable RCLONE_CONFIG_MYS3_REGION
2022/01/11 12:29:30 DEBUG : Setting region="us-east-2" for "mys3" from environment variable RCLONE_CONFIG_MYS3_REGION
2022/01/11 12:29:30 DEBUG : Setting endpoint="s3.amazonaws.com" for "mys3" from environment variable RCLONE_CONFIG_MYS3_ENDPOINT
2022/01/11 12:29:30 DEBUG : Setting endpoint="s3.amazonaws.com" for "mys3" from environment variable RCLONE_CONFIG_MYS3_ENDPOINT
2022/01/11 12:29:30 DEBUG : Setting endpoint="s3.amazonaws.com" for "mys3" from environment variable RCLONE_CONFIG_MYS3_ENDPOINT
2022/01/11 12:29:30 DEBUG : Setting endpoint="s3.amazonaws.com" for "mys3" from environment variable RCLONE_CONFIG_MYS3_ENDPOINT
2022/01/11 12:29:30 DEBUG : Setting endpoint="s3.amazonaws.com" for "mys3" from environment variable RCLONE_CONFIG_MYS3_ENDPOINT
2022/01/11 12:29:30 DEBUG : Setting endpoint="s3.amazonaws.com" for "mys3" from environment variable RCLONE_CONFIG_MYS3_ENDPOINT
2022/01/11 12:29:30 DEBUG : Setting endpoint="s3.amazonaws.com" for "mys3" from environment variable RCLONE_CONFIG_MYS3_ENDPOINT
2022/01/11 12:29:30 DEBUG : mys3: detected overridden config - adding "{lvIe0}" suffix to name
2022/01/11 12:29:30 DEBUG : Using config file from "/Users/jamshid/.rclone.conf"
2022/01/11 12:29:30 DEBUG : Setting access_key_id="AKIAJV3VAEW6OFKXWMLQ" for "mys3" from environment variable RCLONE_CONFIG_MYS3_ACCESS_KEY_ID
2022/01/11 12:29:30 DEBUG : Setting secret_access_key="X" for "mys3" from environment variable RCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY
2022/01/11 12:29:30 DEBUG : Setting region="us-east-2" for "mys3" from environment variable RCLONE_CONFIG_MYS3_REGION
2022/01/11 12:29:30 DEBUG : Setting endpoint="s3.amazonaws.com" for "mys3" from environment variable RCLONE_CONFIG_MYS3_ENDPOINT
2022/01/11 12:29:30 DEBUG : fs cache: renaming cache item "mys3:s3backup20210924" to be canonical "mys3{lvIe0}:s3backup20210924"
2022/01/11 12:29:31 ERROR : S3 bucket s3backup20210924: Failed to update region for bucket: reading bucket location failed: AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-2' is wrong; expecting 'us-east-1'
	status code: 400, request id: QF9BJZ121F4SQZM5, host id: rePrb1hyXCEuQyhF8gaME+PaoZ7UjgjQHy6JoGiCspf3+rhYzkAZKwUnAkfObyISoy+IAF23FJ0=
2022/01/11 12:29:31 DEBUG : 4 go routines active
2022/01/11 12:29:31 Failed to ls: BucketRegionError: incorrect region, the bucket is not in 'us-east-2' region at endpoint 's3.amazonaws.com'
	status code: 301, request id: , host id:

It works if I specify the endpoint as s3.us-east-2.amazonaws.com but it seems the generic s3.amazonaws.com endpoint should work since that works when specified via the config file instead of env vars.

2

hi,
took a quick read of your post,

region and endpoint need to be self-consistent.

  • took a quick look at my config file
    --- aws s3 remotes, use region only
    --- with wasabi s3 clone, i use endpoint only.

  • your setup
    from the config file region=us-east-2
    from the env var, RCLONE_CONFIG_MYS3_ENDPOINT=s3.amazonaws.com
    and aws is complaining about that
    BucketRegionError: incorrect region, the bucket is not in 'us-east-2' region at endpoint 's3.amazonaws.com'

1 Like
3

Ah I see, sorry, I thought I was specifying both the region and generic endpoint in my .rclone.conf, but I was not. So never mind, there is no rclone inconsistency here -- using this .rclone.conf fails the same way as with the env var configuration.

Ok so the region is required when using AWS S3 (and not using the default us-east-1 region). The endpoint is optional, but if it is specified it must be the specific endpoint not the generic s3.amazonaws.com.

E.g. since my bucket is in us-east-2 both of these configs fail:

[tests3]
type = s3
access_key_id = AKIAJV3VAEW6OFKXWMLQ
secret_access_key = X
region=us-east-2
endpoint = https://s3.amazonaws.com

[tests3]
type = s3
access_key_id = AKIAJV3VAEW6OFKXWMLQ
secret_access_key = X
endpoint = https://s3.us-east-2.amazonaws.com

% rclone ls -vv tests3:s3backup20210924
2022/01/11 13:23:21 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "ls" "-vv" "tests3:s3backup20210924"]
2022/01/11 13:23:21 DEBUG : Creating backend with remote "tests3:s3backup20210924"
2022/01/11 13:23:21 DEBUG : Using config file from "/Users/jamshid/.rclone.conf"

2022/01/11 13:23:21 ERROR : S3 bucket s3backup20210924: Failed to update region for bucket: reading bucket location failed: AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-2' is wrong; expecting 'us-east-1'
	status code: 400, request id: 5AD35230DMKWPE4H, host id: AkLrokiziv7NPxn3+NG+r10RhR0hZPWypkYqU7Vnopt5MHL6sNxKO0tBr+6+tHYMA6j3HB24G0o=
2022/01/11 13:23:21 DEBUG : 4 go routines active
2022/01/11 13:23:21 Failed to ls: BucketRegionError: incorrect region, the bucket is not in 'us-east-2' region at endpoint 'https://s3.amazonaws.com'
	status code: 301, request id: , host id: 
...

% rclone ls -vv tests3:s3backup20210924
2022/01/11 13:36:10 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "ls" "-vv" "tests3:s3backup20210924"]
2022/01/11 13:36:10 DEBUG : Creating backend with remote "tests3:s3backup20210924"
2022/01/11 13:36:10 DEBUG : Using config file from "/Users/jamshid/.rclone.conf"
2022/01/11 13:36:10 DEBUG : 4 go routines active
2022/01/11 13:36:10 Failed to ls: AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-east-2'
	status code: 400, request id: X04GR477FF03DBV1, host id: 9X1rWPJzkod805OmZ5vUX0hM0b163nFTwmQNDV4jCXe7RQoZicLDzQqo/v7FFkY6zRV+4bSC4dY=

This works but specifying the endpoint is unnecessary:

[tests3]
type = s3
access_key_id = AKIAJV3VAEW6OFKXWMLQ
secret_access_key = X
region=us-east-2
endpoint = https://s3.us-east-2.amazonaws.com
4

sure, that is ok.

the good news, this is not a rclone bug, yeah!

5

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.