What is the problem you are having with rclone?
rclone operation fails when S3 region is specified via an environment variable. Same configuration works when specified in ~/.rclone.confg so this seems to be a bug with env var configuration.
Run the command 'rclone version' and share the full output of the command.
rclone v1.57.0
- os/version: darwin 12.1 (64 bit)
- os/kernel: 21.2.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.17.2
- go/linking: dynamic
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
AWS S3
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
env RCLONE_CONFIG_MYS3_TYPE=s3 RCLONE_CONFIG_MYS3_ACCESS_KEY_ID=AKIAJV3VAEW6OFKXWMLQ RCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY=X RCLONE_CONFIG_MYS3_ENDPOINT=s3.amazonaws.com RCLONE_CONFIG_MYS3_REGION=us-east-2 rclone ls -vv mys3:s3backup20210924
The rclone config contents with secrets removed.
[mys3]
type = s3
access_key_id = AKIAJV3VAEW6OFKXWMLQ
secret_access_key = X
region=us-east-2
A log from the command with the -vv
flag
% env RCLONE_CONFIG_MYS3_TYPE=s3 RCLONE_CONFIG_MYS3_ACCESS_KEY_ID=AKIAJV3VAEW6OFKXWMLQ RCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY=X RCLONE_CONFIG_MYS3_ENDPOINT=s3.amazonaws.com RCLONE_CONFIG_MYS3_REGION=us-east-2 rclone ls -vv mys3:s3backup20210924
2022/01/11 12:29:30 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "ls" "-vv" "mys3:s3backup20210924"]
2022/01/11 12:29:30 DEBUG : Creating backend with remote "mys3:s3backup20210924"
2022/01/11 12:29:30 DEBUG : Setting type="s3" for "mys3" from environment variable RCLONE_CONFIG_MYS3_TYPE
2022/01/11 12:29:30 DEBUG : Setting access_key_id="AKIAJV3VAEW6OFKXWMLQ" for "mys3" from environment variable RCLONE_CONFIG_MYS3_ACCESS_KEY_ID
2022/01/11 12:29:30 DEBUG : Setting secret_access_key="X" for "mys3" from environment variable RCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY
2022/01/11 12:29:30 DEBUG : Setting region="us-east-2" for "mys3" from environment variable RCLONE_CONFIG_MYS3_REGION
2022/01/11 12:29:30 DEBUG : Setting region="us-east-2" for "mys3" from environment variable RCLONE_CONFIG_MYS3_REGION
2022/01/11 12:29:30 DEBUG : Setting region="us-east-2" for "mys3" from environment variable RCLONE_CONFIG_MYS3_REGION
2022/01/11 12:29:30 DEBUG : Setting endpoint="s3.amazonaws.com" for "mys3" from environment variable RCLONE_CONFIG_MYS3_ENDPOINT
2022/01/11 12:29:30 DEBUG : Setting endpoint="s3.amazonaws.com" for "mys3" from environment variable RCLONE_CONFIG_MYS3_ENDPOINT
2022/01/11 12:29:30 DEBUG : Setting endpoint="s3.amazonaws.com" for "mys3" from environment variable RCLONE_CONFIG_MYS3_ENDPOINT
2022/01/11 12:29:30 DEBUG : Setting endpoint="s3.amazonaws.com" for "mys3" from environment variable RCLONE_CONFIG_MYS3_ENDPOINT
2022/01/11 12:29:30 DEBUG : Setting endpoint="s3.amazonaws.com" for "mys3" from environment variable RCLONE_CONFIG_MYS3_ENDPOINT
2022/01/11 12:29:30 DEBUG : Setting endpoint="s3.amazonaws.com" for "mys3" from environment variable RCLONE_CONFIG_MYS3_ENDPOINT
2022/01/11 12:29:30 DEBUG : Setting endpoint="s3.amazonaws.com" for "mys3" from environment variable RCLONE_CONFIG_MYS3_ENDPOINT
2022/01/11 12:29:30 DEBUG : mys3: detected overridden config - adding "{lvIe0}" suffix to name
2022/01/11 12:29:30 DEBUG : Using config file from "/Users/jamshid/.rclone.conf"
2022/01/11 12:29:30 DEBUG : Setting access_key_id="AKIAJV3VAEW6OFKXWMLQ" for "mys3" from environment variable RCLONE_CONFIG_MYS3_ACCESS_KEY_ID
2022/01/11 12:29:30 DEBUG : Setting secret_access_key="X" for "mys3" from environment variable RCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY
2022/01/11 12:29:30 DEBUG : Setting region="us-east-2" for "mys3" from environment variable RCLONE_CONFIG_MYS3_REGION
2022/01/11 12:29:30 DEBUG : Setting endpoint="s3.amazonaws.com" for "mys3" from environment variable RCLONE_CONFIG_MYS3_ENDPOINT
2022/01/11 12:29:30 DEBUG : fs cache: renaming cache item "mys3:s3backup20210924" to be canonical "mys3{lvIe0}:s3backup20210924"
2022/01/11 12:29:31 ERROR : S3 bucket s3backup20210924: Failed to update region for bucket: reading bucket location failed: AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-2' is wrong; expecting 'us-east-1'
status code: 400, request id: QF9BJZ121F4SQZM5, host id: rePrb1hyXCEuQyhF8gaME+PaoZ7UjgjQHy6JoGiCspf3+rhYzkAZKwUnAkfObyISoy+IAF23FJ0=
2022/01/11 12:29:31 DEBUG : 4 go routines active
2022/01/11 12:29:31 Failed to ls: BucketRegionError: incorrect region, the bucket is not in 'us-east-2' region at endpoint 's3.amazonaws.com'
status code: 301, request id: , host id:
It works if I specify the endpoint as s3.us-east-2.amazonaws.com
but it seems the generic s3.amazonaws.com
endpoint should work since that works when specified via the config file instead of env vars.