--s3-profile flag appears to be broken in 1.53.4 and 1.54.1

bitlifter · March 24, 2021, 4:13pm

What is the problem you are having with rclone?

Connecting from one account S3 bucket to another account's S3 bucket using the S3 profile feature fails regardless of whether its specified in the config file or when using the --s3-profile flag. On 1.53.4 rclone uses the default profile even when --s3-profile is defined. If AWS_PROFILE is exported, it works but only one profile at a time can be used this way and limits the ability to connect to multiple accounts. On 1.54.1 profiles do not appear to t work at all, even when AWS_PROFILE is exported

I only have one set of AWS credentials and I'm assuming an admin role via different AWS profiles to rclone copy between buckets in different accounts. In order to copy from one AWS S3 bucket to another, rclone needs to support multiple profiles via the --s3-profile or "profile = " in the configuration file.

steps to reproduce on 1.53.4:

create a profile in ~/.aws/config called test with the role it can assume in a different account (different than default)
export AWS_PROFILE=test
run rclone lsd {name of backend remote}:
It will return a list of buckets from the test profile.
If we unset AWS_PROFILE and use rclone --s3-profile test lsd {name of backend remote}:
it will only show a list of folders in the default profile, not the profile test

On 1.54.1 it no longer uses the default profile unless specifically defined using "--s3-profile default" and it fails for any other specified profile. It also fails when exporting AWS_PROFILE
e.g. rclone --s3-profile default lsd remote: = working
rclone --s3-profile test lsd remote: = not working
aws s3 ls --profile test = working

What is your rclone version (output from `rclone version`)

1.53.4, 1.54.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

ubuntu 20.04

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

Command below on 1.53.4 returns the default bucket instead the ybc-builds profile bucket directory listing:

rclone --s3-profile ybc-builds -vvvv  lsd ybc-prod-engineering-useast1-ybcs-release:  
2021/03/24 15:52:37 DEBUG : rclone: Version "v1.53.4" starting with parameters ["rclone" "--s3-profile" "ybc-builds" "-vvvv" "lsd" "ybc-prod-
engineering-useast1-ybcs-release:"]                            
2021/03/24 15:52:37 DEBUG : Using config file from "/home/cedgar/.config/rclone/rclone.conf"
2021/03/24 15:52:37 DEBUG : Creating backend with remote "ybc-prod-engineering-useast1-ybcs-release:"
          -1 2021-02-08 21:33:51        -1 az--useast1-adsync--us-west-2-demosync
          -1 2020-12-20 00:00:43        -1 cf-aws-config-enable-master-acct-configbucket
          -1 2020-12-19 23:59:49        -1 cf-templates-1v7mowxzq1vyf-us-east-1
          -1 2021-02-08 18:24:11        -1 cf-templates-1v7mowxzq1vyf-us-west-2
          -1 2020-12-14 18:16:35        -1 config-bucket
          -1 2020-05-29 16:29:37        -1 jrtest
          -1 2020-04-08 20:48:31        -1 ybc-prod-costreports
          -1 2019-11-07 01:10:38        -1 ybc-prod-terraform-useast1-log                                      
          -1 2019-11-07 01:10:45        -1 ybc-prod-terraform-useast1-state                                                                  2021/03/24 15:52:37 DEBUG : 4 go routines active

Same command on 1.54.1 returns

rclone --s3-profile ybc-builds -vvvv  lsd ybc-prod-engineering-useast1-ybcs-release:  
2021/03/24 16:11:57 DEBUG : rclone: Version "v1.54.1" starting with parameters ["rclone" "--s3-profile" "ybc-builds" "-vvvv" "lsd" "ybc-prod-engineering-useast1-ybcs-release:"]
2021/03/24 16:11:57 DEBUG : Using config file from "/home/cedgar/.config/rclone/rclone.conf"
2021/03/24 16:11:57 DEBUG : Creating backend with remote "ybc-prod-engineering-useast1-ybcs-release:"
2021/03/24 16:11:58 ERROR : : error listing: AccessDenied: Access Denied
        status code: 403, request id: H9FS9ZACYZ6BB77J, host id: EvD+zdVeMcObxJTbSm1mYCcnEmXIEmu3jhLFM1+AyX77ukSh0Q0+o2ITl3+9EAyADVd+ml3+sKY=
2021/03/24 16:11:58 DEBUG : 4 go routines active
2021/03/24 16:11:58 Failed to lsd with 2 errors: last error was: AccessDenied: Access Denied
        status code: 403, request id: H9FS9ZACYZ6BB77J, host id: EvD+zdVeMcObxJTbSm1mYCcnEmXIEmu3jhLFM1+AyX77ukSh0Q0+o2ITl3+9EAyADVd+ml3+sKY=

Same command on 1.54.1 but specifying the profile "default" works:

rclone --s3-profile default -vvvv  lsd ybc-prod-engineering-useast1-ybcs-release:  
2021/03/24 16:00:05 DEBUG : rclone: Version "v1.54.1" starting with parameters ["rclone" "--s3-profile" "default" "-vvvv" "lsd" "ybc-prod-engineering-useast1-ybcs-release:"]
2021/03/24 16:00:05 DEBUG : Using config file from "/home/cedgar/.config/rclone/rclone.conf"
2021/03/24 16:00:05 DEBUG : Creating backend with remote "ybc-prod-engineering-useast1-ybcs-release:"
           -1 2021-02-08 21:33:51        -1 az--useast1-adsync--us-west-2-demosync
          -1 2020-12-20 00:00:43        -1 cf-aws-config-enable-master-acct-configbucket
          -1 2020-12-19 23:59:49        -1 cf-templates-1v7mowxzq1vyf-us-east-1
          -1 2021-02-08 18:24:11        -1 cf-templates-1v7mowxzq1vyf-us-west-2
          -1 2020-12-14 18:16:35        -1 config-bucket
          -1 2020-05-29 16:29:37        -1 jrtest
          -1 2020-04-08 20:48:31        -1 ybc-prod-costreports
          -1 2019-11-07 01:10:38        -1 ybc-prod-terraform-useast1-log                                      
          -1 2019-11-07 01:10:45        -1 ybc-prod-terraform-useast1-state
2021/03/24 16:00:05 DEBUG : 4 go routines active

The rclone config contents with secrets removed.

[ybc-prod-engineering-useast1-ybcs-release]
type = s3
provider = AWS
env_auth = true
region = us-east-1
acl = private
server_side_encryption = AES256
storage_class = STANDARD
profile = ybc-builds

[ybcotd-prod-engineering-useast1]
type = s3
provider = AWS
env_auth = true
region = us-east-1
server_side_encryption = AES256
storage_class = STANDARD
sse_customer_algorithm = AES256
profile = ybc-opsbuilds
upload_concurrency = 18

A log from the command with the `-vv` flag

2021/03/24 15:52:37 DEBUG : rclone: Version "v1.53.4" starting with parameters ["rclone" "--s3-profile" "ybc-builds" "-vvvv" "lsd" "ybc-prod-engineering-useast1-ybcs-release:"]
2021/03/24 15:52:37 DEBUG : Using config file from "/home/cedgar/.config/rclone/rclone.conf"
2021/03/24 15:52:37 DEBUG : Creating backend with remote "ybc-prod-engineering-useast1-ybcs-release:"
         {shows contents of default bucket, not ybc-builds account bucket}


rclone --s3-profile ybc-builds -vvvv  lsd ybc-prod-engineering-useast1-ybcs-release:  
2021/03/24 15:57:05 DEBUG : rclone: Version "v1.54.1" starting with parameters ["rclone" "--s3-profile" "ybc-builds" "-vvvv" "lsd" "ybc-prod-engineering-useast1-ybcs-release:"]
2021/03/24 15:57:05 DEBUG : Using config file from "/home/cedgar/.config/rclone/rclone.conf"
2021/03/24 15:57:05 DEBUG : Creating backend with remote "ybc-prod-engineering-useast1-ybcs-release:"
2021/03/24 15:57:05 ERROR : : error listing: AccessDenied: Access Denied
        status code: 403, request id: DPJSDDK4J9K3YZXH, host id: bWcOW+qb/OXaL12T8y9+FIk39d51ToqERYUW4mVUOu7wdUJ7gzBfjVO/P3FKbBMnvr86Or/8kVI=
2021/03/24 15:57:05 DEBUG : 4 go routines active
2021/03/24 15:57:05 Failed to lsd with 2 errors: last error was: AccessDenied: Access Denied
        status code: 403, request id: DPJSDDK4J9K3YZXH, host id: bWcOW+qb/OXaL12T8y9+FIk39d51ToqERYUW4mVUOu7wdUJ7gzBfjVO/P3FKbBMnvr86Or/8kVI=

ncw · March 24, 2021, 5:24pm

I think this is fixed in the latest beta

see s3: fix --s3-profile which wasn't working - fixes #4757 · rclone/rclone@b9a015e · GitHub

bitlifter · March 25, 2021, 10:50pm

Thanks for responding. I just tested with v1.55.0-beta.5340.f6dbb98a1 and it is exhibiting the same behavior as 1.54.1. It works with --s3-profile default but not other profiles.

rclone --s3-profile default lsd endpoint: works
rclone --s3-profile account-a lsd endpoint: doesn't work
aws s3 ls --profile account-a works

Real world examples below:

rclone --s3-profile default  lsd ybc-prod-engineering-useast1-ybcs-release: -vvvv
2021/03/25 22:37:32 DEBUG : Using config file from "/home/cedgar/.config/rclone/rclone.conf"
2021/03/25 22:37:32 DEBUG : rclone: Version "v1.55.0-beta.5340.f6dbb98a1" starting with parameters ["rclone" "--s3-profile" "default" "lsd" "ybc-prod-engineering-useast1-ybcs-release:" "-vvvv"]
2021/03/25 22:37:32 DEBUG : Creating backend with remote "ybc-prod-engineering-useast1-ybcs-release:"
2021/03/25 22:37:32 DEBUG : ybc-prod-engineering-useast1-ybcs-release: detected overridden config - adding "{yEefc}" suffix to name
2021/03/25 22:37:32 DEBUG : fs cache: renaming cache item "ybc-prod-engineering-useast1-ybcs-release:" to be canonical "ybc-prod-engineering-useast1-ybcs-release{yEefc}:"
          -1 2020-05-29 16:29:37        -1 jrtest

rclone --s3-profile ybc-build  lsd ybc-prod-engineering-useast1-ybcs-release: -vvvv
2021/03/25 22:43:41 DEBUG : Using config file from "/home/cedgar/.config/rclone/rclone.conf"
2021/03/25 22:43:41 DEBUG : rclone: Version "v1.55.0-beta.5340.f6dbb98a1" starting with parameters ["rclone" "--s3-profile" "ybc-build" "lsd" "ybc-prod-engineering-useast1-ybcs-release:" "-vvvv"]
2021/03/25 22:43:41 DEBUG : Creating backend with remote "ybc-prod-engineering-useast1-ybcs-release:"
2021/03/25 22:43:41 DEBUG : ybc-prod-engineering-useast1-ybcs-release: detected overridden config - adding "{LizRq}" suffix to name
2021/03/25 22:43:41 DEBUG : fs cache: renaming cache item "ybc-prod-engineering-useast1-ybcs-release:" to be canonical "ybc-prod-engineering-useast1-ybcs-release{LizRq}:"
2021/03/25 22:43:41 ERROR : : error listing: AccessDenied: Access Denied
        status code: 403, request id: 1HK8B2SJGESXCW4F, host id: SSADivjSPd001tmB4B4WFjSxtpl5MfhpdxZrYZdGDglZVk4uyQebpxxsTnnxD9nc2UNkXOAQztc=
2021/03/25 22:43:41 DEBUG : 4 go routines active
2021/03/25 22:43:41 Failed to lsd with 2 errors: last error was: AccessDenied: Access Denied
        status code: 403, request id: 1HK8B2SJGESXCW4F, host id: SSADivjSPd001tmB4B4WFjSxtpl5MfhpdxZrYZdGDglZVk4uyQebpxxsTnnxD9nc2UNkXOAQztc=

aws s3 ls --profile ybc-build
2020-12-19 23:27:34 config-bucket-138760815728

ncw · March 29, 2021, 5:57pm

Rclone definitely uses the profile you name - I've just checked that with the beta by creating a copy of my profile and editing the original to be wrong.

What do you have in the profile - just

[default]
aws_access_key_id=XXX
aws_secret_access_key=XXX

Or do you have other things?

I'm not sure which takes precedence things from rclone's config or things from your profile.

Can you show your profile file with secrets XXX-ed out?

I'm pretty sure this is the issue you are having

And the fix for that should be in the beta

system · May 29, 2021, 1:58pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.