--s3-profile failing when explicit s3 endpoint is present

Suspected Bug
4

Continued testing just illustrates and affirms the problem. --s3-profile alone works fine. endpoint alone works fine. The two together fail.

I'm wondering if the interaction between the AWS credentials file and the AWS config file is entirely correct in the code. The rclone documentation often refers to a "profile" residing in the AWS credentials file, but the term "profile" is badly overloaded. The usual (AWS) usage is that the profile is in the AWS config file and the source_profile that the profile relies upon is in the AWS credentials file. The profile in config ties together a source_profile from credentials with a role in the config profile. The net effect is:

  1. The reference in CLI --profile, and rclone --s3-profile refers to the config profile entry
  2. The --profile or (hopefully) --s3-profile reads the source_profile entry in config, finds the corresponding section in credentials, then establishes a session using the keys in credentials and...
  3. Immediately uses that session to assume the role listed in config

That's enough to glaze anybody's eyes. Could we have possibly not quite gotten that right in rclone? Because the term profile is so overloaded that we really have two possible use cases with different interpretations of the word:

  1. The use case this ticket involves, where the indicated --s3-profile points to an entry in the config file which does the authentication and subsequent role assumption indicated above, and
  2. The much more common use case where --s3-profile merely refers to the heading of an entry in AWS credentials.

Maybe we have those two cases mixed up. But why that would interact with the presence or absence of a custom endpoint is a mystery.

I'll keep looking at the code, but don't really have a build toolchain to test anything in go.

Here's an example of the use case of concern (profile is in config, role is assumed after initial auth):

~/.aws/config file, has the profile, a pointer to a credential file entry and a role to assume

[profile dr-systems-storage-team]
role_arn = arn:aws:iam:999999999999:role/ROLE-RCLONE-WILL-ASSUME
source_profile = 888888888888-service-user-dr-systems-storageteam

~/.aws/credentials file has an entry that config's source_profile refers to

[888888888888-service-user-dr-systems-storageteam]
aws_access_key_id=AKIAstuff
aws_secret_access_key = ssssshItsASecret