Rclone not honoring AWS credential_process option

Suspected Bug
1

What is the problem you are having with rclone?

I'm trying to connect to AWS S3 with authentication via credential_process. I can't get rclone to pick up the option and get the credentials that way, although this method was mentioned as a workaround for SSO support in t/rclone-and-aws-sso-credentials/18477. For a minimal repro, I created a credential file as follows:

$ cat ~/.aws/credentials
[default]
credential_process = touch /tmp/credentials_requested

When I run aws s3 ls s3://test for example, the file I specified is created (i.e., the process is being executed). With rclone, I get (after some delay) an error message that no valid credential provides can be found, and the file is not created, i.e. the credential process has not been executed.

What is your rclone version (output from rclone version)

rclone v1.56.2
- os/version: darwin 11.6 (64 bit)
- os/kernel: 20.6.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.17.1
- go/linking: dynamic
- go/tags: none

Which OS you are using and how many bits (eg Windows 7, 64 bit)

macOS 11.6

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Any rclone command connecting to the remote, e.g. rclone ls s3:

The rclone config contents with secrets removed.

[s3]
type = s3
provider = AWS
env_auth = true
region = us-east-1

I haven't set any AWS_ or RCLONE_ environment variables, i.e. env_auth=true is just set to pick up the credentials file.

A log from the command with the -vv flag

$ rclone -vv ls s3:
2021/10/20 23:53:40 DEBUG : rclone: Version "v1.56.2" starting with parameters ["rclone" "-vv" "ls" "s3:"]
2021/10/20 23:53:40 DEBUG : Creating backend with remote "s3:"
2021/10/20 23:53:40 DEBUG : Using config file from "/Users/$USER/.config/rclone/rclone.conf"
2021/10/20 23:53:58 DEBUG : 5 go routines active
2021/10/20 23:53:58 Failed to ls: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2

hi,

if worked for that OP in that topic, did it not work for you?

based on a quick internet search, seems to be an issue with the aws go lang package and not a rclone bug.

for example,
https://serverfault.com/questions/973933/how-to-fix-nocredentialproviders-no-valid-providers-in-chain-deprecated/973934

3
4

Indeed, to be honest I don't know how the OP in that workaround managed to actually go for their suggestion. It's indeed an issue with the Go AWS SDK. The source file for the credential_process provider documents how this option can be used: aws-sdk-go/provider.go at main · aws/aws-sdk-go · GitHub. Unfortunately, simply using the shared credentials provider is not sufficient. It seems like supporting this option in rclone requires some manual work, so I opened a PR (linked in the previous comment).

5

when i hit an problem like that, i script it away.
if rclone cannot do it, i use the aws s3 sdk directly via the python boto3 package.

for example,
with aws s3, all my buckets have a IAM policy has MFA enabled.
rclone cannot create that TOTP token and feed to s3.
so my script

  1. creates the TOTP token
  2. feeds to s3 via boto3
  3. get back the session info
  4. using that info, create an on the fly remote.
  5. have rclone use that remote

if you want i could share that python code.

6

Yep, that's also an option of course. The concrete issue that credential_process is solving for me right now is that my SSO tokens (which I can also request and pass to rclone with a script) have a limited life-time, so if rclone takes a long time (which frequently happens), they expire and I have to restart it manually. The PR scratches my specific itch, but it's alright if rclone doesn't support every conceivable auth mechanism of every backend storage and provider :slight_smile: .