Shared_credentials_file config option not works

Suspected Bug
1

What is the problem you are having with rclone?

S3 auth not works when I'm using shared_credentials_file = /secrets/s3 config option.
But it works well if I set AWS_SHARED_CREDENTIALS_FILE=/secrets/s3 env variable.

P.S. Command line option representing same config option also not works for me.

What is your rclone version (output from rclone version)

rclone v1.54.0

  • os/arch: linux/amd64
  • go version: go1.15.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux x64. Official rclone docker image

Which cloud storage system are you using? (eg Google Drive)

Amazon S3 (Ceph provider)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone --config=/data/rclone.conf ls led:bucket -vv

The rclone config contents with secrets removed.

[gcs]
service_account_file = /secrets/gcs
type = google cloud storage

[led]
endpoint = https://s3.<mydomain>
profile = default
provider = Ceph
shared_credentials_file = /secrets/s3
env_auth = true
type = s3

A log from the command with the -vv flag

2021/02/15 00:23:13 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "--config=/data/rclone.conf" "ls" "led:bucket" "-vv"]
2021/02/15 00:23:13 DEBUG : Using config file from "/data/rclone.conf"
2021/02/15 00:23:13 DEBUG : Creating backend with remote "led:bucket"
2021/02/15 00:24:16 DEBUG : 2 go routines active
2021/02/15 00:24:16 Failed to ls: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2

Hmm, I can replicate this.

We do apparently tell the AWS SDK about the shared credentials file

That code looks correct but the SDK is ignoring it...

Setting "Profile" in that struct works... So I suspect an SDK bug.

Have you got time to dig in to the reason?

4

Have some debugging and found that it's actually rclone, not SDK issue.

Here's a block of code that brakes auth, as far as I understood. With these lines commented shared-credentials-file options works fine for me https://github.com/rclone/rclone/blob/5b84adf3b983f6208341ebcbbc3f8b3fbfccdb97/backend/s3/s3.go#L1559-L1565

But I don't understand why this part of code not break auth when ENV variable set.

5

Well done tracking that down.

I wonder if that is from a previous version of the SDK....

Did you try just commenting out this line

I think if you have explicitly set shared_credentials_file then it will use it, no need for env_auth.

6

Yep, I have commented that line and all worked fine.

7

Do you want to send a pull request deleting these 3 lines?

I think that is the correct fix.

8

I can but I'm not sure if this deletion will break anything. Suddenly right now I haven't got enough time to test all cases

9
10

Thank you for the PR. I've merged it.

I'm reasonably sure this is left over from a previous version of the SDK as my tests seem to work :crossed_fingers:

11

Yep, integration test for that during CI could catch such problems, but it's not very easy. :frowning:

12

There are a huge number of combinations of AWS auth methods! We integration test some of them, but we can't do all of them :frowning:

13

IAM based auth methods is not easy to test automatically. But I think that shared-credentials-file test scenario should be too complex if you have access_key_id/secret_access_key, isn't it?

1 Like
14

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.