Rclone and AWS SSO credentials

Help and Support
#1

What is the problem you are having with rclone?

It does not seem to work with AWS SSO credentials

What is your rclone version (output from rclone version)

    ./rclone version
    rclone v1.52.2-281-g324077fb-beta
    - os/arch: darwin/amd64
    - go version: go1.14.6

Which OS you are using and how many bits (eg Windows 7, 64 bit)

macOS 10.15.6

Which cloud storage system are you using? (eg Google Drive)

S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

./rclone --s3-profile my-profile -vvvvvvv ls s3:mybucket

The rclone config contents with secrets removed.

[s3]
type = s3
env_auth = true
provider = AWS
region = us-west-2

A log from the command with the -vv flag

2020/08/11 09:42:14 DEBUG : rclone: Version "v1.52.2-281-g324077fb-beta" starting with parameters ["./rclone" "--s3-profile" "my-profile" "-vvvvvvv" "ls" "s3:mybucket"]
2020/08/11 09:42:14 DEBUG : Using config file from "/Users/me/.config/rclone/rclone.conf"

Also worth noting, using the default aws cli, this works as expected:

aws s3 ls

For additional context, I have AWS_PROFILE set as an env var.

#2

hello and welcome to the forum,

what is the output of rclone ls s3:?
i know that there is a beta that when using an environment variable, will display the value in the debug log.
perhaps the feature made it into the latest stable, 1.52.3

#3

Here is the output from that command:

./rclone --s3-profile my-profile -vvvvvvv ls s3:
2020/08/12 14:11:25 DEBUG : rclone: Version "v1.52.2-281-g324077fb-beta" starting with parameters ["./rclone" "--s3-profile" "my-profile" "-vvvvvvv" "ls" "s3:"]
2020/08/12 14:11:25 DEBUG : Using config file from "/Users/me/.config/rclone/rclone.conf"


2020/08/12 14:13:25 DEBUG : 4 go routines active
2020/08/12 14:13:25 Failed to ls: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
#4

Can you show what env vars you've got set? Is it just AWS_PROFILE?

I'm not sure you need to set --s3-profile as well maybe setting them both is confusing rclone somehow.

#5

I've tried it both ways with the same error. Only AWS_PROFILE is set in my env.

./rclone -vvvvvvv ls s3:

2020/08/12 18:27:02 DEBUG : rclone: Version "v1.52.2-281-g324077fb-beta" starting with parameters ["./rclone" "-vvvvvvv" "ls" "s3:"]
2020/08/12 18:27:02 DEBUG : Using config file from "/Users/me/.config/rclone/rclone.conf"
2020/08/12 18:29:02 DEBUG : 4 go routines active
2020/08/12 18:29:02 Failed to ls: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

env:

TERM_SESSION_ID=w0t0p0:C2C6C4FF-75BF-4F94-A0B1-820A48E4100F
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.6BrQHMW6cP/Listeners
LC_TERMINAL_VERSION=3.3.12
COLORFGBG=15;0
ITERM_PROFILE=Default
XPC_FLAGS=0x0
LANG=en_US.UTF-8
PWD=/Users/me/Desktop/BigSurTesting/Catalina
SHELL=/bin/zsh
TERM_PROGRAM_VERSION=3.3.12
TERM_PROGRAM=iTerm.app
PATH=/Users/me/.pyenv/bin:/Users/me/.rbenv/shims:/Applications/Postgres.app/Contents/Versions/latest/bin:/Library/Frameworks/Python.framework/Versions/3.7/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion Tech Preview.app/Contents/Public:/usr/local/go/bin:/usr/local/munki:/usr/local/vfuse/bin:/Applications/Visual Studio Code.app/Contents/Resources/app/bin:/usr/local/go/bin:/Users/me/gocode/bin
LC_TERMINAL=iTerm2
COLORTERM=truecolor
TERM=xterm-256color
HOME=/Users/me
TMPDIR=/var/folders/8_/lrkf29dn27s9sqme_5_wyqbw0000gn/T/
USER=me
XPC_SERVICE_NAME=0
LOGNAME=me
ITERM_SESSION_ID=w0t0p0:C2C6C4FF-75BF-4F94-A0B1-820A48E4100F
__CF_USER_TEXT_ENCODING=0x0:0:0
SHLVL=1
OLDPWD=/Users/me/Desktop/BigSurTesting
KEYTIMEOUT=1
EDITOR=/usr/bin/vim
MYZSH=/Users/me/code/dotfiles/zsh
GOPATH=/Users/me/gocode
NVM_DIR=/Users/nate/.nvm
RBENV_SHELL=zsh
VIRTUAL_ENV=/Users/me/.pyenv
#6

I just tried this locally. I made my default profile fail and added a new test profile

Both rclone v1.52.3 and the beta seem to work like this (I didn't have vars set in the env for the test)

AWS_PROFILE=test rclone lsd s3env:
# OK

--s3-profile is a beta feature but it doesn't appear to be working :frowning:

rclone --s3-profile test lsd s3env:
# FAIL

I'm not sure why the --s3-profile flag doesn't work yet - it looks like a bug in the SDK...

I note from this issue that the SDK only supports reading hardcoded access/secret from the profile - what do you have in your profile?

#7

I do not have a ~/.aws/credentials file, I only have ~/.aws/config.

I tried copying config to credentials and it gave the same messages as above.

I am configuring my profile using aws configure sso. The aws CLI command seems to be able to find these credentials and use them, so I'm not sure what the difference is.

~/.aws/config looks like this:

[profile my-profile]
sso_start_url = https://company.somedomain.com
sso_region = us-west-2
sso_account_id = 123456789
sso_role_name = MyRoleName
region = us-west-2
output = json
#8

I'm not familiar with aws SSO.

I haven't found any docs about it yet, need to use the power of grep...

Rclone uses the official AWS SDK so it can probably do it if it is configured correctly.

Are you using SSO from within ec2 or just on your laptop?

Add support for AWS SSO
#9

This is local on my laptop.

#10

Apparently the SDK does not currently support SSO credentials out of the box. Using this project allowed rclone to work as expected: https://github.com/benkehoe/aws-sso-credential-process

Once the SDK supports SSO properly, it should just work, but until then this project seems to be an ok solution :slight_smile:

#11

Thanks for working this out. I was thinking SSO might not be supported but I couldn't find any issues in the AWS Go SDK about that.

#12

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.