S3 with AWS IAM roles anywhere

Help and Support
1

I have installed the latest version of rclone and I am trying to use aws s3 profile stored in aws config, I can successfully list the buckets with ```aws s3 ls` --profile XXXX-XXXX-XXX-XXXX-XXX `` but not with rclone and I get the below error. Please can you help.

~/.aws/config


[profile XXXX-XXXX-XXX-XXXX-XXX]
credential_process = /usr/local/bin/aws_signing_helper credential-process --certificate /home/XXXX/.aws/XXXX.pem --private-key /home/XXXX/.aws/XXXXX.key --trust-anchor-arn xxxxxx   --role-arn xxxxx--profile-arn xxxxx --with-proxy |  jq -r '. | "export AWS_ACCESS_KEY_ID=\(.AccessKeyId)\nexport AWS_SECRET_ACCESS_KEY=\(.SecretAccessKey)\nexport AWS_SESSION_TOKEN=\(.SessionToken)\n"'

~/.config/rclone/rclone.conf:


[s3-XX-backups]
type = s3
provider = AWS
env_auth = true
region = eu-west-2
location_constraint = EU
server_side_encryption = AES256
storage_class = ONEZONE_IA
bucket_acl = private
sse_customer_algorithm = AES256
profile = XXXX-XXXX-XXX-XXXX-XXX

error with rclone ls:

rclone ls s3-XX-backups:/__bucket_name__/
2023/06/07 15:57:00 Failed to ls: ProcessProviderParseError: parse failed of credential_process output: export 
export AWS_ACCESS_KEY_ID=xXXXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=XXXXXXX
export AWS_SESSION_TOKEN=XXXXXXXXX
caused by: invalid character 'e' looking for the beginning of the value
2

Have you read documentation?

how do you think rclone can guess where your shared credentials file is? You do not specify location so default one is used. Check what it is.

3

Run this with -vv --dump bodies and what is going on might be clearer.

4

@ncw , please find the below, redacted some sensitive information

2023/06/14 12:54:39 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "ls" "s3-XX-backups:/bucket t" "-vv" "--dump" "bodies"]

2023/06/14 12:54:39 DEBUG : Creating backend with remote "s3-XX-backups:/bucket"

2023/06/14 12:54:39 DEBUG : Using config file from "/home/XXXXX/.config/rclone/rclone.conf"

2023/06/14 12:54:39 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.

2023/06/14 12:54:39 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.

2023/06/14 12:54:39 DEBUG : fs cache: renaming cache item "s3-XX-backups:/bucket" to be canonical "s3-XX-backups:bucket"

2023/06/14 12:54:39 DEBUG : 2 go routines active

2023/06/14 12:54:39 Failed to ls: ProcessProviderParseError: parse failed of credential_process output:
export AWS_ACCESS_KEY_ID=....
export AWS_SECRET_ACCESS_KEY=...
export AWS_SESSION_TOKEN=....

caused by: invalid character 'e' looking for beginning of value

5

It is complaining about the e of the export lines I think.

I'm not familiar with the credential_process method but I think the SDK isn't expecting export NAME=VALUE lines.

According to this page the output should look like JSON

Expected output from the Credentials program

The AWS CLI runs the command as specified in the profile and then reads data from STDOUT. The command you specify must generate JSON output on STDOUT that matches the following syntax.

{
  "Version": 1,
  "AccessKeyId": "an AWS access key",
  "SecretAccessKey": "your AWS secret access key",
  "SessionToken": "the AWS session token for temporary credentials",
  "Expiration": "ISO8601 timestamp when the credentials expire",
 }

I think this is the native output of the aws_signing_helper program, so I would have thought something like this would work better

[profile XXXX-XXXX-XXX-XXXX-XXX]
credential_process = /usr/local/bin/aws_signing_helper credential-process --certificate /home/XXXX/.aws/XXXX.pem --private-key /home/XXXX/.aws/XXXXX.key --trust-anchor-arn xxxxxx   --role-arn xxxxx--profile-arn xxxxx --with-proxy
6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.