Rclone S3 sync running in ECS between different accounts - issue with accessing account using IAM execution role

Help and Support
1

What is the problem you are having with rclone?

I am trying to use rclone to sync two AWS S3 buckets in different AWS accounts and diff regions. This container is running in AWS ECS in the region us-west-1, the target S3 bucket is in the same account and region where ECS runs.
The auth to the source S3 bucket using access_key_id and secret_access_key works fine. However the auth to the target S3 bucket using env_auth = true and using the ECS IAM Execution role does not work and gives the error Failed to ls: NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors

If I want to use the ECS IAM Execution role do I need to include env_auth = true?
From the documentation I think I do, since it will check your aws_profile and then try the ec2 instance_profile, and then I was hoping it would try to use the ECS execution_profile.

I am building my own container and creating the creds when the container runs with a bash script and using environment variables for the access key and secret key.

FROM ubuntu:22.04

ENV TZ=America/New_York

WORKDIR /opt/

COPY *.sh .

RUN chmod +x *.sh

RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone && \
    apt-get -yqq update && \
    apt-get -yqq install curl unzip && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*
RUN curl https://rclone.org/install.sh | bash

ENTRYPOINT ["/opt/entrypoint.sh"]

contents of file /opt/entrypoint.sh

./generate_credentials_file.sh

rclone --version

$@

contents of file /opt/generate_credentials_file.sh

#!/bin/sh

cat > creds.cfg << ENDOFFILE
# creds.cfg:
[source]
type = s3
provider = AWS
access_key_id = $RCLONE_SOURCE_ACCESS_KEY
secret_access_key = $RCLONE_SOURCE_SECRET_KEY
region = us-east-2

[target]
type = s3
provider = AWS
env_auth = true
region = us-west-1
ENDOFFILE

'rclone version'

rclone v1.62.2	ecs_cli_rclone
 os/version: ubuntu 22.04 (64 bit)	ecs_cli_rclone
 os/kernel: 5.10.178-162.673.amzn2.x86_64 (x86_64)	ecs_cli_rclone
 os/type: linux	ecs_cli_rclone
 os/arch: amd64	ecs_cli_rclone
 go/version: go1.20.2	ecs_cli_rclone
 go/linking: static	ecs_cli_rclone
 go/tags: none

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone --progress -vv --dump bodies --retries 1 --config=/opt/creds.cfg sync source:source-bucket target:target-bucket
Elapsed time: 2m20.4s	ecs_cli_rclone
2023-05-16 13:28:55 ERROR : S3 bucket target-bucket path test: error reading destination root directory: NoCredentialProviders: no valid providers in chain. Deprecated.	ecs_cli_rclone
For verbose messaging see aws.Config.CredentialsChainVerboseErrors	ecs_cli_rclone
Transferred: 0 B / 0 B, -, 0 B/s, ETA -	ecs_cli_rclone
Elapsed time: 2m20.4s	ecs_cli_rclone
2023-05-16 13:28:55 DEBUG : S3 bucket target-bucket path test: Waiting for checks to finish	ecs_cli_rclone
Transferred: 0 B / 0 B, -, 0 B/s, ETA -	ecs_cli_rclone
Errors: 1 (retrying may help)	ecs_cli_rclone
Elapsed time: 2m20.4s	ecs_cli_rclone
2023-05-16 13:28:55 DEBUG : S3 bucket target-bucket path test: Waiting for transfers to finish	ecs_cli_rclone
Transferred: 0 B / 0 B, -, 0 B/s, ETA -	ecs_cli_rclone
Errors: 1 (retrying may help)	ecs_cli_rclone
Elapsed time: 2m20.4s	ecs_cli_rclone
2023-05-16 13:28:55 ERROR : S3 bucket target-bucket path test: not deleting files as there were IO errors	ecs_cli_rclone
Transferred: 0 B / 0 B, -, 0 B/s, ETA -	ecs_cli_rclone
Errors: 1 (retrying may help)	ecs_cli_rclone
Elapsed time: 2m20.4s	ecs_cli_rclone
2023-05-16 13:28:55 ERROR : S3 bucket target-bucket path test: not deleting directories as there were IO errors	ecs_cli_rclone
Transferred: 0 B / 0 B, -, 0 B/s, ETA -	ecs_cli_rclone
Errors: 1 (retrying may help)	ecs_cli_rclone
Elapsed time: 2m20.4s

The rclone config contents with secrets removed.

[source]
type = s3
provider = AWS
access_key_id = ACCESS_KEY
secret_access_key = SECRET_KEY
region = us-east-2

[target]
type = s3
provider = AWS
env_auth = true
region = us-west-1

rclone -vv --config=/opt/creds.cfg ls source:

message
rclone v1.62.2
'"- os/version: ubuntu 22.04 (64 bit)"
'"- os/kernel: 5.10.178-162.673.amzn2.x86_64 (x86_64)"
'"- os/type: linux"
'"- os/arch: amd64"
'"- go/version: go1.20.2"
'"- go/linking: static"
'"- go/tags: none"
"2023/05/16 14:14:00 DEBUG : rclone: Version ""v1.62.2"" starting with parameters [""rclone"" ""-vv"" ""--config=/opt/creds.cfg"" ""ls"" ""source:""]"
"2023/05/16 14:14:00 DEBUG : Creating backend with remote ""source:"""
"2023/05/16 14:14:00 DEBUG : Using config file from ""/opt/creds.cfg"""
     4668 sensitive-file-name-2020-09.csv.zip
     4806 sensitive-file-name-2020-10.csv.zip
     2351 sensitive-file-name-2020-11.csv.zip
     2765 sensitive-file-name-2020-12.csv.zip
     2612 sensitive-file-name-2021-01.csv.zip
     2753 sensitive-file-name-2021-02.csv.zip
     2719 sensitive-file-name-2021-03.csv.zip
     2488 sensitive-file-name-2021-04.csv.zip
     2557 sensitive-file-name-2021-05.csv.zip
     2898 sensitive-file-name-2021-06.csv.zip
     2993 sensitive-file-name-2021-07.csv.zip
     2957 sensitive-file-name-2021-08.csv.zip
     2596 sensitive-file-name-2021-09.csv.zip
     2579 sensitive-file-name-2021-10.csv.zip
     2651 sensitive-file-name-2021-11.csv.zip
     2627 sensitive-file-name-2021-12.csv.zip
     2476 sensitive-file-name-2022-01.csv.zip
     3968 sensitive-file-name-2022-02.csv.zip
     2595 sensitive-file-name-2022-03.csv.zip
     2496 sensitive-file-name-2022-04.csv.zip
     2498 sensitive-file-name-2022-05.csv.zip
     1677 sensitive-file-name-2022-06.csv.zip
     2583 sensitive-file-name-2022-07.csv.zip
     3120 sensitive-file-name-2022-08.csv.zip
     2439 sensitive-file-name-2022-09.csv.zip
     2531 sensitive-file-name-2022-10.csv.zip
     2810 sensitive-file-name-2022-11.csv.zip
     2497 sensitive-file-name-2022-12.csv.zip
     2578 sensitive-file-name-2023-01.csv.zip
     2475 sensitive-file-name-2023-02.csv.zip
     3750 sensitive-file-name-2023-04.csv.zip
     7501 sensitive-file-name-2023-05.csv.zip
2266087384 sensitive-file-name-2020-09.csv.zip
2897709850 sensitive-file-name-2020-10.csv.zip
2766905098 sensitive-file-name-2020-11.csv.zip
2722475632 sensitive-file-name-2020-12.csv.zip
3056768811 sensitive-file-name-2021-01.csv.zip
3552192077 sensitive-file-name-2021-02.csv.zip
3944198109 sensitive-file-name-2021-03.csv.zip
3839475100 sensitive-file-name-2021-04.csv.zip
4124190524 sensitive-file-name-2021-05.csv.zip
3934521995 sensitive-file-name-2021-06.csv.zip
4085312153 sensitive-file-name-2021-07.csv.zip
3966093772 sensitive-file-name-2021-08.csv.zip
4272796613 sensitive-file-name-2021-09.csv.zip
4362198848 sensitive-file-name-2021-10.csv.zip
4458582471 sensitive-file-name-2021-11.csv.zip
4939010010 sensitive-file-name-2021-12.csv.zip
4821279389 sensitive-file-name-2022-01.csv.zip
4499496878 sensitive-file-name-2022-02.csv.zip
5142775149 sensitive-file-name-2022-03.csv.zip
4899893864 sensitive-file-name-2022-04.csv.zip
5108559017 sensitive-file-name-2022-05.csv.zip
2717704377 sensitive-file-name-2022-06.csv.zip
5170410271 sensitive-file-name-2022-07.csv.zip
5261278204 sensitive-file-name-2022-08.csv.zip
4851825596 sensitive-file-name-2022-09.csv.zip
5034378479 sensitive-file-name-2022-10.csv.zip
4788425831 sensitive-file-name-2022-11.csv.zip
4616708126 sensitive-file-name-2022-12.csv.zip
 86408636 sensitive-file-name-2023-01.csv.zip
4087426268 sensitive-file-name-2023-02.csv.zip
4466516552 sensitive-file-name-2023-03.csv.zip
4053636475 sensitive-file-name-2023-04.csv.zip
2198124310 sensitive-file-name-2023-05.csv.zip
4087426250 sensitive-file-name-2023-02.csv.zip
4466516534 sensitive-file-name-2023-03.csv.zip
4053636457 sensitive-file-name-2023-04.csv.zip
2198124292 sensitive-file-name-2023-05.csv.zip
2023/05/16 14:14:01 DEBUG : 8 go routines active

rclone -vv --config=/opt/creds.cfg ls target:

rclone v1.62.2
'"- os/version: ubuntu 22.04 (64 bit)"
'"- os/kernel: 5.10.178-162.673.amzn2.x86_64 (x86_64)"
'"- os/type: linux"
'"- os/arch: amd64"
'"- go/version: go1.20.2"
'"- go/linking: static"
'"- go/tags: none"
rclone: Version ""v1.62.2"" starting with parameters [""rclone"" ""-vv"" ""--config=/opt/creds.cfg"" ""ls"" ""target:""]"
Creating backend with remote ""target:"""
Using config file from ""/opt/creds.cfg"""
2 go routines active
Failed to ls: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
2

If you want to use any auth which isn't in the config file you need env_auth = true. Provided you've got it set up properly this should be enough to run as an ECS task with an IAM role.

I don't know if that is different to a "ECS IAM Execution role" though.

That sounds like rclone failed to pick up any auth from the environment so I'd guess it the role isn't quite right or isn't being applied.

3

I was thinking the same, I think the easiest way to test the ECS execution role would be to install aws_cli in that container and test aws s3 ls.

4

Rclone uses the aws go SDK so picking up the creds usually just works.

5

Everything looks correct assuming that your ECS Task Execution IAM policy looks correct for S3.

There is a bit of a difference with ECS vs regular EC2 metadata in that it is found on a different internal address at 169.254.170.2 vs 169.254.169.254.

I looked at the AWS Go SDK and I do see there's a reference to the ECS Metadata URL at the correct location.

6

ECS support was contributed in this commit. I think it has been working since then thought the code may have changed a bit. I don't explicitly run integration tests for ECS though.

7

Thanks for the replies, I'm adding aws cli to test the IAM ECS execution role.

this is the IAM ECS execution role's policy

{
    "Statement": [
        {
            "Action": [
                "secretsmanager:GetSecretValue",
                "kms:Decrypt"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:secretsmanager:us-west-1:xxxxxxxxxxxxxx:secret:secret-id",
                "arn:aws:secretsmanager:us-west-1:xxxxxxxxxxxxxx:secret:secret-id",
                "arn:aws:secretsmanager:us-west-1:xxxxxxxxxxxxxx:secret:secret-id",
                "arn:aws:kms:us-west-1:xxxxxxxxxxxxxx:key/key-id",
                "arn:aws:kms:us-west-1:xxxxxxxxxxxxxx:key/key-id"
            ],
            "Sid": ""
        },
        {
            "Action": "s3:*",
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::target-bucket/*",
                "arn:aws:s3:::target-bucket"
            ],
            "Sid": ""
        },
        {
            "Action": [
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ],
    "Version": "2012-10-17"
}
8

Thank you all for the help.
I figured out the issue, in my Terraform to deploy the ECS tasks I did have the ecs_execution_role attached to the ecs task, but I did not attached a ecs_task_role to the task. This is working now!!!

1 Like
9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.