What is the problem you are having with rclone?
I am trying to use rclone to sync two AWS S3 buckets in different AWS accounts and diff regions. This container is running in AWS ECS in the region us-west-1
, the target S3 bucket is in the same account and region where ECS runs.
The auth to the source S3 bucket using access_key_id and secret_access_key works fine. However the auth to the target S3 bucket using env_auth = true
and using the ECS IAM Execution role does not work and gives the error Failed to ls: NoCredentialProviders: no valid providers in chain. Deprecated. For verbose messaging see aws.Config.CredentialsChainVerboseErrors
If I want to use the ECS IAM Execution role do I need to include env_auth = true
?
From the documentation I think I do, since it will check your aws_profile and then try the ec2 instance_profile, and then I was hoping it would try to use the ECS execution_profile.
I am building my own container and creating the creds when the container runs with a bash script and using environment variables for the access key and secret key.
FROM ubuntu:22.04
ENV TZ=America/New_York
WORKDIR /opt/
COPY *.sh .
RUN chmod +x *.sh
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone && \
apt-get -yqq update && \
apt-get -yqq install curl unzip && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
RUN curl https://rclone.org/install.sh | bash
ENTRYPOINT ["/opt/entrypoint.sh"]
contents of file /opt/entrypoint.sh
./generate_credentials_file.sh
rclone --version
$@
contents of file /opt/generate_credentials_file.sh
#!/bin/sh
cat > creds.cfg << ENDOFFILE
# creds.cfg:
[source]
type = s3
provider = AWS
access_key_id = $RCLONE_SOURCE_ACCESS_KEY
secret_access_key = $RCLONE_SOURCE_SECRET_KEY
region = us-east-2
[target]
type = s3
provider = AWS
env_auth = true
region = us-west-1
ENDOFFILE
'rclone version'
rclone v1.62.2 ecs_cli_rclone
os/version: ubuntu 22.04 (64 bit) ecs_cli_rclone
os/kernel: 5.10.178-162.673.amzn2.x86_64 (x86_64) ecs_cli_rclone
os/type: linux ecs_cli_rclone
os/arch: amd64 ecs_cli_rclone
go/version: go1.20.2 ecs_cli_rclone
go/linking: static ecs_cli_rclone
go/tags: none
Which cloud storage system are you using? (eg Google Drive)
AWS S3
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone --progress -vv --dump bodies --retries 1 --config=/opt/creds.cfg sync source:source-bucket target:target-bucket
Elapsed time: 2m20.4s ecs_cli_rclone
2023-05-16 13:28:55 ERROR : S3 bucket target-bucket path test: error reading destination root directory: NoCredentialProviders: no valid providers in chain. Deprecated. ecs_cli_rclone
For verbose messaging see aws.Config.CredentialsChainVerboseErrors ecs_cli_rclone
Transferred: 0 B / 0 B, -, 0 B/s, ETA - ecs_cli_rclone
Elapsed time: 2m20.4s ecs_cli_rclone
2023-05-16 13:28:55 DEBUG : S3 bucket target-bucket path test: Waiting for checks to finish ecs_cli_rclone
Transferred: 0 B / 0 B, -, 0 B/s, ETA - ecs_cli_rclone
Errors: 1 (retrying may help) ecs_cli_rclone
Elapsed time: 2m20.4s ecs_cli_rclone
2023-05-16 13:28:55 DEBUG : S3 bucket target-bucket path test: Waiting for transfers to finish ecs_cli_rclone
Transferred: 0 B / 0 B, -, 0 B/s, ETA - ecs_cli_rclone
Errors: 1 (retrying may help) ecs_cli_rclone
Elapsed time: 2m20.4s ecs_cli_rclone
2023-05-16 13:28:55 ERROR : S3 bucket target-bucket path test: not deleting files as there were IO errors ecs_cli_rclone
Transferred: 0 B / 0 B, -, 0 B/s, ETA - ecs_cli_rclone
Errors: 1 (retrying may help) ecs_cli_rclone
Elapsed time: 2m20.4s ecs_cli_rclone
2023-05-16 13:28:55 ERROR : S3 bucket target-bucket path test: not deleting directories as there were IO errors ecs_cli_rclone
Transferred: 0 B / 0 B, -, 0 B/s, ETA - ecs_cli_rclone
Errors: 1 (retrying may help) ecs_cli_rclone
Elapsed time: 2m20.4s
The rclone config contents with secrets removed.
[source]
type = s3
provider = AWS
access_key_id = ACCESS_KEY
secret_access_key = SECRET_KEY
region = us-east-2
[target]
type = s3
provider = AWS
env_auth = true
region = us-west-1
rclone -vv --config=/opt/creds.cfg ls source:
message
rclone v1.62.2
'"- os/version: ubuntu 22.04 (64 bit)"
'"- os/kernel: 5.10.178-162.673.amzn2.x86_64 (x86_64)"
'"- os/type: linux"
'"- os/arch: amd64"
'"- go/version: go1.20.2"
'"- go/linking: static"
'"- go/tags: none"
"2023/05/16 14:14:00 DEBUG : rclone: Version ""v1.62.2"" starting with parameters [""rclone"" ""-vv"" ""--config=/opt/creds.cfg"" ""ls"" ""source:""]"
"2023/05/16 14:14:00 DEBUG : Creating backend with remote ""source:"""
"2023/05/16 14:14:00 DEBUG : Using config file from ""/opt/creds.cfg"""
4668 sensitive-file-name-2020-09.csv.zip
4806 sensitive-file-name-2020-10.csv.zip
2351 sensitive-file-name-2020-11.csv.zip
2765 sensitive-file-name-2020-12.csv.zip
2612 sensitive-file-name-2021-01.csv.zip
2753 sensitive-file-name-2021-02.csv.zip
2719 sensitive-file-name-2021-03.csv.zip
2488 sensitive-file-name-2021-04.csv.zip
2557 sensitive-file-name-2021-05.csv.zip
2898 sensitive-file-name-2021-06.csv.zip
2993 sensitive-file-name-2021-07.csv.zip
2957 sensitive-file-name-2021-08.csv.zip
2596 sensitive-file-name-2021-09.csv.zip
2579 sensitive-file-name-2021-10.csv.zip
2651 sensitive-file-name-2021-11.csv.zip
2627 sensitive-file-name-2021-12.csv.zip
2476 sensitive-file-name-2022-01.csv.zip
3968 sensitive-file-name-2022-02.csv.zip
2595 sensitive-file-name-2022-03.csv.zip
2496 sensitive-file-name-2022-04.csv.zip
2498 sensitive-file-name-2022-05.csv.zip
1677 sensitive-file-name-2022-06.csv.zip
2583 sensitive-file-name-2022-07.csv.zip
3120 sensitive-file-name-2022-08.csv.zip
2439 sensitive-file-name-2022-09.csv.zip
2531 sensitive-file-name-2022-10.csv.zip
2810 sensitive-file-name-2022-11.csv.zip
2497 sensitive-file-name-2022-12.csv.zip
2578 sensitive-file-name-2023-01.csv.zip
2475 sensitive-file-name-2023-02.csv.zip
3750 sensitive-file-name-2023-04.csv.zip
7501 sensitive-file-name-2023-05.csv.zip
2266087384 sensitive-file-name-2020-09.csv.zip
2897709850 sensitive-file-name-2020-10.csv.zip
2766905098 sensitive-file-name-2020-11.csv.zip
2722475632 sensitive-file-name-2020-12.csv.zip
3056768811 sensitive-file-name-2021-01.csv.zip
3552192077 sensitive-file-name-2021-02.csv.zip
3944198109 sensitive-file-name-2021-03.csv.zip
3839475100 sensitive-file-name-2021-04.csv.zip
4124190524 sensitive-file-name-2021-05.csv.zip
3934521995 sensitive-file-name-2021-06.csv.zip
4085312153 sensitive-file-name-2021-07.csv.zip
3966093772 sensitive-file-name-2021-08.csv.zip
4272796613 sensitive-file-name-2021-09.csv.zip
4362198848 sensitive-file-name-2021-10.csv.zip
4458582471 sensitive-file-name-2021-11.csv.zip
4939010010 sensitive-file-name-2021-12.csv.zip
4821279389 sensitive-file-name-2022-01.csv.zip
4499496878 sensitive-file-name-2022-02.csv.zip
5142775149 sensitive-file-name-2022-03.csv.zip
4899893864 sensitive-file-name-2022-04.csv.zip
5108559017 sensitive-file-name-2022-05.csv.zip
2717704377 sensitive-file-name-2022-06.csv.zip
5170410271 sensitive-file-name-2022-07.csv.zip
5261278204 sensitive-file-name-2022-08.csv.zip
4851825596 sensitive-file-name-2022-09.csv.zip
5034378479 sensitive-file-name-2022-10.csv.zip
4788425831 sensitive-file-name-2022-11.csv.zip
4616708126 sensitive-file-name-2022-12.csv.zip
86408636 sensitive-file-name-2023-01.csv.zip
4087426268 sensitive-file-name-2023-02.csv.zip
4466516552 sensitive-file-name-2023-03.csv.zip
4053636475 sensitive-file-name-2023-04.csv.zip
2198124310 sensitive-file-name-2023-05.csv.zip
4087426250 sensitive-file-name-2023-02.csv.zip
4466516534 sensitive-file-name-2023-03.csv.zip
4053636457 sensitive-file-name-2023-04.csv.zip
2198124292 sensitive-file-name-2023-05.csv.zip
2023/05/16 14:14:01 DEBUG : 8 go routines active
rclone -vv --config=/opt/creds.cfg ls target:
rclone v1.62.2
'"- os/version: ubuntu 22.04 (64 bit)"
'"- os/kernel: 5.10.178-162.673.amzn2.x86_64 (x86_64)"
'"- os/type: linux"
'"- os/arch: amd64"
'"- go/version: go1.20.2"
'"- go/linking: static"
'"- go/tags: none"
rclone: Version ""v1.62.2"" starting with parameters [""rclone"" ""-vv"" ""--config=/opt/creds.cfg"" ""ls"" ""target:""]"
Creating backend with remote ""target:"""
Using config file from ""/opt/creds.cfg"""
2 go routines active
Failed to ls: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors