RC sync/sync fails when configuring rclone rcd with environment variables

thomasjm · September 10, 2025, 11:26pm

What is the problem you are having with rclone?

I’m trying to run an rclone rcd session which is configured entirely with environment variables. The environment variables are meant to create two remotes named src and dst. src is local, and dst is an Amazon S3 remote.

RCLONE_CONFIG="" RCLONE_CONFIG_SRC_TYPE="local" RCLONE_CONFIG_DST_TYPE="s3" RCLONE_CONFIG_DST_ACCESS_KEY_ID="..." RCLONE_CONFIG_DST_DIRECTORY_MARKERS="true" RCLONE_CONFIG_DST_LOCATION_CONSTRAINT="us-west-2" RCLONE_CONFIG_DST_PROVIDER="AWS" RCLONE_CONFIG_DST_REGION="us-west-2" RCLONE_CONFIG_DST_SECRET_ACCESS_KEY="..." rclone rcd --rc-no-auth -vv --dump requests,responses,bodies,headers

Then, in a separate terminal:

rclone rc sync/sync srcFs=src:/Users/tom/test1  dstFs=dst:tom-rclone-test/localtest1/tom_test14

This fails with an error about the user not being authorized to perform “s3:CreateBucket”. However, the bucket already exists so it shouldn’t be trying to create it.

Also, if I set up an actual rclone.conf config file with the equivalent config, then the sync command does work, either via rclone sync or rclone rc sync/sync .

So, I’m starting to think that something is going wrong with the environment variable setup. You can see that something odd is happening with repeated lines in the logs below like Setting region="us-west-2" for "dst" from environment variable RCLONE_CONFIG_DST_REGION. And also the line that says dst: detected overridden config - adding "{K4O1W}" suffix to name .

Run the command 'rclone version' and share the full output of the command.

❯ rclone version
rclone v1.71.0

os/version: darwin 15.5 (64 bit)
os/kernel: 24.5.0 (arm64)
os/type: darwin
os/arch: arm64 (ARMv8 compatible)
go/version: go1.25.0
go/linking: dynamic
go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Amazon S3.

The rclone config contents with secrets removed.

<no config>

A log from the command with the `-vv` flag

The log output is long, but the relevant parts seem to be here.

2025/09/10 15:58:36 DEBUG : Setting --config "" from environment variable RCLONE_CONFIG=""
2025/09/10 15:58:36 NOTICE: Automatically setting -vv as --dump is enabled
2025/09/10 15:58:36 DEBUG : rclone: Version "v1.71.0" starting with parameters ["rclone" "rcd" "--rc-no-auth" "-vv" "--dump" "requests,responses,bodies,headers"]
2025/09/10 15:58:36 NOTICE: Serving remote control on http://127.0.0.1:5572/
2025/09/10 15:58:50 DEBUG : rc: "sync/sync": with parameters map[dstFs:dst:tom-rclone-test/localtest1/tom_test14 srcFs:src:/Users/tom/test1]
2025/09/10 15:58:50 DEBUG : Creating backend with remote "src:/Users/tom/test1"
2025/09/10 15:58:50 DEBUG : Using config file from ""
2025/09/10 15:58:50 DEBUG : Setting type="local" for "src" from environment variable RCLONE_CONFIG_SRC_TYPE
2025/09/10 15:58:50 DEBUG : Creating backend with remote "dst:tom-rclone-test/localtest1/tom_test14"
2025/09/10 15:58:50 DEBUG : Setting type="s3" for "dst" from environment variable RCLONE_CONFIG_DST_TYPE
2025/09/10 15:58:50 DEBUG : Setting provider="AWS" for "dst" from environment variable RCLONE_CONFIG_DST_PROVIDER
2025/09/10 15:58:50 DEBUG : Setting access_key_id="..." for "dst" from environment variable RCLONE_CONFIG_DST_ACCESS_KEY_ID
2025/09/10 15:58:50 DEBUG : Setting secret_access_key="..." for "dst" from environment variable RCLONE_CONFIG_DST_SECRET_ACCESS_KEY
2025/09/10 15:58:50 DEBUG : Setting region="us-west-2" for "dst" from environment variable RCLONE_CONFIG_DST_REGION
2025/09/10 15:58:50 DEBUG : Setting region="us-west-2" for "dst" from environment variable RCLONE_CONFIG_DST_REGION
2025/09/10 15:58:50 DEBUG : Setting region="us-west-2" for "dst" from environment variable RCLONE_CONFIG_DST_REGION
2025/09/10 15:58:50 DEBUG : Setting region="us-west-2" for "dst" from environment variable RCLONE_CONFIG_DST_REGION
2025/09/10 15:58:50 DEBUG : Setting region="us-west-2" for "dst" from environment variable RCLONE_CONFIG_DST_REGION
2025/09/10 15:58:50 DEBUG : Setting region="us-west-2" for "dst" from environment variable RCLONE_CONFIG_DST_REGION
2025/09/10 15:58:50 DEBUG : Setting region="us-west-2" for "dst" from environment variable RCLONE_CONFIG_DST_REGION
2025/09/10 15:58:50 DEBUG : Setting region="us-west-2" for "dst" from environment variable RCLONE_CONFIG_DST_REGION
2025/09/10 15:58:50 DEBUG : Setting region="us-west-2" for "dst" from environment variable RCLONE_CONFIG_DST_REGION
2025/09/10 15:58:50 DEBUG : Setting region="us-west-2" for "dst" from environment variable RCLONE_CONFIG_DST_REGION
2025/09/10 15:58:50 DEBUG : Setting region="us-west-2" for "dst" from environment variable RCLONE_CONFIG_DST_REGION
2025/09/10 15:58:50 DEBUG : Setting region="us-west-2" for "dst" from environment variable RCLONE_CONFIG_DST_REGION
2025/09/10 15:58:50 DEBUG : Setting region="us-west-2" for "dst" from environment variable RCLONE_CONFIG_DST_REGION
2025/09/10 15:58:50 DEBUG : Setting region="us-west-2" for "dst" from environment variable RCLONE_CONFIG_DST_REGION
2025/09/10 15:58:50 DEBUG : Setting location_constraint="us-west-2" for "dst" from environment variable RCLONE_CONFIG_DST_LOCATION_CONSTRAINT
2025/09/10 15:58:50 DEBUG : Setting location_constraint="us-west-2" for "dst" from environment variable RCLONE_CONFIG_DST_LOCATION_CONSTRAINT
2025/09/10 15:58:50 DEBUG : Setting location_constraint="us-west-2" for "dst" from environment variable RCLONE_CONFIG_DST_LOCATION_CONSTRAINT
2025/09/10 15:58:50 DEBUG : Setting location_constraint="us-west-2" for "dst" from environment variable RCLONE_CONFIG_DST_LOCATION_CONSTRAINT
2025/09/10 15:58:50 DEBUG : Setting location_constraint="us-west-2" for "dst" from environment variable RCLONE_CONFIG_DST_LOCATION_CONSTRAINT
2025/09/10 15:58:50 DEBUG : Setting location_constraint="us-west-2" for "dst" from environment variable RCLONE_CONFIG_DST_LOCATION_CONSTRAINT
2025/09/10 15:58:50 DEBUG : Setting location_constraint="us-west-2" for "dst" from environment variable RCLONE_CONFIG_DST_LOCATION_CONSTRAINT
2025/09/10 15:58:50 DEBUG : Setting directory_markers="true" for "dst" from environment variable RCLONE_CONFIG_DST_DIRECTORY_MARKERS
2025/09/10 15:58:50 DEBUG : dst: detected overridden config - adding "{K4O1W}" suffix to name
2025/09/10 15:58:50 DEBUG : Setting provider="AWS" for "dst" from environment variable RCLONE_CONFIG_DST_PROVIDER
2025/09/10 15:58:50 DEBUG : Setting access_key_id="..." for "dst" from environment variable RCLONE_CONFIG_DST_ACCESS_KEY_ID
2025/09/10 15:58:50 DEBUG : Setting secret_access_key="..." for "dst" from environment variable RCLONE_CONFIG_DST_SECRET_ACCESS_KEY
2025/09/10 15:58:50 DEBUG : Setting region="us-west-2" for "dst" from environment variable RCLONE_CONFIG_DST_REGION
2025/09/10 15:58:50 DEBUG : Setting location_constraint="us-west-2" for "dst" from environment variable RCLONE_CONFIG_DST_LOCATION_CONSTRAINT
2025/09/10 15:58:50 DEBUG : Setting directory_markers="true" for "dst" from environment variable RCLONE_CONFIG_DST_DIRECTORY_MARKERS

...

2025/09/10 15:58:51 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/09/10 15:58:51 ERROR : foo: Failed to copy: failed to prepare upload: operation error S3: CreateBucket, https response error StatusCode: 403, RequestID: NZDWSV7BTFNQM7YQ, HostID: ...=, api error AccessDenied: User: arn:aws:iam::320548971708:user/rclone-test-user is not authorized to perform: s3:CreateBucket on resource: "arn:aws:s3:::tom-rclone-test" because no identity-based policy allows the s3:CreateBucket action
2025/09/10 15:58:51 ERROR : S3 bucket tom-rclone-test path localtest1/tom_test14: not deleting files as there were IO errors
2025/09/10 15:58:51 ERROR : S3 bucket tom-rclone-test path localtest1/tom_test14: not deleting directories as there were IO errors

asdffdsa · September 10, 2025, 11:30pm

hi,

a quirk of rclone from the early days.

--s3-no-check-bucket
or
RCLONE_CONFIG_SRC_NO_CHECK_BUCKET="true"

If you supply extra configuration to a backend by command line flag, environment variable or connection string then rclone will add a suffix based on the hash of the config to the name of the remote

thomasjm · September 10, 2025, 11:41pm

One more data point–the following works:

RCLONE_CONFIG="" rclone rcd --rc-no-auth -vv --dump requests,responses,bodies,headers

and in a separate terminal:

rclone rc config/create name=src type=local parameters='{}'
rclone rc config/create name=dst type=s3 parameters='{ "provider": "AWS", "access_key_id": "...", "secret_access_key": "...", "region": "us-west-2", "location_constraint": "us-west-2" }'
rclone rc sync/sync srcFs=src:/Users/tom/test1  dstFs=dst:tom-rclone-test/localtest1/tom_test15

Tried RCLONE_S3_NO_CHECK_BUCKET="true" with my original method and it worked.

Hmm, but I’m not trying to supply “extra configuration.” I want to fully configure the backend using environment variables, which the docs seem to indicate I can do.

RCLONE_S3_NO_CHECK_BUCKET="true" works as a workaround, but why do I only need it when I’m doing environment variable based config?

system · October 10, 2025, 11:42pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.