What is the problem you are having with rclone?
I used to copy files to Amazon S3 with rclone, giving parameters using environment variables.
It works with the version 1.67.0, but not with newer versions.
Run the command 'rclone version' and share the full output of the command.
For rclone 1.67.0:
rclone v1.67.0
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 6.8.0-59-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.4
- go/linking: static
- go/tags: none
For rclone 1.68.0:
rclone v1.68.0
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 6.8.0-59-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.23.1
- go/linking: static
- go/tags: none
I have the same behaviour with the same rclone executable files on a different computer.
For rclone 1.67.0:
rclone v1.67.0
- os/version: ubuntu 24.04 (64 bit)
- os/kernel: 6.8.0-62-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.4
- go/linking: static
- go/tags: none
For rclone 1.68.0:
rclone v1.68.0
- os/version: ubuntu 24.04 (64 bit)
- os/kernel: 6.8.0-62-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.23.1
- go/linking: static
- go/tags: none
Outputs are more or less the same for rclone 1.69.0 and rclone 1.70.0 (only the Go version changes).
Which cloud storage system are you using? (eg Google Drive)
Amazon S3
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
This command works (see below for the output):
RCLONE_CONFIG_STORAGE_TYPE=s3 \
RCLONE_CONFIG_STORAGE_PROVIDER=AWS \
RCLONE_CONFIG_STORAGE_ACCESS_KEY_ID='EDITED' \
RCLONE_CONFIG_STORAGE_SECRET_ACCESS_KEY='EDITED' \
RCLONE_CONFIG_STORAGE_REGION=eu-west-3 \
./rclone-1.67.0 copy foo.txt storage:EDITED/foo.txt-1.67.0 -vv
This command doesn't work (only the rclone version changes; see below for the output):
RCLONE_CONFIG_STORAGE_TYPE=s3 \
RCLONE_CONFIG_STORAGE_PROVIDER=AWS \
RCLONE_CONFIG_STORAGE_ACCESS_KEY_ID='EDITED' \
RCLONE_CONFIG_STORAGE_SECRET_ACCESS_KEY='EDITED' \
RCLONE_CONFIG_STORAGE_REGION=eu-west-3 \
./rclone-1.68.0 copy foo.txt storage:EDITED/foo.txt-1.68.0 -vv
It's the same with newer versions of rclone (only the version number changes).
The rclone config contents with secrets removed.
There is no rclone configuration. All parameters are given using environment variables.
A log from the command with the -vv
flag
With rclone version 1.67.0:
2025/06/22 09:15:29 DEBUG : rclone: Version "v1.67.0" starting with parameters ["./rclone-1.67.0" "copy" "foo.txt" "storage:EDITED/foo.txt-1.67.0" "-vv"]
2025/06/22 09:15:29 DEBUG : Creating backend with remote "foo.txt"
2025/06/22 09:15:29 NOTICE: Config file "/home/EDITED/.config/rclone/rclone.conf" not found - using defaults
2025/06/22 09:15:29 DEBUG : fs cache: adding new entry for parent of "foo.txt", "/home/EDITED"
2025/06/22 09:15:29 DEBUG : Creating backend with remote "storage:EDITED/foo.txt-1.67.0"
2025/06/22 09:15:29 DEBUG : Setting type="s3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_TYPE
2025/06/22 09:15:29 DEBUG : Setting provider="AWS" for "storage" from environment variable RCLONE_CONFIG_STORAGE_PROVIDER
2025/06/22 09:15:29 DEBUG : Setting access_key_id="EDITED" for "storage" from environment variable RCLONE_CONFIG_STORAGE_ACCESS_KEY_ID
2025/06/22 09:15:29 DEBUG : Setting secret_access_key="EDITED" for "storage" from environment variable RCLONE_CONFIG_STORAGE_SECRET_ACCESS_KEY
2025/06/22 09:15:29 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:15:29 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:15:29 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:15:29 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:15:29 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:15:29 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:15:29 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:15:29 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:15:29 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:15:29 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:15:29 DEBUG : storage: detected overridden config - adding "{1BfYY}" suffix to name
2025/06/22 09:15:29 DEBUG : Setting provider="AWS" for "storage" from environment variable RCLONE_CONFIG_STORAGE_PROVIDER
2025/06/22 09:15:29 DEBUG : Setting access_key_id="EDITED" for "storage" from environment variable RCLONE_CONFIG_STORAGE_ACCESS_KEY_ID
2025/06/22 09:15:29 DEBUG : Setting secret_access_key="EDITED" for "storage" from environment variable RCLONE_CONFIG_STORAGE_SECRET_ACCESS_KEY
2025/06/22 09:15:29 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:15:29 DEBUG : fs cache: renaming cache item "storage:EDITED/foo.txt-1.67.0" to be canonical "storage{1BfYY}:EDITED/foo.txt-1.67.0"
2025/06/22 09:15:29 DEBUG : foo.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2025/06/22 09:15:29 DEBUG : foo.txt: Unchanged skipping
2025/06/22 09:15:29 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.0s
2025/06/22 09:15:29 DEBUG : 6 go routines active
With rclone version 1.68.0 and above:
2025/06/22 09:15:46 DEBUG : rclone: Version "v1.68.0" starting with parameters ["./rclone-1.68.0" "copy" "foo.txt" "storage:EDITED/foo.txt-1.68.0" "-vv"]
2025/06/22 09:15:46 DEBUG : Creating backend with remote "foo.txt"
2025/06/22 09:15:46 NOTICE: Config file "/home/EDITED/.config/rclone/rclone.conf" not found - using defaults
2025/06/22 09:15:46 DEBUG : fs cache: adding new entry for parent of "foo.txt", "/home/EDITED"
2025/06/22 09:15:46 DEBUG : Creating backend with remote "storage:EDITED/foo.txt-1.68.0"
2025/06/22 09:15:46 DEBUG : Setting type="s3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_TYPE
2025/06/22 09:15:46 DEBUG : Setting provider="AWS" for "storage" from environment variable RCLONE_CONFIG_STORAGE_PROVIDER
2025/06/22 09:15:46 DEBUG : Setting access_key_id="EDITED" for "storage" from environment variable RCLONE_CONFIG_STORAGE_ACCESS_KEY_ID
2025/06/22 09:15:46 DEBUG : Setting secret_access_key="EDITED" for "storage" from environment variable RCLONE_CONFIG_STORAGE_SECRET_ACCESS_KEY
2025/06/22 09:15:46 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:15:46 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:15:46 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:15:46 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:15:46 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:15:46 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:15:46 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:15:46 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:15:46 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:15:46 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:15:46 DEBUG : storage: detected overridden config - adding "{1BfYY}" suffix to name
2025/06/22 09:15:46 DEBUG : Setting provider="AWS" for "storage" from environment variable RCLONE_CONFIG_STORAGE_PROVIDER
2025/06/22 09:15:46 DEBUG : Setting access_key_id="EDITED" for "storage" from environment variable RCLONE_CONFIG_STORAGE_ACCESS_KEY_ID
2025/06/22 09:15:46 DEBUG : Setting secret_access_key="EDITED" for "storage" from environment variable RCLONE_CONFIG_STORAGE_SECRET_ACCESS_KEY
2025/06/22 09:15:46 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:15:46 DEBUG : fs cache: renaming cache item "storage:EDITED/foo.txt-1.68.0" to be canonical "storage{1BfYY}:EDITED/foo.txt-1.68.0"
2025/06/22 09:15:46 DEBUG : foo.txt: Need to transfer - File not found at Destination
2025/06/22 09:15:46 ERROR : foo.txt: Failed to copy: failed to prepare upload: operation error S3: CreateBucket, https response error StatusCode: 400, RequestID: EDITED, HostID: EDITED, api error IllegalLocationConstraintException: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.
2025/06/22 09:15:46 ERROR : Attempt 1/3 failed with 1 errors and: failed to prepare upload: operation error S3: CreateBucket, https response error StatusCode: 400, RequestID: EDITED, HostID: EDITED, api error IllegalLocationConstraintException: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.
2025/06/22 09:15:47 DEBUG : foo.txt: Need to transfer - File not found at Destination
2025/06/22 09:15:47 ERROR : foo.txt: Failed to copy: failed to prepare upload: operation error S3: CreateBucket, https response error StatusCode: 400, RequestID: EDITED, HostID: EDITED, api error IllegalLocationConstraintException: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.
2025/06/22 09:15:47 ERROR : Attempt 2/3 failed with 1 errors and: failed to prepare upload: operation error S3: CreateBucket, https response error StatusCode: 400, RequestID: EDITED, HostID: EDITED, api error IllegalLocationConstraintException: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.
2025/06/22 09:15:47 DEBUG : foo.txt: Need to transfer - File not found at Destination
2025/06/22 09:15:47 ERROR : foo.txt: Failed to copy: failed to prepare upload: operation error S3: CreateBucket, https response error StatusCode: 400, RequestID: EDITED, HostID: EDITED, api error IllegalLocationConstraintException: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.
2025/06/22 09:15:47 ERROR : Attempt 3/3 failed with 1 errors and: failed to prepare upload: operation error S3: CreateBucket, https response error StatusCode: 400, RequestID: EDITED, HostID: EDITED, api error IllegalLocationConstraintException: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.
2025/06/22 09:15:47 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.1s
2025/06/22 09:15:47 DEBUG : 6 go routines active
2025/06/22 09:15:47 NOTICE: Failed to copy: failed to prepare upload: operation error S3: CreateBucket, https response error StatusCode: 400, RequestID: EDITED, HostID: EDITED, api error IllegalLocationConstraintException: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.
It looks like rclone 1.68.0+ tries to create the bucket (but it is already created, and the IAM user doesn't have the right to create buckets).
More information
If the file already exists on Amazon S3, there is no error when uploading a new version using rclone 1.68.0+.
Then, this command :
RCLONE_CONFIG_STORAGE_TYPE=s3 \
RCLONE_CONFIG_STORAGE_PROVIDER=AWS \
RCLONE_CONFIG_STORAGE_ACCESS_KEY_ID='EDITED' \
RCLONE_CONFIG_STORAGE_SECRET_ACCESS_KEY='EDITED' \
RCLONE_CONFIG_STORAGE_REGION=eu-west-3 \
./rclone-1.68.0 copy foo.txt storage:EDITED/foo.txt-1.67.0 -vv
(considering that the file EDITED/foo.txt-1.67.0/foo.txt
already exists in the bucket)
Results with:
2025/06/22 09:28:40 DEBUG : rclone: Version "v1.68.0" starting with parameters ["./rclone-1.68.0" "copy" "foo.txt" "storage:EDITED/foo.txt-1.67.0" "-vv"]
2025/06/22 09:28:40 DEBUG : Creating backend with remote "foo.txt"
2025/06/22 09:28:40 NOTICE: Config file "/home/EDITED/.config/rclone/rclone.conf" not found - using defaults
2025/06/22 09:28:40 DEBUG : fs cache: adding new entry for parent of "foo.txt", "/home/EDITED"
2025/06/22 09:28:40 DEBUG : Creating backend with remote "storage:EDITED/foo.txt-1.67.0"
2025/06/22 09:28:40 DEBUG : Setting type="s3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_TYPE
2025/06/22 09:28:40 DEBUG : Setting provider="AWS" for "storage" from environment variable RCLONE_CONFIG_STORAGE_PROVIDER
2025/06/22 09:28:40 DEBUG : Setting access_key_id="EDITED" for "storage" from environment variable RCLONE_CONFIG_STORAGE_ACCESS_KEY_ID
2025/06/22 09:28:40 DEBUG : Setting secret_access_key="EDITED" for "storage" from environment variable RCLONE_CONFIG_STORAGE_SECRET_ACCESS_KEY
2025/06/22 09:28:40 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:28:40 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:28:40 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:28:40 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:28:40 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:28:40 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:28:40 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:28:40 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:28:40 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:28:40 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:28:40 DEBUG : storage: detected overridden config - adding "{1BfYY}" suffix to name
2025/06/22 09:28:40 DEBUG : Setting provider="AWS" for "storage" from environment variable RCLONE_CONFIG_STORAGE_PROVIDER
2025/06/22 09:28:40 DEBUG : Setting access_key_id="EDITED" for "storage" from environment variable RCLONE_CONFIG_STORAGE_ACCESS_KEY_ID
2025/06/22 09:28:40 DEBUG : Setting secret_access_key="EDITED" for "storage" from environment variable RCLONE_CONFIG_STORAGE_SECRET_ACCESS_KEY
2025/06/22 09:28:40 DEBUG : Setting region="eu-west-3" for "storage" from environment variable RCLONE_CONFIG_STORAGE_REGION
2025/06/22 09:28:40 DEBUG : fs cache: renaming cache item "storage:EDITED/foo.txt-1.67.0" to be canonical "storage{1BfYY}:EDITED/foo.txt-1.67.0"
2025/06/22 09:28:40 DEBUG : foo.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2025/06/22 09:28:40 DEBUG : foo.txt: Unchanged skipping
2025/06/22 09:28:40 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.0s
2025/06/22 09:28:40 DEBUG : 6 go routines active