Trouble copying between 2 s3 buckets using --s3-version-at

Help and Support
1

What is the problem you are having with rclone?

I’m using rclone to copy data from one S3 bucket to another. The source bucket is an Amazon S3 bucket (cloud) and the destination bucket is an on-premise S3-compatible bucket.

The issue I am having is when using the –s3-version-at flag. I am using this flag because the objects in the source bucket were deleted (made non current) and I want to select them just prior to when they were deleted. So I am setting –s3-version-at to be 1 day before the objects were deleted on the source.
On the destination, I just want to copy the selected objects to the destination. The destination is empty.
I can see that rclone is able to correctly identify the list of objects to copy, and running a dry-run works correctly. However the actual copy is failing.

If I copy the same objects to my local workstation using rclone (using –s3-version-at), and then copy them again to the destination (without the –s3-version-at flag) it works.

So it seems like rclone is applying the –s3-version-at flag to the destination remote as well as to the source, which is incorrect as logically it should only apply to the source when selecting the files to copy.

Run the command 'rclone version' and share the full output of the command.

rclone v1.71.0
- os/version: darwin 15.7.2 (64 bit)
- os/kernel: 24.6.0 (arm64)
- os/type: darwin
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.25.0
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

  • Amazon S3 (source)
  • On-prem S3-compatible storage (destination)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy \
  source:source-za/raw/edw/cc_d/ctxn/2025/09/01/ \
  dest:dest-za/raw/edw/cc_d/ctxn/2025/09/01/ \
  --s3-version-at 2025-11-15T00:00:00Z

The rclone config contents with secrets removed.


[source]
type = s3
provider = AWS
env_auth = true
region = af-south-1
location_constraint = af-south-1
no_check_bucket = true
server_side_encryption = aws:kms
sse_kms_key_id = arn:aws:kms:af-south-1:123456:key/123456

[dest]
# Read-write access to corp bucket: dest-za
type = s3
provider = Other
endpoint = https://my-corp-endpoint:9021
access_key_id = my-key
secret_access_key = my-secret
no_check_bucket = true

A log from the command with the -vv flag

rclone copy \
  source:source-za/raw/edw/cc_d/ctxn/2025/09/01/ \
  dest:dest-za/raw/edw/cc_d/ctxn/2025/09/01/ \
  --s3-version-at 2025-11-15T00:00:00Z -vv

2025/11/18 08:48:38 DEBUG : rclone: Version "v1.71.0" starting with parameters ["rclone" "copy” “source:source-za/raw/edw/cc_d/ctxn/2025/09/01/“ “dest:dest-za/raw/edw/cc_d/ctxn/2025/09/01/" "--s3-version-at" "2025-11-15T00:00:00Z" "-vv"]
2025/11/18 08:48:38 DEBUG : Creating backend with remote “source:source-za/raw/edw/cc_d/ctxn/2025/09/01/"
2025/11/18 08:48:38 DEBUG : Using config file from "/Users/me/.config/rclone/rclone.conf"
2025/11/18 08:48:38 DEBUG : source: detected overridden config - adding "{Xtr7e}" suffix to name
2025/11/18 08:48:38 DEBUG : fs cache: renaming cache item “source:source-za/raw/edw/cc_d/ctxn/2025/09/01/" to be canonical “source{Xtr7e}:source-za/raw/edw/cc_d/ctxn/2025/09/01"
2025/11/18 08:48:38 DEBUG : Creating backend with remote “dest:dest-za/raw/edw/cc_d/ctxn/2025/09/01/"
2025/11/18 08:48:38 DEBUG : dest-rw: detected overridden config - adding "{Xtr7e}" suffix to name
2025/11/18 08:48:38 DEBUG : fs cache: renaming cache item “dest-rw:dest-za/raw/edw/cc_d/ctxn/2025/09/01/" to be canonical “dest-rw{Xtr7e}:dest-za/raw/edw/cc_d/ctxn/2025/09/01"
2025/11/18 08:48:40 DEBUG : FWORK.T01CTXN.BKUP2.SEP01: Need to transfer - File not found at Destination
2025/11/18 08:48:40 DEBUG : FWORK.T02CTXN.BKUP2.SEP01: Need to transfer - File not found at Destination
2025/11/18 08:48:40 DEBUG : FWORK.T03CTXN.BKUP2.SEP01: Need to transfer - File not found at Destination
2025/11/18 08:48:40 DEBUG : FWORK.T04CTXN.BKUP2.SEP01: Need to transfer - File not found at Destination
2025/11/18 08:48:40 DEBUG : FWORK.T05CTXN.BKUP2.SEP01: Need to transfer - File not found at Destination
2025/11/18 08:48:40 DEBUG : FWORK.T06CTXN.BKUP2.SEP01: Need to transfer - File not found at Destination
2025/11/18 08:48:40 DEBUG : FWORK.T07CTXN.BKUP2.SEP01: Need to transfer - File not found at Destination
2025/11/18 08:48:40 DEBUG : FWORK.T08CTXN.BKUP2.SEP01: Need to transfer - File not found at Destination
2025/11/18 08:48:40 DEBUG : FWORK.T09CTXN.BKUP2.SEP01: Need to transfer - File not found at Destination
2025/11/18 08:48:40 DEBUG : FWORK.T10CTXN.BKUP2.SEP01: Need to transfer - File not found at Destination
2025/11/18 08:48:40 DEBUG : FWORK.T11CTXN.BKUP2.SEP01: Need to transfer - File not found at Destination
2025/11/18 08:48:40 DEBUG : FWORK.T12CTXN.BKUP2.SEP01: Need to transfer - File not found at Destination
2025/11/18 08:48:40 DEBUG : FWORK.T13CTXN.BKUP2.SEP01: Need to transfer - File not found at Destination
2025/11/18 08:48:40 DEBUG : FWORK.T14CTXN.BKUP2.SEP01: Need to transfer - File not found at Destination
2025/11/18 08:48:40 DEBUG : FWORK.T15CTXN.BKUP2.SEP01: Need to transfer - File not found at Destination
2025/11/18 08:48:40 DEBUG : FWORK.T16CTXN.BKUP2.SEP01: Need to transfer - File not found at Destination
2025/11/18 08:48:40 DEBUG : FWORK.T17CTXN.BKUP2.SEP01: Need to transfer - File not found at Destination
2025/11/18 08:48:40 DEBUG : FWORK.T18CTXN.BKUP2.SEP01: Need to transfer - File not found at Destination
2025/11/18 08:48:40 DEBUG : FWORK.T19CTXN.BKUP2.SEP01: Need to transfer - File not found at Destination
2025/11/18 08:48:40 DEBUG : FWORK.T20CTXN.BKUP2.SEP01: Need to transfer - File not found at Destination
2025/11/18 08:48:40 DEBUG : S3 bucket dest-za path raw/edw/cc_d/ctxn/2025/09/01: Waiting for checks to finish
2025/11/18 08:48:40 DEBUG : S3 bucket dest-za path raw/edw/cc_d/ctxn/2025/09/01: Waiting for transfers to finish
2025/11/18 08:48:40 ERROR : FWORK.T02CTXN.BKUP2.SEP01: Failed to copy: can't modify or delete files in --s3-version-at mode
2025/11/18 08:48:41 ERROR : FWORK.T01CTXN.BKUP2.SEP01: Failed to copy: can't modify or delete files in --s3-version-at mode
2025/11/18 08:48:41 ERROR : FWORK.T03CTXN.BKUP2.SEP01: Failed to copy: can't modify or delete files in --s3-version-at mode
2025/11/18 08:48:41 ERROR : FWORK.T04CTXN.BKUP2.SEP01: Failed to copy: can't modify or delete files in --s3-version-at mode
2025/11/18 08:48:41 ERROR : FWORK.T05CTXN.BKUP2.SEP01: Failed to copy: can't modify or delete files in --s3-version-at mode
2025/11/18 08:48:43 ERROR : FWORK.T08CTXN.BKUP2.SEP01: Failed to copy: can't modify or delete files in --s3-version-at mode
2025/11/18 08:48:43 ERROR : FWORK.T09CTXN.BKUP2.SEP01: Failed to copy: can't modify or delete files in --s3-version-at mode
2025/11/18 08:48:45 ERROR : FWORK.T07CTXN.BKUP2.SEP01: Failed to copy: can't modify or delete files in --s3-version-at mode
2025/11/18 08:48:45 ERROR : FWORK.T06CTXN.BKUP2.SEP01: Failed to copy: can't modify or delete files in --s3-version-at mode
2025/11/18 08:48:45 ERROR : FWORK.T10CTXN.BKUP2.SEP01: Failed to copy: can't modify or delete files in --s3-version-at mode
2025/11/18 08:48:46 ERROR : FWORK.T11CTXN.BKUP2.SEP01: Failed to copy: can't modify or delete files in --s3-version-at mode
2025/11/18 08:48:47 ERROR : FWORK.T12CTXN.BKUP2.SEP01: Failed to copy: can't modify or delete files in --s3-version-at mode
2025/11/18 08:48:53 ERROR : FWORK.T13CTXN.BKUP2.SEP01: Failed to copy: can't modify or delete files in --s3-version-at mode
2025/11/18 08:48:53 ERROR : FWORK.T14CTXN.BKUP2.SEP01: Failed to copy: can't modify or delete files in --s3-version-at mode
2025/11/18 08:48:54 ERROR : FWORK.T15CTXN.BKUP2.SEP01: Failed to copy: can't modify or delete files in --s3-version-at mode
2025/11/18 08:48:55 ERROR : FWORK.T16CTXN.BKUP2.SEP01: Failed to copy: can't modify or delete files in --s3-version-at mode
2025/11/18 08:49:01 ERROR : FWORK.T17CTXN.BKUP2.SEP01: Failed to copy: can't modify or delete files in --s3-version-at mode
2025/11/18 08:49:02 ERROR : FWORK.T18CTXN.BKUP2.SEP01: Failed to copy: can't modify or delete files in --s3-version-at mode
2025/11/18 08:49:02 ERROR : FWORK.T19CTXN.BKUP2.SEP01: Failed to copy: can't modify or delete files in --s3-version-at mode
2025/11/18 08:49:03 ERROR : FWORK.T20CTXN.BKUP2.SEP01: Failed to copy: can't modify or delete files in --s3-version-at mode
2025/11/18 08:49:03 ERROR : Attempt 1/3 failed with 20 errors and: can't modify or delete files in --s3-version-at mode

[... repeated log lines deleted for brevity...]

2025/11/18 08:50:41 ERROR : Attempt 3/3 failed with 20 errors and: can’t modify or delete files in --s3-version-at mode
2025/11/18 08:50:41 INFO  : 
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                20 (retrying may help)
Checks:                 0 / 0, -, Listed 60
Elapsed time:       2m2.6s

2025/11/18 08:50:41 DEBUG : 6 go routines active
2025/11/18 08:50:41 NOTICE: Failed to copy with 20 errors: last error was: can't modify or delete files in --s3-version-at mode

(edited a few times to fix formatting)

2

instead of using --s3-version-at 2025-11-15T00:00:00Z flag (which indeed can mess up things when both source and destination are S3) put it into your source remote config as:

version_at = 2025-11-15T00:00:00Z

Or create all new remote just for at:

[source_at]
type = s3
provider = AWS
env_auth = true
region = af-south-1
location_constraint = af-south-1
no_check_bucket = true
server_side_encryption = aws:kms
sse_kms_key_id = arn:aws:kms:af-south-1:123456:key/123456
version_at = 2025-11-15T00:00:00Z

You can see details (for config) in docs:

  • Config: version_at

  • Env Var: RCLONE_S3_VERSION_AT

  • Type: Time

  • Default: off

3

Spot on advice from @kapitainsky

You can also put it in a connection string which can be easier to work with, though the quoting is a bit of a pain. This quoting is for linux/macOS bash style shells. You'd want something different for Windows.

rclone copy \
  source:source-za/raw/edw/cc_d/ctxn/2025/09/01/ \
  dest:dest-za/raw/edw/cc_d/ctxn/2025/09/01/ \
  --s3-version-at 2025-11-15T00:00:00Z

Becomes

rclone copy \
  'source,version_at="2025-11-15T00:00:00Z":source-za/raw/edw/cc_d/ctxn/2025/09/01/' \
  dest:dest-za/raw/edw/cc_d/ctxn/2025/09/01/
4

Thanks for the feedback, I'll try that and provide an update

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.