Rclone Re-copy files while files already exist in remote S3

What is the problem you are having with rclone?

I got debug message "Need to transfer - File not found at Destination" after running rclone. Some of the files were already in remote S3 path but rclone re-copyed them again.

I have already checked the file size stored in s3 (by downloading the file) and compared to the local file. They were identical.

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.0

  • os/version: debian 12.7 (64 bit)
  • os/kernel: 6.5.0-18-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.22.7
  • go/linking: dynamic
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

--bwlimit 512k:off --size-only --no-traverse --s3-no-check-bucket --min-age 15m -v copy /recordings xxxxxxxx

The rclone config contents with secrets removed.

[xxxxxx-s3]
type = s3
provider = AWS
env_auth = false
access_key_id = xxxxx
secret_access_key = xxxxxx
location_constraint = 
server_side_encryption = 
region = ap-southeast-1

A log from the command with the -vv flag

2024/11/09 00:10:25 DEBUG : Setting --config "/config/rclone.conf" from environment variable RCLONE_CONFIG="/config/rclone.conf"
2024/11/09 00:10:25 INFO  : Starting bandwidth limiter at 800Ki:off Byte/s
2024/11/09 00:10:25 DEBUG : rclone: Version "v1.68.0" starting with parameters ["rclone" "--bwlimit" "800k:off" "--size-only" "--no-traverse" "--s3-no-check-bucket" "--min-age" "15m" "-vv" "copy" "/recordings" "xxxxxxx"]
2024/11/09 00:10:25 DEBUG : Creating backend with remote "/recordings"
2024/11/09 00:10:25 DEBUG : Using config file from "/config/rclone.conf"
2024/11/09 00:10:25 DEBUG : Creating backend with remote "xxxxxxxxx"
2024/11/09 00:10:25 DEBUG : xxxxxxx: detected overridden config - adding "{Dn7qA}" suffix to name
2024/11/09 00:10:25 DEBUG : fs cache: renaming cache item "xxxxxx" to be canonical "xxxxxxx"
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_02-00-02-817639.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_06-50-02-632163.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_07-00-02-693507.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_07-10-02-694199.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_07-20-02-698417.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_07-30-02-690759.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_07-40-02-691201.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_07-50-02-695772.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_08-00-02-694377.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_08-10-02-691550.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_08-20-02-688931.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_08-30-02-691801.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_08-40-02-690158.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_08-50-02-682177.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_09-00-02-690653.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_09-10-02-688695.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_09-20-02-694036.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_09-30-02-681178.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_09-40-02-677898.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_09-50-02-678195.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_10-00-02-672563.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_10-10-02-682930.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_10-20-02-679472.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_10-30-02-676554.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_10-40-02-673099.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_10-50-02-676937.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_11-00-02-679288.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_11-10-02-671382.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_11-20-02-677201.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_11-30-02-670793.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_11-40-02-679022.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_11-50-02-667117.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_12-00-02-674281.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_12-10-02-677056.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_12-20-02-671709.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_12-30-02-671736.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_12-40-02-666943.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_12-50-02-664193.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_13-00-02-668164.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_13-10-02-667311.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_13-20-02-666031.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_13-30-02-661392.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_13-40-02-718563.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_13-50-02-720934.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_14-00-02-662509.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_14-10-02-662405.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_14-20-02-664049.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_14-30-02-665171.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_14-40-02-668955.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_14-50-02-667306.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_15-00-02-667889.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_15-10-02-719591.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_15-20-02-717101.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_15-30-02-719209.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_15-40-02-720212.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_15-50-02-714602.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_16-00-02-719024.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_16-10-02-720753.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_16-20-02-717415.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_16-30-02-717554.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_16-40-02-716735.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_16-50-02-720589.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_17-00-02-719505.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_17-10-02-713739.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_17-20-02-720134.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_17-30-02-722566.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_17-40-02-720191.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_17-50-02-717158.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_18-00-02-707234.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_18-10-02-704473.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_18-20-02-711149.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_18-30-02-710901.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_18-40-02-705643.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_18-50-02-719663.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_19-00-02-713618.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_19-10-02-712038.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_19-20-02-709331.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_19-30-02-711495.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_19-40-02-701113.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_19-50-02-702686.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_20-00-02-697135.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_20-10-02-695823.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_20-20-02-695195.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_20-30-02-694256.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_20-40-02-703899.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_20-50-02-694466.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_21-00-02-696025.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_21-10-02-698765.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_21-20-02-692936.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_21-30-02-690414.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_21-40-02-697139.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_21-50-02-688556.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_22-00-02-685821.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_22-10-02-690598.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_22-20-02-682033.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_22-30-02-694822.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_22-40-02-686226.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_22-50-02-685878.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_23-00-02-687485.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_23-10-02-687706.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_23-20-02-688284.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_23-30-02-696551.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_23-40-02-690047.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_23-50-02-677963.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_00-00-02-813769.mp4: Sizes identical
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_00-00-02-813769.mp4: Unchanged skipping
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_00-20-02-804747.mp4: Sizes identical
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_00-20-02-804747.mp4: Unchanged skipping
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_00-10-02-820271.mp4: Sizes identical
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_00-10-02-820271.mp4: Unchanged skipping
2024/11/09 00:10:26 DEBUG : S3 bucket XXXXXXXXXXX: Waiting for checks to finish
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_00-40-02-802627.mp4: Sizes identical
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_00-40-02-802627.mp4: Unchanged skipping
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_01-00-02-807234.mp4: Sizes identical
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_00-50-02-808545.mp4: Sizes identical
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_01-00-02-807234.mp4: Unchanged skipping
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_00-50-02-808545.mp4: Unchanged skipping
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_01-10-02-805947.mp4: Sizes identical
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_01-10-02-805947.mp4: Unchanged skipping
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_01-20-02-810169.mp4: Sizes identical
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_01-20-02-810169.mp4: Unchanged skipping
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_01-40-02-855996.mp4: Sizes identical
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_01-40-02-855996.mp4: Unchanged skipping
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_00-30-02-795908.mp4: Sizes identical
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_00-30-02-795908.mp4: Unchanged skipping
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_01-50-02-824214.mp4: Sizes identical
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_01-50-02-824214.mp4: Unchanged skipping
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_01-30-02-814417.mp4: Sizes identical
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_01-30-02-814417.mp4: Unchanged skipping
2024/11/09 00:10:26 DEBUG : S3 bucket XXXXXXXXXXX: Waiting for transfers to finish

welcome to the forum,

to simplify testing and keep the logs small.

  1. please pick a single file
  2. post the output of rclone lsl -vv for the source file and the dest file
    something like, rclone lsl webcam/2024-11-02/2024-11-02_02-00-02-817639.mp4

Thank you for fast reply, The issue environment has been removed. Current sync item behaviors normally. I will past a single file log & rclone lsl -vv output & update this issue when I found another re-copy behavior.

1 Like

This looks similar to the issue I'm having with OneDrive Existing files not found by rclone - #4 by OliverL

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.