What is the problem you are having with rclone?
I got debug message "Need to transfer - File not found at Destination" after running rclone. Some of the files were already in remote S3 path but rclone re-copyed them again.
I have already checked the file size stored in s3 (by downloading the file) and compared to the local file. They were identical.
Run the command 'rclone version' and share the full output of the command.
rclone v1.68.0
- os/version: debian 12.7 (64 bit)
- os/kernel: 6.5.0-18-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.7
- go/linking: dynamic
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
AWS S3
The command you were trying to run (eg rclone copy /tmp remote:tmp)
--bwlimit 512k:off --size-only --no-traverse --s3-no-check-bucket --min-age 15m -v copy /recordings xxxxxxxx
The rclone config contents with secrets removed.
[xxxxxx-s3]
type = s3
provider = AWS
env_auth = false
access_key_id = xxxxx
secret_access_key = xxxxxx
location_constraint =
server_side_encryption =
region = ap-southeast-1
A log from the command with the -vv flag
2024/11/09 00:10:25 DEBUG : Setting --config "/config/rclone.conf" from environment variable RCLONE_CONFIG="/config/rclone.conf"
2024/11/09 00:10:25 INFO : Starting bandwidth limiter at 800Ki:off Byte/s
2024/11/09 00:10:25 DEBUG : rclone: Version "v1.68.0" starting with parameters ["rclone" "--bwlimit" "800k:off" "--size-only" "--no-traverse" "--s3-no-check-bucket" "--min-age" "15m" "-vv" "copy" "/recordings" "xxxxxxx"]
2024/11/09 00:10:25 DEBUG : Creating backend with remote "/recordings"
2024/11/09 00:10:25 DEBUG : Using config file from "/config/rclone.conf"
2024/11/09 00:10:25 DEBUG : Creating backend with remote "xxxxxxxxx"
2024/11/09 00:10:25 DEBUG : xxxxxxx: detected overridden config - adding "{Dn7qA}" suffix to name
2024/11/09 00:10:25 DEBUG : fs cache: renaming cache item "xxxxxx" to be canonical "xxxxxxx"
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_02-00-02-817639.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_06-50-02-632163.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_07-00-02-693507.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_07-10-02-694199.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_07-20-02-698417.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_07-30-02-690759.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_07-40-02-691201.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_07-50-02-695772.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_08-00-02-694377.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_08-10-02-691550.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_08-20-02-688931.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_08-30-02-691801.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_08-40-02-690158.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_08-50-02-682177.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_09-00-02-690653.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_09-10-02-688695.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_09-20-02-694036.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_09-30-02-681178.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_09-40-02-677898.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_09-50-02-678195.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_10-00-02-672563.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_10-10-02-682930.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_10-20-02-679472.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_10-30-02-676554.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_10-40-02-673099.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_10-50-02-676937.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_11-00-02-679288.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_11-10-02-671382.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_11-20-02-677201.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_11-30-02-670793.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_11-40-02-679022.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_11-50-02-667117.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_12-00-02-674281.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_12-10-02-677056.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_12-20-02-671709.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_12-30-02-671736.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_12-40-02-666943.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_12-50-02-664193.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_13-00-02-668164.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_13-10-02-667311.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_13-20-02-666031.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_13-30-02-661392.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_13-40-02-718563.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_13-50-02-720934.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_14-00-02-662509.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_14-10-02-662405.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_14-20-02-664049.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_14-30-02-665171.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_14-40-02-668955.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_14-50-02-667306.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_15-00-02-667889.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_15-10-02-719591.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_15-20-02-717101.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_15-30-02-719209.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_15-40-02-720212.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_15-50-02-714602.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_16-00-02-719024.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_16-10-02-720753.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_16-20-02-717415.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_16-30-02-717554.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_16-40-02-716735.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_16-50-02-720589.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_17-00-02-719505.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_17-10-02-713739.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_17-20-02-720134.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_17-30-02-722566.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_17-40-02-720191.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_17-50-02-717158.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_18-00-02-707234.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_18-10-02-704473.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_18-20-02-711149.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_18-30-02-710901.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_18-40-02-705643.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_18-50-02-719663.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_19-00-02-713618.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_19-10-02-712038.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_19-20-02-709331.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_19-30-02-711495.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_19-40-02-701113.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_19-50-02-702686.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_20-00-02-697135.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_20-10-02-695823.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_20-20-02-695195.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_20-30-02-694256.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_20-40-02-703899.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_20-50-02-694466.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_21-00-02-696025.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_21-10-02-698765.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_21-20-02-692936.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_21-30-02-690414.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_21-40-02-697139.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_21-50-02-688556.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_22-00-02-685821.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_22-10-02-690598.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_22-20-02-682033.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_22-30-02-694822.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_22-40-02-686226.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_22-50-02-685878.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_23-00-02-687485.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_23-10-02-687706.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_23-20-02-688284.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_23-30-02-696551.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_23-40-02-690047.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_23-50-02-677963.mp4: Need to transfer - File not found at Destination
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_00-00-02-813769.mp4: Sizes identical
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_00-00-02-813769.mp4: Unchanged skipping
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_00-20-02-804747.mp4: Sizes identical
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_00-20-02-804747.mp4: Unchanged skipping
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_00-10-02-820271.mp4: Sizes identical
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_00-10-02-820271.mp4: Unchanged skipping
2024/11/09 00:10:26 DEBUG : S3 bucket XXXXXXXXXXX: Waiting for checks to finish
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_00-40-02-802627.mp4: Sizes identical
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_00-40-02-802627.mp4: Unchanged skipping
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_01-00-02-807234.mp4: Sizes identical
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_00-50-02-808545.mp4: Sizes identical
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_01-00-02-807234.mp4: Unchanged skipping
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_00-50-02-808545.mp4: Unchanged skipping
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_01-10-02-805947.mp4: Sizes identical
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_01-10-02-805947.mp4: Unchanged skipping
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_01-20-02-810169.mp4: Sizes identical
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_01-20-02-810169.mp4: Unchanged skipping
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_01-40-02-855996.mp4: Sizes identical
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_01-40-02-855996.mp4: Unchanged skipping
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_00-30-02-795908.mp4: Sizes identical
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_00-30-02-795908.mp4: Unchanged skipping
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_01-50-02-824214.mp4: Sizes identical
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_01-50-02-824214.mp4: Unchanged skipping
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_01-30-02-814417.mp4: Sizes identical
2024/11/09 00:10:26 DEBUG : webcam/2024-11-02/2024-11-02_01-30-02-814417.mp4: Unchanged skipping
2024/11/09 00:10:26 DEBUG : S3 bucket XXXXXXXXXXX: Waiting for transfers to finish