First of all, many thanks to rclone devs and community members.
What is the problem you are having with rclone?
I'm getting a few can't copy - source file is being updated errors that stop files from being deleted with rclone sync (with backup-dir).
Specifically, the errors for me concern some database files in my Apple Photo Library and another video file that Agent DVR (iSpy) is constantly writing a 24-hours timelapse to.
The can't copy - source file is being updated errors are not the problem. I like that the log shows that some files were being written to and that they have been skipped.
The problem is that these errors prevent all files and all folders that need to be deleted in the destination from being deleted, not just the files that were giving errors.
These are the errors:
not deleting files as there were IO errors
not deleting directories as there were IO errors
So, if there is even just 1 file being written to in the entire source while rclone is running, the sync command will fail to delete anything in the destination (basically performing a copy instead of a sync).
In my opinion, rclone is creating an unnecessary problem here. Rclone should just skip files that are being written to, log the skipped files with a NOTICE tag (not ERROR), and proceed like normal, deleting files and directories as requested. Maybe it should also not touch those specific files should they already be present in the destination, even if --delete-excluded is used.
Is there a way to achieve this behavior?
The solutions I see are:
Solution 1: the flag --local-no-check-updated
Here the problem is that files get uploaded no matter if they are being written to. This results in corrupted files or incomplete files being uploaded and then kept in the backup-dir, without warnings in the log.
Solution 2: the flag--ignore-errors
I guess that this could work. However, I suppose that this will ignore any error or i/o error and not just specifically can't copy - source file is being updated errors, which is concerning to me. (I haven't tested this yet, so I don't even know if files being updated are skipped or uploaded.)
Solution 3: exclude by name
Excluding the files that often or always cause these errors could be a solution. The problem is that the Apple Photos library database files are vaguely important (I would prefer not to skip them, especially since they do not always give problems). And the most recent timelapse has the date and time in the file name, so excluding it by file name is tricky. (maybe --exclude \Non-Backup\Downloads\iSpy\video\Logi\1_TimeLapse_$(Get-Date -Format "yyyy-MM-dd")_*.mp4" would work using powershell)
Solution 4: exclude by date with --min-age 1s
I'm not sure about this one. Feels to me that there is potential for unintentional exclusions. Worse yet, they would be silent exclusion, with nothing in the log and no way to find out what was excluded (unless I'm missing something).
Run the command 'rclone version' and share the full output of the command.
rclone v1.62.2
- os/version: Microsoft Windows 10 Pro 22H2 (64 bit)
- os/kernel: 10.0.19045.2728 Build 19045.2728.2728 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: cmount
Which cloud storage system are you using?
Dropbox
The command you were trying to run
this is on Windows using powershell
rclone sync "F:\" "Dropbox3:/Only Online/rclone/F - Downloads" `
--backup-dir="Dropbox3:/Only Online/rclone/π deleted/F - Downloads" `
--exclude-from "D:\- 10 - Dropbox\- 20 - more\Apps Data\rclone\exclude\exclude for downloads.txt" `
`
--log-level INFO `
--stats-file-name-length 0 `
--create-empty-src-dirs `
--suffix " -saved $(Get-Date -Format "yyyy-MM-dd HH.mm.ss")" --suffix-keep-extension `
--delete-excluded `
--tpslimit 12 --tpslimit-burst 0 `
`
--progress `
--log-file="T:\Non-Backup\logs\rclone\rclone log dropbox - downloads.txt" `
The rclone config contents with secrets removed.
[Dropbox]
type = dropbox
token = REDACTED
client_id = REDACTED
client_secret =REDACTED
[Gsuite]
type = drive
client_id = REDACTED
client_secret = REDACTED
token = REDACTED
team_drive =
[local]
type = local
[Dropbox2]
type = dropbox
client_id = REDACTED
client_secret = REDACTED
token = REDACTED
[Dropbox3]
type = dropbox
client_id = REDACTED
client_secret = REDACTED
token = REDACTED
A log from the command with the -vv flag
2023/03/25 08:55:10 ERROR : Dropbox root 'Only Online/rclone/F - Downloads': not deleting files as there were IO errors
2023/03/25 08:55:10 ERROR : Dropbox root 'Only Online/rclone/F - Downloads': not deleting directories as there were IO errors
2023/03/25 08:55:10 ERROR : Attempt 1/3 failed with 1 errors and: upload failed: Post "https://content.dropboxapi.com/2/files/upload_session/append_v2": can't copy - source file is being updated (size changed from 31195184 to 234618928)
2023/03/25 08:56:13 ERROR : Non-Backup/Downloads/iSpy/video/Logi/1_TimeLapse_2023-03-24_17-59-27.mp4: Failed to copy: upload failed: Post "https://content.dropboxapi.com/2/files/upload_session/append_v2": can't copy - source file is being updated (size changed from 236978224 to 251658288)
2023/03/25 08:57:08 ERROR : Dropbox root 'Only Online/rclone/F - Downloads': not deleting files as there were IO errors
2023/03/25 08:57:08 ERROR : Dropbox root 'Only Online/rclone/F - Downloads': not deleting directories as there were IO errors
2023/03/25 08:57:08 ERROR : Attempt 2/3 failed with 1 errors and: upload failed: Post "https://content.dropboxapi.com/2/files/upload_session/append_v2": can't copy - source file is being updated (size changed from 236978224 to 251658288)
2023/03/25 08:58:04 ERROR : Non-Backup/Downloads/iSpy/video/Logi/1_TimeLapse_2023-03-24_17-59-27.mp4: Failed to copy: upload failed: Post "https://content.dropboxapi.com/2/files/upload_session/append_v2": can't copy - source file is being updated (size changed from 251658288 to 252182576)
2023/03/25 08:59:05 ERROR : Dropbox root 'Only Online/rclone/F - Downloads': not deleting files as there were IO errors
2023/03/25 08:59:05 ERROR : Dropbox root 'Only Online/rclone/F - Downloads': not deleting directories as there were IO errors
2023/03/25 08:59:05 ERROR : Attempt 3/3 failed with 1 errors and: upload failed: Post "https://content.dropboxapi.com/2/files/upload_session/append_v2": can't copy - source file is being updated (size changed from 251658288 to 252182576)