[SOLVED] Rclone sync never finishes, keeps going through the same files over and over again

Hello Everyone,

Sync’ing a large (~8M files, ~4TB) from a local filesystem to an encrypted iunlimited GDrive remote , and it never finishes…

Apart from the time it is taking (more than 24h now), what tipped me to the issue is that the “Checks:” line on the stats block is now much larger than the total number of files on the source (local) filesystem: it is now showing “Checks: 15879348” and still growing, and as mentioned this filesystem only has about 8M files… and in fact, I see the exact same files being checked over and over again, for example:

grep 'REDACTED REDACTED-REDACTED - REDACTED REDACTED REDACTED.mobi' ~/20170813Z051207_rclone_sync_REDACTED_egd

Shows the following:

2017/08/13 14:32:29 DEBUG : REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED REDACTED-REDACTED - REDACTED REDACTED REDACTED.mobi: Size and modification time the same (differ by 0s, within tolerance 1s)
2017/08/13 14:32:29 DEBUG : REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED REDACTED-REDACTED - REDACTED REDACTED REDACTED.mobi: Unchanged skipping
2017/08/13 17:04:16 DEBUG : REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED REDACTED-REDACTED - REDACTED REDACTED REDACTED.mobi: Size and modification time the same (differ by 0s, within tolerance 1s)
2017/08/13 17:04:16 DEBUG : REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED REDACTED-REDACTED - REDACTED REDACTED REDACTED.mobi: Unchanged skipping
2017/08/13 19:37:17 DEBUG : REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED REDACTED-REDACTED - REDACTED REDACTED REDACTED.mobi: Size and modification time the same (differ by 0s, within tolerance 1s)
2017/08/13 19:37:17 DEBUG : REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED/REDACTED REDACTED-REDACTED - REDACTED REDACTED REDACTED.mobi: Unchanged skipping

It is still running, and I have complete logs (-v -v --logfile), so what should I look for in order to solve this (or at least properly document it as a bug)?

BTW, I faintly remember coming across something like this many moons ago (perhaps it was while I was still suffering under ACD), but for the heck of it I can’t neither remember details nor find it on the forum or on github…

Thanks in advance,
– Durval.

Hello,

I figured it out when, after a few more hours, rclone finally finished with the following as the two last lines in the log:

2017/08/13 19:45:21 ERROR : Attempt 3/3 failed with 0 errors and: error reading source directory "REDACTED": failed to open directory "REDACTED": open REDACTED: permission denied
2017/08/13 19:45:21 Failed to sync: error reading source directory "REDACTED": failed to open directory "REDACTED": open REDACTED: permission denied

(“REDACTED” above has nothing to do with any of the “REDACTED” parts of the DEBUG messages I transcribed on my previous post).

So, what I think happened is that each time rclone hit the above permission error, it interpreted it as a “low level error” and retried everything again from the start until it hit the maximum “–low-level-retries” setting, which by default is 3 (that’s what the “Attempt 3/3” above is telling), and only then aborted. This is what I faintly remembered from a previous occasion, but now it’s recorded here, so anyone going through that can find it via Google (including myself a few months from now :wink: ).

I re-executed the exact same rclone command as root (so no more permission errors), and now it seems to be progressing OK: after almost another day of processing, it has gone through 7634119 checks and 85014 transfers (totalling 27.2 GBytes) and should take at least a few more hours to finish.

A case could be made on whether retrying everything from the start because of a permission error on one of the local files is a sensible strategy for rclone… I would say that a permission error can hardly be called a “low level error” and it would very probably not “fix itself” if rclone just tries again… so perhaps for rclone just aborting immediately when hitting one of these errors would be a better strategy? It would certainly have saved me a lot of hours and puzzling…

Cheers,
– Durval.