How to Avoid Getting Throttled

is there a way to have rclone use a database to track file changes, as opposed to just comparing the source/destination for every sync? when using rclone sync with large drives, such as 400,000+ files, i keep getting throttled due to exceeded activity. (OneDrive Business is my remote)

a large part of what causes the problem, actually, is rclone's method of dealing with an errror. as rclone is checking the source/destination for differences if at progress 300,000 there is one service error (usually it's serviceunavailable) rclone will finish checking the 400,000 files, and then check them all again because of the ONE error. can't it just check the error file? i quite frequently land up having 1,600,000/1,600,000 files checked. that's where i end up getting throttled.

It depends on the version of rclone and what the actual error is.

Some errors retry, some don't.

Can you include what version you are using and a log of the error message?

the error does retry; rclone will retry the sync 3 times. but that means it ends up checking all over the several hundred-thousand files 3 times. that's what i can't have.

i'm using rclone 1.53.0. i'm not sure how to extract the error log :confused: if you tell me the flag to use i'd be more than happy to.

This is a known problem with rclone at the moment, one which should be addressed in the next couple of releases.

In the mean time there is quite a good work-around.

You can do a "top-up" sync like this

rclone copy --max-age 1h --no-traverse path/to/source remote:path/to/destination

The --max-age 1h will mean that only files younger than 1 hour will be copied and the --no-traverse will mean rclone does not look through the entire destination file tree.

Note that this uses rclone copy not rclone sync so it only copies new file, it doesn't delete any files on the destination.

This works very well for backing up recently changed files quickly. You still need to do a sync to propagate deletions but you can do it less often.

alright man. really glad to hear it's being addressed in the next couple of releases. as simply of a utility as rclone is, it's the perfect backup solution for me.

in the meantime. i'll use the work-around you mentioned.


This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.