OneDrive: rclone getting throttled and pacer sleep time skyrockets to >5m

[asdffdsa (jojothehumanmonkey)]

hi, not sure your exact use case but as a possible workaround.

reduce the number of api calls by use filtering; tho there can be caveats.
--max-age 24 - would only sync files where the source file modtime is older than 24 hours.
if the source file modtime is older than 24 hours, rclone will not check onedrive and thus not use api calls.

[Ole (Ole Frost)]

@Francesco_Pretto

This is just a feeling because you explained me that rclone has (really?) no better way to sync the whole tree than querying recursively (and concurrently) for folders content.

Correct, there is no better way to do it - and it can be lightning fast as you see in the above example posted by Jojo.

@Francesco_Pretto

• experienced a sync tool that did the same, but faster?

When syncing some large whole tree accounts from scratch with the official OneDrive client in the company don't seems to hang the way rclone did for me, but I can't inspect them the same way, so I can't say for true.

The native Windows OneDrive Client does the same as rclone when it syncs for the first time and at approximately the same speed – and it may get throttled too. Pay attention to the number next to “Processing changes” in your OneDrive Client extended view – the throttling is quite easy to spot once you know the pattern to look for. The client and File Explorer will remain responsive, but content will be out of sync, thus higher risk of sync conflicts.

I got myself into trouble when I did a series of aggressive rclone throttling tests, because my Windows OneDrive and iOS OneDrive got throttled too – almost blocking me from cross device file synchronization in a couple of days…

@Francesco_Pretto

Really there's not a single API to download the whole index starting from a location, with the server optimizing the transfer as much as possible (file names are highly compressible) and that is free of throttling by definition because the server also decides a paging strategy? Sure: this single API call can't be made parallel, but it seems to me a better strategy for me for this kind of unidirectional synchronizations. When the index is fully downloaded, then you can do any amount of needed transfers concurrently. This is just a speculation of mines: you (rclone devs) may already have decided this is not a good strategy because of how modern clouds work. But as a matter of fact I notice throttling also when syncing unchanged trees, where the amount of actual file data transfers is (and will be) 0 bytes.

The limit isn’t due to the technical design of OneDrive or rclone, both can support speeds like the above example from Wasabi.

I see the rate limit/throttling as a purely commercial decision, to keep the price of OneDrive (the server park) down and push the most demanding (professional/power) users over to other more expensive (and better suited) alternatives. It is the same concept for Dropbox, Google Drive, etc.

If you do a first-time sync with a lot of small files, then you hit a (somewhat dynamic) limit on file creations per second. There is a similar limit on deletions.

If you do a first-time sync with large files, then you hit another (somewhat dynamic) limit on bytes uploaded per second. There is a similar limit of downloads.

If you do a sync without changes, then you hit yet another (somewhat dynamic) limit on directory content lookups per second.

OneDrive enforces the limits by slowing down the responses to you, which sometimes result in a queue of requests building up at their servers. If this queue is too long for too long time, then OneDrive tells all the clients connected to that account (or tenant) to stop sending further requests for a while. This is the “429/Too Many request” response you have seen.

@Francesco_Pretto

Have you
• tried --fast-list?

This was one of the first tries I did and didn't seem to help me. Sorry, I haven't tried it more recently.

It is worth a try, but the outcome heavily depends on the characteristics of your data and the balance between checking and transferring.

It has no positive effect for my daily usage.

@Francesco_Pretto

• traced the rclone communication with --dump bodies?

Sorry, no. Should I?

No, unless you understand the OneDrive REST API in depth and really want to check if rclone is sending optimal requests.