OneDrive: rclone getting throttled and pacer sleep time skyrockets to >5m

First, sorry for the very late reply. I had to request re-opening of the post to be able to reply here, since it was automatically closed, but I really wanted to post my final feedback and keep it close to the original report. To summarize: I had these OneDrive sync operations that triggered throttling very easily, and the amount of throttling was unsustainable (>5 minutes). To add one more detail that maybe was not better specified, I'm using a OneDrive for Business account and the company uses some SharePoint features, such as a SharePoint drive/link added in the root of "My Files" as a shortcut, but I'm not currently writing/reading from this folder. Still, looking to other feedbacks it seems throttling is often linked to use SharePoint features. Following the suggestions of commenters and rclone guides I:

  • Set up correctly a client ID and key as explained in the guide;
  • Attempted to create different user agent combinations, such as ISV vs NONISV, AppName(s) as explained by Microsoft. Still Microsoft doesn't explain very clearly what these attributes should be.

After all these attempts and upgrade to latest 1.57 version of rclone, I was still experiencing severe throttling, confirming that the issue is reproducible in my scenario and not related to use of other unofficial clients. Fortunately, I stumbled upon this post, that basically reduces the amount of concurrent transactions, and the overall rate of them. In particular I tried various combinations of --checkers. I tried to reduce it to 4, 2 and I always was able to reproduce the throttling. Finally I set it to 1 and the throttling finally stopped arising. The bandwidth performance is greatly reduced but at least the connection doesn't unpredictably hang anymore.

In conclusion, in my scenario (which may be a combination of the type of account, the exact infrastructure and kind of the cloud service being used by the company, the type of the data being synced, the use of some particular European regional server and other variables) any use of concurrent/parallel transactions decided by the --checkers options in rclone is strictly forbidden, no matter what else I do to better decorate the connection. I still have the feeling something is wrong in rclone with regard to OneDrive number of generated transactions (which seems to be huge and not optimal in case of synchronization which huge number of flies and folders), but I'm satisfied enough with the result achieved. I would love to hear if there are updates with regard to better handling of throttling/reduction of number of transactions in rclone with OneDrive, even if I can't promise I will be able to test them promptly, similarly to the delay in posting my feedback with regard to the synchronization task I was implementing.

2 Likes