Onedrive for Business - activity limits and throttling


#1

I’m having issues with using Onedrive for Business with rclone. The error I’m getting is

 Failed to create file system for "remote:": failed to get root: activityLimitReached: throttledRequest: The request has been throttled

How do I work around this?


#2

Are the activity limits for onedrive for business documented?

If you want to slow rclone down you have several options, --tpslimit is probably the most effective in avoiding bans. --bwlimit may help too.


#3

I haven’t seen anything about the limits, but I wasn’t able to do any more than run rclone size more than once - there’s only about 80GB of files, and less than 1000.


#4

That can’t be more than a few 100 operations - that sounds really strange…


#5

Hi. I have the same issue with One Drive for Business accounts. I have tried all manner of adjusting combinations of --bwlimit, --tpslimit, --low-level-retries and --contimeout in an effort to overcome this.

I’m familiar with using rsync successfully since many years and I’m really appreciating that there is a tool like clone in development to achieve similar results with cloud services. Unfortunately, I’m falling at the first hurdle with OneDrive for Business of even getting an lsd command to work consistently, and that’s before I start trying to use copy or sync.

I’m not a developer but wondered if there is anything here that might help resolve this?

Thanks for your kind attention.


#6

That is a useful link - thanks.

Rclone should be doing retries with exponential backoff when it gets a throttled message.

Can you run the command with -vv and post the log please so I can check?


#7

I saw your post in Onedrive for Business errors: “Failed to mkdir: 429” and “failed to get root: 429” and I’ve replied there


#8

I can confirm that rclone does backs-off, as expected.

2018/05/15 11:09:46 INFO : Starting HTTP transaction limiter: max 0.5 transactions/s with burst 1
2018/05/15 11:09:47 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2018/05/15 11:09:47 DEBUG : pacer: low level retry 1/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/15 11:09:48 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2018/05/15 11:09:48 DEBUG : pacer: low level retry 2/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/15 11:09:50 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2018/05/15 11:09:50 DEBUG : pacer: low level retry 3/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/15 11:09:52 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2018/05/15 11:09:52 DEBUG : pacer: low level retry 4/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/15 11:09:54 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2018/05/15 11:09:54 DEBUG : pacer: low level retry 5/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/15 11:09:56 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2018/05/15 11:09:56 DEBUG : pacer: low level retry 6/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/15 11:09:58 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2018/05/15 11:09:58 DEBUG : pacer: low level retry 7/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/15 11:10:00 DEBUG : pacer: Rate limited, increasing sleep to 2s
2018/05/15 11:10:00 DEBUG : pacer: low level retry 8/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/15 11:10:02 DEBUG : pacer: low level retry 9/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2018/05/15 11:10:04 DEBUG : pacer: low level retry 10/10 (error activityLimitReached: throttledRequest: The request has been throttled)