Local Drive to OneDrive: Too many requests

Help and Support
1

What is the problem you are having with rclone?

I'm running into a Too Many Requests error trying to upload to a Sharepoint Site through OneDrive.
Over the holidays I worked with two separate Office 365 Tenants - I migrated one successfully from Box . Com to OneDrive with no problems. The second one is a local drive to two SharePoints that is giving me problems. The smaller one I Was able to migrate, but the large one I am not.

I started the migration mid-december and have successfully uploaded about 48GB/60GB. Then it started to get really slow and struggle. Logs showed too many requests. I gave it a break with no commands from Dec 23 until today Jan 4. Immediately this morning I got too many requests errors. I am using a custom OneDrive ID.

I am using rclone vs just rsync into the Sync folder because I appreciate the fixing of illegal file names (I have thousands)

What is your rclone version (output from rclone version)

1.53.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

macOS 10.15.7 (64 but)

Which cloud storage system are you using? (eg Google Drive)

OneDrive (Sharepoint)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -vv --interactive --ignore-size --ignore-checksum copy /Volumes/Data/ThinkHQ/ ThinkHQ:General

The rclone config contents with secrets removed.

[ThinkHQ]
type = onedrive
token = [redacted]
drive_id = b!lF2iWhGNbEWzKMSZGy9GVGs8lHI5C-VIl4v0UF-ON5ny-UeSo5RgRb_OrtLD08id
drive_type = documentLibrary
client_id = 4c6d6a45-91d2-48f9-8d1b-5458faca52ae
client_secret = [redacted]

A log from the command with the -vv flag

2021/01/04 07:22:01 DEBUG : Too many requests. Trying again in 43 seconds.
2021/01/04 07:22:01 DEBUG : pacer: low level retry 1/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2021/01/04 07:22:01 DEBUG : pacer: Rate limited, increasing sleep to 43s
2021/01/04 07:22:01 DEBUG : pacer: Reducing sleep to 32.25s
2021/01/04 07:22:01 DEBUG : pacer: Reducing sleep to 24.1875s
2021/01/04 07:22:01 DEBUG : pacer: Reducing sleep to 18.140625s
2021/01/04 07:22:01 DEBUG : pacer: Reducing sleep to 13.60546875s
2021/01/04 07:22:01 DEBUG : pacer: Reducing sleep to 10.204101562s
2021/01/04 07:22:01 DEBUG : pacer: Reducing sleep to 7.653076171s
2021/01/04 07:22:01 DEBUG : pacer: Reducing sleep to 5.739807128s
2021/01/04 07:22:01 DEBUG : pacer: Reducing sleep to 4.304855346s
2021/01/04 07:22:28 INFO  : 
Transferred:   	         0 / 12.791 GBytes, 0%, 0 Bytes/s, ETA -
Checks:              6753 / 9883, 68%
Transferred:            1 / 7934, 0%
Elapsed time:       2m2.8s

2021/01/04 07:22:44 DEBUG : Too many requests. Trying again in 231 seconds.
2021/01/04 07:22:44 DEBUG : pacer: low level retry 1/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2021/01/04 07:22:44 DEBUG : pacer: Rate limited, increasing sleep to 3m51s
2021/01/04 07:22:48 DEBUG : Too many requests. Trying again in 39 seconds.
2021/01/04 07:22:48 DEBUG : pacer: low level retry 1/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2021/01/04 07:22:48 DEBUG : pacer: Rate limited, increasing sleep to 39s
2

hello and welcome to the forum,

have you tried to search the forum, for example,
OneDrive activityLimitReached / throttledRequest error despite using client ID and key - #4 by asdffdsa

3

Thanks for the that link. I did search, but did not come across that post. I will review. Thanks.

4

Onedrive for Business - activity limits and throttling

5

I have added in a --tpslimit of 5 and it seems to be helping. I suspect that when rclone is checking all the existing files in sharepoint it's creating the rate limit. So this slowed that down and uploads are proceeding.

I notice it still appears to be transferring 4 or 5 files at a time (which I think is ok). TPS Limit documentation lists limiting http requests. I suspect that an upload would be a request, and so in theory it could still overload itself. Where is the documentation on concurrent uploads?

Thanks for the fast help!

6

sure no problem,

you can reduce the number of transfers and/or checkers.

7

Perfect. Found those in the docs. But perhaps I spoke too soon. After 4 hours, the average speed has dropped to 7 kBytes a sec with over 3 weeks remaining. It uploaded about 100MB in the first couple of minutes and basically nothing in the remaining time.

So It's not erroring on Too many requests anymore, and nor does the log show throttled, but it would appear I'm still being limited in some capacity. I've since revised my command to include the following restrictions:

--tpslimit 5
--checkers 1
--transfers 1
--exclude .DS_Store

I've confirmed its not the network throttling, so I'm left to believe I'm just at the perils of Microsoft here. I've started a new copy operations and with those parameters it is behaving a bit better - done about 330M in 26 minutes.

8

what command are you using?
what is the size of data that needs to be transferred?
how many folders?
how many files?
what is the result of speedtest.net?

9

Same Command:
rclone -vv --ignore-size --ignore-checksum --tpslimit 5 --checkers 1 --transfers 1 --exclude .DS_Store copy /Volumes/Data/ThinkHQ/ ThinkHQ:General
The data set is about 60GB. The first 40GB had no problems. Mostly small documents.
About 44,000 Files
About 6000 Folders
It's on a 100/15 Coax connection

1 Like
10

that is a very slow upload.
what was the ping?

ok, we have onedrive experts, hopefully one will stop by soon.

11

Ping is Good. In the end it was respectable:

Transferred:   	   20.316G / 20.316 GBytes, 100%, 481.496 kBytes/s, ETA 0s
Checks:             50049 / 50049, 100%
Transferred:        13752 / 13752, 100%
Elapsed time:  12h39m27.9s

I'd be more concerned if this was a scheduled regular sync. But as it's a migration of data - I'm good now and we can just do an update this evening and then move SharePoint to being production tomorrow.

12

I've noticed the most random rate limiting with OneDrive. Seems they have multiple (not published) limits and rate limit tables. The limits are way more aggressive for free users vs paid (both Personal and Business)

When I face the limit, best I can see to work is let it "breath" for an hour and then restart the same script. I've tried fruitless to find the limits, but seem they depend on their load and overall cloud health as I've been able to upload 400GB on one day, but get throttled on exactly the same command/data after 20GB.

This affects all applications outside their desktop app, even mover.io (from Microsoft) gets limited the same.

2 Likes
13

Yup. I agree. On the weekends I get full 1gbps upload speed where on other days its 500-600mbps.

14

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.