OneDrive activityLimitReached / throttledRequest error despite using client ID and key

What is the problem you are having with rclone?

I have set up rclone on my FreeNAS server to back up several OneDrive libraries. This runs successfully on all except for one (the largest, which is several hundreds of gigabytes and has thousands of subfolders). I get the following errors:

2020/12/08 11:11:17 ERROR : <specific folder>:error reading source directory: couldn't list files: activityLimitReached: throttledRequest: The request has been throttled
.... repetition across a selection folders ....
2020/12/08 12:35:58 ERROR : Local file system at <top folder>: not deleting files as there were IO errors
2020/12/08 12:35:58 ERROR : Local file system at  <top folder>: not deleting directories as there were IO errors
2020/12/08 12:35:58 ERROR : Attempt 3/3 failed with 37 errors and: march failed with 36 error(s): first error: couldn't list files: activityLimitReached: throttledRequest: The request has been throttled
2020/12/08 12:35:58 Failed to sync with 37 errors: last error was: march failed with 36 error(s): first error: couldn't list files: activityLimitReached: throttledRequest: The request has been throttled

The script runs for circa 60-90 mins and the error typically repeats 25-50 times, every time for different folders.

Elsewhere on this forum, I found that using a client ID and secret key can address this issue. Unfortunately it did not, I am hence seeking additional suggestions. Thank you very much in advance!

What is your rclone version (output from rclone version)

rclone v1.50.2

Which OS you are using and how many bits (eg Windows 7, 64 bit)

FreeNAS-11.3-U5

Which cloud storage system are you using? (eg Google Drive)

OneDrive Business

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync <CONFIGNAME>: <target folder> --backup-dir <backup folder>

The rclone config contents with secrets removed.

[CONFIGNAME]
type = onedrive
token = {"access_token":"<token>","expiry":"2020-12-08T15:08:32.957086808+01:00"}
drive_id = <id>
drive_type = business
client_id = <id>
client_secret = <ww>

A log from the command with the -vv flag

2020/12/08 14:45:37 DEBUG : rclone: Version "v1.50.2" starting with parameters ["rclone" "sync" "<CONFIGNAME>:" "<target folder>" "--backup-dir" "<backup folder>" "-vv"]
2020/12/08 14:45:37 DEBUG : Using config file from "<folder>/.config/rclone/rclone.conf"
2020/12/08 14:45:38 DEBUG : <specific file>: Size and modification time the same (differ by 0s, within tolerance 1s)
2020/12/08 14:45:38 DEBUG : <specific file>: Unchanged skipping
2020/12/08 14:45:38 DEBUG : <url>: Size and modification time the same (differ by 0s, within tolerance 1s)
2020/12/08 14:45:38 DEBUG : <url>: Unchanged skipping
2020/12/08 14:45:38 DEBUG : <blabla> OneNote file not shown in directory listing
2020/12/08 14:45:38 DEBUG : <specific file>: Size and modification time the same (differ by 0s, within tolerance 1s)
2020/12/08 14:45:38 DEBUG : <specific file>: Unchanged skipping
2020/12/08 14:45:38 DEBUG : <specific file>: Size and modification time the same (differ by 0s, within tolerance 1s)
…
…
…
2020/12/08 14:46:37 INFO  : 
Transferred:   	    3.231M / 3.231 MBytes, 100%, 746.550 kBytes/s, ETA 0s
Errors:                 0
Checks:             14240 / 14240, 100%
Transferred:            7 / 7, 100%
Elapsed time:        4.4s
…
…
…
2020/12/08 14:47:02 DEBUG : pacer: low level retry 1/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 14:47:02 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/12/08 14:47:02 DEBUG : pacer: Reducing sleep to 15ms
2020/12/08 14:47:02 DEBUG : pacer: Reducing sleep to 11.25ms
…
…
2020/12/08 14:47:02 DEBUG : pacer: low level retry 2/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 14:47:02 DEBUG : pacer: Rate limited, increasing sleep to 22.5ms
2020/12/08 14:47:02 DEBUG : pacer: Reducing sleep to 16.875ms
…
2020/12/08 14:47:02 DEBUG : pacer: Reducing sleep to 12.65625ms
…
2020/12/08 14:47:02 DEBUG : pacer: Reducing sleep to 10ms
…
2020/12/08 14:47:02 DEBUG : pacer: low level retry 3/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 14:47:02 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/12/08 14:47:02 DEBUG : pacer: low level retry 1/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 14:47:02 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2020/12/08 14:47:02 DEBUG : pacer: Reducing sleep to 30ms
…
…
…
…
…
…
2020/12/08 15:02:23 DEBUG : pacer: low level retry 3/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:25 DEBUG : pacer: low level retry 3/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:27 DEBUG : pacer: low level retry 3/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:29 DEBUG : pacer: low level retry 2/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:31 DEBUG : pacer: low level retry 4/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:33 DEBUG : pacer: low level retry 5/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:35 DEBUG : pacer: low level retry 3/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:37 INFO  : 
Transferred:   	   40.151M / 40.151 MBytes, 100%, 1.958 MBytes/s, ETA 0s
Errors:                20 (retrying may help)
Checks:             51375 / 51375, 100%
Transferred:           51 / 51, 100%
Elapsed time:       20.5s

2020/12/08 15:02:37 DEBUG : pacer: low level retry 5/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:39 DEBUG : pacer: low level retry 4/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:41 DEBUG : pacer: low level retry 4/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:43 DEBUG : pacer: low level retry 4/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:45 DEBUG : pacer: low level retry 3/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:47 DEBUG : pacer: low level retry 5/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:49 DEBUG : pacer: low level retry 6/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:51 DEBUG : pacer: low level retry 4/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:53 DEBUG : pacer: low level retry 6/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:55 DEBUG : pacer: low level retry 5/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:57 DEBUG : pacer: low level retry 5/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:59 DEBUG : pacer: low level retry 5/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:02 DEBUG : pacer: low level retry 4/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:04 DEBUG : pacer: low level retry 6/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:06 DEBUG : pacer: low level retry 7/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:08 DEBUG : pacer: low level retry 5/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:10 DEBUG : pacer: low level retry 7/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:12 DEBUG : pacer: low level retry 6/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:14 DEBUG : pacer: low level retry 6/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:16 DEBUG : pacer: low level retry 6/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:18 DEBUG : pacer: low level retry 5/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:20 DEBUG : pacer: low level retry 7/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:22 DEBUG : pacer: low level retry 8/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:24 DEBUG : pacer: low level retry 6/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:26 DEBUG : pacer: low level retry 8/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:28 DEBUG : pacer: low level retry 7/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:30 DEBUG : pacer: low level retry 7/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:32 DEBUG : pacer: low level retry 7/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:34 DEBUG : pacer: low level retry 6/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:36 DEBUG : pacer: low level retry 8/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:37 INFO  : 
Transferred:   	   40.151M / 40.151 MBytes, 100%, 1.958 MBytes/s, ETA 0s
Errors:                20 (retrying may help)
Checks:             51375 / 51375, 100%
Transferred:           51 / 51, 100%
Elapsed time:       20.5s

2020/12/08 15:03:38 DEBUG : pacer: low level retry 9/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:40 DEBUG : pacer: low level retry 7/10 (error activityLimitReached: throttledRequest: The request has been throttled)
….
etc…..



hello and welcome to the forum,

the latest stable version of rclone is v1.53.3, perhaps update and test again.

have you tried searching the forum, for example,
Onedrive for Business - activity limits and throttling

Thanks @asdffdsa,
As far as I am aware, I cannot update rclone on FreeNAS, is that correct?
Also, I have searched the forum and only found suggestions related to client IDs and keys, which I already implemented.
Any other suggestion would be great!

the best way to install/update rclone is this.
https://rclone.org/downloads/#script-download-and-install

best to update and test again.

Sorry, user root is not allowed to execute '/bin/bash' as root on freenas.local.

You can try using --tpslimit to slow rclone down that usually helps. Try --tpslimit 10 and increase/decrease if that works/fails.

1 Like

I feel a bit stupid... I tested this, but without any effect. Apparently I did something wrong as my second attempt did yield the sought for effect. Thanks!

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.