What is the problem you are having with rclone?
I have set up rclone on my FreeNAS server to back up several OneDrive libraries. This runs successfully on all except for one (the largest, which is several hundreds of gigabytes and has thousands of subfolders). I get the following errors:
2020/12/08 11:11:17 ERROR : <specific folder>:error reading source directory: couldn't list files: activityLimitReached: throttledRequest: The request has been throttled
.... repetition across a selection folders ....
2020/12/08 12:35:58 ERROR : Local file system at <top folder>: not deleting files as there were IO errors
2020/12/08 12:35:58 ERROR : Local file system at <top folder>: not deleting directories as there were IO errors
2020/12/08 12:35:58 ERROR : Attempt 3/3 failed with 37 errors and: march failed with 36 error(s): first error: couldn't list files: activityLimitReached: throttledRequest: The request has been throttled
2020/12/08 12:35:58 Failed to sync with 37 errors: last error was: march failed with 36 error(s): first error: couldn't list files: activityLimitReached: throttledRequest: The request has been throttled
The script runs for circa 60-90 mins and the error typically repeats 25-50 times, every time for different folders.
Elsewhere on this forum, I found that using a client ID and secret key can address this issue. Unfortunately it did not, I am hence seeking additional suggestions. Thank you very much in advance!
What is your rclone version (output from rclone version)
rclone v1.50.2
Which OS you are using and how many bits (eg Windows 7, 64 bit)
FreeNAS-11.3-U5
Which cloud storage system are you using? (eg Google Drive)
OneDrive Business
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone sync <CONFIGNAME>: <target folder> --backup-dir <backup folder>
The rclone config contents with secrets removed.
[CONFIGNAME]
type = onedrive
token = {"access_token":"<token>","expiry":"2020-12-08T15:08:32.957086808+01:00"}
drive_id = <id>
drive_type = business
client_id = <id>
client_secret = <ww>
A log from the command with the -vv flag
2020/12/08 14:45:37 DEBUG : rclone: Version "v1.50.2" starting with parameters ["rclone" "sync" "<CONFIGNAME>:" "<target folder>" "--backup-dir" "<backup folder>" "-vv"]
2020/12/08 14:45:37 DEBUG : Using config file from "<folder>/.config/rclone/rclone.conf"
2020/12/08 14:45:38 DEBUG : <specific file>: Size and modification time the same (differ by 0s, within tolerance 1s)
2020/12/08 14:45:38 DEBUG : <specific file>: Unchanged skipping
2020/12/08 14:45:38 DEBUG : <url>: Size and modification time the same (differ by 0s, within tolerance 1s)
2020/12/08 14:45:38 DEBUG : <url>: Unchanged skipping
2020/12/08 14:45:38 DEBUG : <blabla> OneNote file not shown in directory listing
2020/12/08 14:45:38 DEBUG : <specific file>: Size and modification time the same (differ by 0s, within tolerance 1s)
2020/12/08 14:45:38 DEBUG : <specific file>: Unchanged skipping
2020/12/08 14:45:38 DEBUG : <specific file>: Size and modification time the same (differ by 0s, within tolerance 1s)
…
…
…
2020/12/08 14:46:37 INFO :
Transferred: 3.231M / 3.231 MBytes, 100%, 746.550 kBytes/s, ETA 0s
Errors: 0
Checks: 14240 / 14240, 100%
Transferred: 7 / 7, 100%
Elapsed time: 4.4s
…
…
…
2020/12/08 14:47:02 DEBUG : pacer: low level retry 1/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 14:47:02 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/12/08 14:47:02 DEBUG : pacer: Reducing sleep to 15ms
2020/12/08 14:47:02 DEBUG : pacer: Reducing sleep to 11.25ms
…
…
2020/12/08 14:47:02 DEBUG : pacer: low level retry 2/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 14:47:02 DEBUG : pacer: Rate limited, increasing sleep to 22.5ms
2020/12/08 14:47:02 DEBUG : pacer: Reducing sleep to 16.875ms
…
2020/12/08 14:47:02 DEBUG : pacer: Reducing sleep to 12.65625ms
…
2020/12/08 14:47:02 DEBUG : pacer: Reducing sleep to 10ms
…
2020/12/08 14:47:02 DEBUG : pacer: low level retry 3/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 14:47:02 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2020/12/08 14:47:02 DEBUG : pacer: low level retry 1/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 14:47:02 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2020/12/08 14:47:02 DEBUG : pacer: Reducing sleep to 30ms
…
…
…
…
…
…
2020/12/08 15:02:23 DEBUG : pacer: low level retry 3/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:25 DEBUG : pacer: low level retry 3/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:27 DEBUG : pacer: low level retry 3/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:29 DEBUG : pacer: low level retry 2/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:31 DEBUG : pacer: low level retry 4/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:33 DEBUG : pacer: low level retry 5/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:35 DEBUG : pacer: low level retry 3/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:37 INFO :
Transferred: 40.151M / 40.151 MBytes, 100%, 1.958 MBytes/s, ETA 0s
Errors: 20 (retrying may help)
Checks: 51375 / 51375, 100%
Transferred: 51 / 51, 100%
Elapsed time: 20.5s
2020/12/08 15:02:37 DEBUG : pacer: low level retry 5/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:39 DEBUG : pacer: low level retry 4/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:41 DEBUG : pacer: low level retry 4/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:43 DEBUG : pacer: low level retry 4/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:45 DEBUG : pacer: low level retry 3/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:47 DEBUG : pacer: low level retry 5/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:49 DEBUG : pacer: low level retry 6/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:51 DEBUG : pacer: low level retry 4/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:53 DEBUG : pacer: low level retry 6/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:55 DEBUG : pacer: low level retry 5/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:57 DEBUG : pacer: low level retry 5/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:02:59 DEBUG : pacer: low level retry 5/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:02 DEBUG : pacer: low level retry 4/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:04 DEBUG : pacer: low level retry 6/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:06 DEBUG : pacer: low level retry 7/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:08 DEBUG : pacer: low level retry 5/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:10 DEBUG : pacer: low level retry 7/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:12 DEBUG : pacer: low level retry 6/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:14 DEBUG : pacer: low level retry 6/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:16 DEBUG : pacer: low level retry 6/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:18 DEBUG : pacer: low level retry 5/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:20 DEBUG : pacer: low level retry 7/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:22 DEBUG : pacer: low level retry 8/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:24 DEBUG : pacer: low level retry 6/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:26 DEBUG : pacer: low level retry 8/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:28 DEBUG : pacer: low level retry 7/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:30 DEBUG : pacer: low level retry 7/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:32 DEBUG : pacer: low level retry 7/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:34 DEBUG : pacer: low level retry 6/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:36 DEBUG : pacer: low level retry 8/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:37 INFO :
Transferred: 40.151M / 40.151 MBytes, 100%, 1.958 MBytes/s, ETA 0s
Errors: 20 (retrying may help)
Checks: 51375 / 51375, 100%
Transferred: 51 / 51, 100%
Elapsed time: 20.5s
2020/12/08 15:03:38 DEBUG : pacer: low level retry 9/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2020/12/08 15:03:40 DEBUG : pacer: low level retry 7/10 (error activityLimitReached: throttledRequest: The request has been throttled)
….
etc…..