Rclone clone slows over time, starts fast and gets slower

What is the problem you are having with rclone?

When performing a sizeable incremental sync, the synchronization slows over time. Stopping the sync and starting over again instantly runs fast again.

It appears to be due to automatic throttling.

What I can't explain is why is it immediately fast again for a while if I stop the first execution and start a new one. Is the destination cloud service more permissive at first and then increases throttling over time?

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.1

  • os/version: ubuntu 22.04 (64 bit)
  • os/kernel: 5.15.0-78-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.20.6
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

I have seen this error on different combinations of source and destination cloud storage, the current issue is being encountered with sync from Microsoft OneDrive -> IDrive e2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

clone sync --cutoff-mode soft --log-level INFO --log-file ${logfile} OneDrive:/ IDrive:/

The rclone config contents with secrets removed.

type = onedrive
token = {"access_token":"-redacted-","token_type":"Bearer","refresh_token":"-redacted-","expiry":"2023-08-09T12:17:53.993506934-05:00"}
drive_id = -redacted-
drive_type = business

type = s3
provider = IDrive
access_key_id = -redacted-
secret_access_key = -redacted-
acl = bucket-owner-full-control
endpoint = v3l8.da.idrivee2-34.com

A log from the command with the -vv flag

The log is extremely long, after only one minute there are over 8000 lines.

But what seems to be of the most relevance is the following.

2023/08/09 11:49:00 DEBUG : Too many requests. Trying again in 322 seconds.
2023/08/09 11:49:00 DEBUG : pacer: low level retry 2/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2023/08/09 11:49:00 DEBUG : pacer: Rate limited, increasing sleep to 5m22s

What seems odd is that the -vv logging doesn't indicate if the 'too many requests' is coming from the source or the destination. I'm assuming the destination because there is a backlog of files to be copied, but the copying is what has halted.

IMO it is OneDrive which is well known for very aggressive throttling.

First most important thing is to create your own clientID/secret - details here - otherwise you share built in one with who knows how many other rclone users - results are usually exactly like yours - almost impossible to perform anything more complex or involving more data.

When you create clientID/secret - I recommend you recreate your [OneDrive] remote - it is the easiest way to ensure that they are bound to the token.

This should alleviate most issues - if not completely then we can try to throttle sync command as well.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.