What is the problem you are having with rclone?
I need to copy almost 1TB from Onedrive to Dropbox but the transfer fails and I received the following error:
Too many requests. Trying again in 300 seconds.
2022/06/21 12:29:29 DEBUG : pacer: low level retry 1/10 (error 429 Too Many Requests: )
2022/06/21 12:29:29 DEBUG : pacer: Rate limited, increasing sleep to 5m0s
2022/06/21 12:29:29 DEBUG : Couldn't decode error response: EOF
2022/06/21 12:29:29 DEBUG : Too many requests. Trying again in 1226 seconds.
2022/06/21 12:29:29 DEBUG : pacer: low level retry 2/10 (error 429 Too Many Requests: )
2022/06/21 12:29:29 DEBUG : pacer: Rate limited, increasing sleep to 20m26s
Run the command 'rclone version' and share the full output of the command.
rclone v1.58.1
os/version: debian 11.3 (64 bit)
os/kernel: 5.10.0-8-amd64 (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.17.9
go/linking: static
go/tags: none
Which cloud storage system are you using? (eg Google Drive)
One Drive
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone sync ode: dbe: --tpslimit=5 --checkers=2 --progress --vv
The rclone config contents with secrets removed.
Paste config here
A log from the command with the -vv
flag
022-06-21 12:43:14 DEBUG : pacer: Rate limited, increasing sleep to 1h16m6s
2022-06-21 12:43:14 DEBUG : Couldn't decode error response: EOF
2022-06-21 12:43:14 DEBUG : Too many requests. Trying again in 4566 seconds.
2022-06-21 12:43:14 DEBUG : pacer: low level retry 1/10 (error 429 Too Many Requests: )
2022-06-21 12:43:14 DEBUG : Couldn't decode error response: EOF
2022-06-21 12:43:14 DEBUG : Too many requests. Trying again in 4565 seconds.
2022-06-21 12:43:14 DEBUG : pacer: low level retry 1/10 (error 429 Too Many Requests: )
2022-06-21 12:43:14 DEBUG : pacer: Rate limited, increasing sleep to 1h16m5s
2022-06-21 12:43:15 DEBUG : Couldn't decode error response: EOF
2022-06-21 12:43:15 DEBUG : Too many requests. Trying again in 4565 seconds.
2022-06-21 12:43:15 DEBUG : pacer: low level retry 1/10 (error 429 Too Many
asdffdsa
(jojothehumanmonkey)
June 21, 2022, 7:08pm
2
hi,
the following info is missing, please post:
--- the redacted config file.
--- top 30 lines of a debug log.
Ole
(Ole Frost)
June 21, 2022, 7:25pm
3
Hi Jean Paul,
You are most likely being throttled by OneDrive.
I suggest you read these posts:
Hey again @Ole
All done. Copied over 1TB to Google Workspace account. Seemed that no matter what i did, OneDrive was sensitive. What i found was:
When i left things overnight and started again using clone --ignore-existing --transfesr=4 --checkers=16 copy... , i would get ~80-90GB before seeing exceeded threshold on the OneDrive side of things. Speed was as you said, around 13-20 Mbyte/s.
I could then restart after a few hours using the previous flags or also dropping the transfers to …
Hi Francesco,
The sleep times you see are due to rclone respecting the Retry-After time returned in the HTTP 429 (Too many requests) responses from OneDrive. There is no way to avoid them for sure, they depend on many parameters including the current load on the OneDrive servers hosting your data.
You can see it being received and handled in this line of your log:
2021-10-02 18:56:48 DEBUG : Too many requests. Trying again in 322 seconds.
You can however find some tips and experiences to re…
I just ran my sync from OneDrive to GoogleDrive and it had the usual speed around 12 Mbyte/s. I use my own client ID on both remotes. The only tuning parameter is --checkers=16, the rest are defaults (that is no --tpslimit, --transfers=4 etc.)
I have found that it is best to stay just under the OneDrive throttle limit all the time, and that corresponds to the rclone default settings in most situations. This is especially important with respect to transfers. OneDrive Personal will only do 4 tran…
asdffdsa
(jojothehumanmonkey)
June 21, 2022, 10:52pm
4
good chance the OP never created client id/secret for the remotes.
asdffdsa
(jojothehumanmonkey)
June 21, 2022, 11:19pm
6
that is you, my fellow rcloner, the O riginal P oster.
Oh =) I have crypt mounts. The client ID is for the main remote or for the crypt one? Or both?
and the Client ID has expiration date?
asdffdsa
(jojothehumanmonkey)
June 21, 2022, 11:33pm
10
it uses a token that expires, usually in one hour.
rclone will refresh the token as needed.
Ole
(Ole Frost)
June 22, 2022, 7:57am
11
I don't see a positive effect of a personal clientID/secret for OneDrive and don't use it anymore.
I don't know about Dropbox, but don't think it will make any difference since OneDrive seems to be the bottleneck.
My simple and easy recommendation for OneDrive is to use default tuning parameters and have a lot of patience with large transfers; and then restart the command if no progress at all in 24 hours.
So I would remove --tpslimit and --checkers (unless needed by Dropbox) and then be prepared for a sync that probably takes several days and sometimes pauses in long periods (minutes/hours).
Finally, I suggest you try to confirm that the throttling errors are indeed from OneDrive and not from Dropbox. You can do this by syncing a few folders from OneDrive to your local disk after the above throttling has started. That should result in the same behavior/errors - if not then try syncing from local disk to DropBox.
system
(system)
Closed
July 22, 2022, 7:58am
12
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.