Google to one drive cloud sync too slow

Hello,
i try sync files to one drive,
i hire a vps for this job, i am using this command for sync.

" rclone sync google:test onedrive:test -v --progress --transfers=2 --ignore-existing --retries 2 --multi-thread-cutoff=1 "

storage server has 10gbit network speed and sometimes upload speed stuck somewhere. It shows generally 9 mb/seconds sometimes kbyte somtimes byte

i try upload 5 paralel users for stay away from one drive upload limit.
that means, i split my files to 5 parts, and every parts going to diffrent one drive user. I have 5 users now.

any help for me ?
thank you.

hello,

when you posted where was a template of questions.
help us to help you and answer the questions.

Hi etn-icin,

It is hard to give specific advice without having a much more detailed picture of your situation.

Perhaps you can find some useful advice in these threads/posts on OneDrive throttling:

As you see above I generally recommend using tuning parameters at default values unless you have a very good (=tested and verified) reason to tweak them. From what I understand there is no need for these tuning parameters: --transfers, --ignore-existing, --retries and --multithread-cutof

You may also want to check if your speed is limited by the download speed from your Google Drive, or the resources available to your VPS.

Finally, it could be that your have a lot of files and most of them already exist at the destination, then rclone uses most of it’s time on checking – not transferring. This could also result in the picture you are describing and can be seen in your stats where checks are increasing with only few transfers.

2021-11-26 02:28:02 DEBUG : Too many requests. Trying again in 57 seconds.
2021-11-26 02:28:02 DEBUG : pacer: low level retry 1/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2021-11-26 02:28:02 DEBUG : pacer: Rate limited, increasing sleep to 57s
2021-11-26 02:28:02 DEBUG : Too many requests. Trying again in 50 seconds.
2021-11-26 02:28:02 DEBUG : pacer: Reducing sleep to 42.75s
2021-11-26 02:28:02 DEBUG : pacer: low level retry 2/10 (error activityLimitReached: throttledRequest: The request has been throttled)
2021-11-26 02:28:02 DEBUG : pacer: Rate limited, increasing sleep to 50s

:((((((((((((((((((((((((((((((((((((((((((((
one drive has limit for account i guess.

Correct!

The important thing to understand about both OneDrive and Google Drive is that they are priced per size only and therefore both come with limitations to the speed and number of requests you can do within an hour/day (that is fair usage limitations that are enforced by some kind of rate limiting or throttling).

If you want unlimited speed and up/download capacity, then you typically get to pay per request (one way or the other).

You may find this thread/post interesting:

I may be completely wrong, but your large amount of data stored in 100GB files indicate that you also may find this Backblaze blog post interesting:

No not at all. you are not wrong.
i have around 700 TB on google and i just try make some backup.
i guess microsoft not good for this. are you agree with me ?
i sended mail to backblaze about backup. i can pay 70 usd for backup.
do you have another idea about my status?
maybe another licanse on microsoft?

Sorry, I have no knowledge or experience with storage and transfers at this scale. You have roughly 1000 times more data than me.

If I had that much data then I would probably use a more professional storage solution like Amazon S3, Microsoft Azure, Google Cloud Storage, Backblace B2, Wasabi or similar.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.