Upload 40mb, rclone transfer to gdrive 30kb?

Help and Support
1

I’m trying to upload about 15TB of data to my gsuite google drive account with “unlimited storage”.

I’m starting the transfer in chunks, last night I ran

rclone --verbose move localpath remotepath

To move 40GB from an external HDD to Google Drive and it took about 11hrs 30mins. Even though my internet connection is about 30-40mb upload on average.

I thought it might be because it came from an external drive, so today I transferred 25GB to the C drive before running

rclone -vv -P moveto localpath remotepath

Now I see the transfer rate is a mere 30-40KB on average. Doing a speed test my internet still gets about 30-40MB upload speed.

rclone is debugging

pacer: Rate limtited, sleeping for 4s
pacer: low level retry 3/10 (error googleapi: Error 403 Rate Limit Exceded, rateLimitExceeded)

Looking into it google has a 750GB limit per day for file uploads in GSuite. However 40GB and 25GB is hardly 7050GB so what is the limiting factor here?

2

Are you using your own API key? You need to reduce the numbers of transfers and checkers as you are getting rate limited as the defaults are a bit much.

1 Like
3

Thanks @Animosity022

Apologies as I am very new to rclone. I used the most basic setup for google drive as per documentation. I setup the remote with all the defaults and did “Use auto config? yes” where it loads the google webpage to select the account for sign in.

With those defaults in mind, could you give me an example of the command to lower the amount of transfers and checkers? Really appreciate it!

4

I slim my down to 3 / 3. Here is my upload script I use:

/usr/bin/rclone move /data/local/ gcrypt: -P --checkers 3 --log-file /home/felix/logs/upload.log -v --tpslimit 3 --transfers 3 --drive-chunk-size 32M --exclude-from /home/felix/scripts/excludes --delete-empty-src-dirs

You can probably go a little higher but it’s a trial/error to see. You can only every do 10 transactions per second.

1 Like
5

Do also set up your own API key as @Animosity022 suggested - that will make a lot of difference.

We should probably change the docs to recommend this more strongly - what do you think @Animosity022 ?

1 Like
6

That would make sense. I’ll see if I have some time this morning and try to get a pull request.

1 Like