[Google Drive] Different transfer speeds for different file types

Help and Support
1

What is the problem you are having with rclone?

I'm having issues with Google Drive to Google Drive transfer speeds for specific file types. Certain file types are maxing out at over 30M/s whereas others are barely hitting 4M/s. Specifically this slowdown seems to affect archive files such as ISOs and gzips.

I was wondering if this is a known example of rate limiting or whether there is some setting I can implement to avoid this.

What is your rclone version (output from rclone version)

rclone v1.51.0

  • os/arch: linux/amd64
  • go version: go1.13.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 20.04 LTS (GNU/Linux 5.4.0-1009-gcp x86_64)

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync gd: td: --fast-list -vP --tpslimit-burst=50 --stats=10s --max-backlog=2000000 --ignore-case --no-update-modtime --drive-chunk-size=128M --drive-use-trash=false --track-renames --drive-stop-on-upload-limit --transfers=16

The rclone config contents with secrets removed.

[gd]
type = drive
client_id = xxxxx.apps.googleusercontent.com
client_secret = xxxxx
scope = drive
token = {"access_token":"xxxx","token_type":"Bearer","refresh_token":"xxxx","expiry":"2020-05-16T16:46:26.143807937Z"}
root_folder_id = xxxxx


[td]
type = drive
client_id = xxxxx.apps.googleusercontent.com
client_secret = xxxxx
scope = drive
token = {"access_token":"xxxx","token_type":"Bearer","refresh_token":"xxxx","expiry":"2020-05-17T07:39:06.378824934Z"}
team_drive = xxx

A log from the command with the -vv flag

Transferred:        243.906G / 78.132 TBytes, 0%, 81.018 MBytes/s, ETA 1w4d16h2m18
Checks:             16358 / 16358, 100%
Transferred:          103 / 26237, 0%
Elapsed time:     51m22.7s
Transferring:
 * xxxxxxx.mp4: 58% /2.265G, 32.266M/s, 29s
 * xxxxxxx.mp4: 73% /1.259G, 25.726M/s, 13s
 * xxxxxxx.mp4: 74% /989.185M, 22.949M/s, 11s
 * xxxxxxx.mp4: 74% /1021.803M, 25.669M/s, 9s
 * xxxxxxx.mp4: 56% /1.079G, 36.575M/s, 13s
 * xxxxxxx.mp4: 40% /1.232G, 41.783M/s, 17s
 * xxxxxxx.mp4: 80% /2.988G, 32.001M/s, 18s
 * xxxxxxx.ISO: 26% /4.019G, 3.864M/s, 12m58
 * xxxxxxx.ISO: 26% /4.096G, 3.938M/s, 13m1s
 * xxxxxxx.ISO: 29% /4.194G, 3.933M/s, 12m45
 * xxxxxxx.ISO: 25% /4.189G, 3.957M/s, 13m25
 * xxxxxxx.ISO: 26% /4.119G, 4.010M/s, 12m52
 * xxxxxxx.ISO: 27% /4.020G, 3.925M/s, 12m41
 * xxxxxxx.ISO: 24% /4.313G, 3.891M/s, 14m11
 * xxxxxxx.ISO: 40% /3.952G, 3.761M/s, 10m42
 * xxxxxxx.ISO: 35% /3.767G, 3.898M/s, 10m39
2

No, there aren't.

Try testing with the one file that you are having a problem with.

You also didn't include a debug log and just a clip of a transfer so it's hard to tell from the log.

Google limits to 10 transactions per second so this

Might be slowing you down as well.

You can increase this even more if you have the memory.

You can reduce this as well and just use the default as that will probably help as well.

3

Hi, thanks for answering!

I implemented the changes you've recommended. No luck unfortunately. I lowered --transfers all the way down to 1 and prepared a debug log (which I've pasted here). The single file transfers at a rate of approx. 25 MBytes/s but this offers no improvement for the overall transfer speed.

Just to clarify, I'm not having the issue with only one file. It extends to all files of the ISO file type. I posted the snapshot to demonstrate the different speeds I am getting with the different file types.

Thanks again for the assistance!

4

There is no throttling on ISOs or any file type.

felix@gemini:/data$ rclone copy ubuntu-20.04-live-server-amd64.iso GD: --drive-chunk-size=256M -P
Transferred:   	      908M / 908 MBytes, 100%, 63.513 MBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:        14.2s

You can test with a single copy and see what works best for you. You'd want to ensure that you have no bottlenecks in term of CPU/Disk/Memory/Router. Are you using a VPS?

5

I am using a VPS but it doesn't seem to be a bottleneck from what I can tell.

The main confusion for me is why I am seeing this only for certain file types. When I am transferring other large files (MP4, MKV, TS etc.) I often see maximum transfer rates exceeding 220 MBytes/s.

6

That's the eternal struggle of correlation and causation though :slight_smile:

I'm not sure what the VPS is rated but getting 220MB/s is well over a gigabit connection so that seems solid.

The challenge with the VPS is you can't see the full picture, if you can produce consistent results with a lot of transfers, try to roll it back to one and work with the VPS provider to see what the bottleneck is.

I have good networking peering to my Google Drive endpoint API and I can control everything on my server and my outbound bandwidth so I can eliminate everything minus my network peering to my drive end point.

7

Yeah you're not wrong. I'll do some more poking around with my VPS provider.

Thanks for your help!

8

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.