Slow file transfer speeds with Google Drive

Help and Support
1

OS:
Debian Linux 11

Issue I'm facing:
I currently run a server with 1 Gbps upload to the internet or about 125 MB/s. Using --drive-chunk-size 2G, I can transfer large files up to 80 MB/s but it usually hovers around 60 MB/s. I'm having issues where for smaller files, my upload speed is often sub 10 MB/s. How can I increase this?

To test, I have a folder of 20 duplicate 2.3 MB files so sort of run as a .
The upload reached around 8 MiB/s and the download reached around 22 MiB/s (same folder and files)

server-1:~/rclone-v1.65.1-linux-amd64$ ./rclone copy ~/localdisk/test1upload gdrive1:testupload --transfers 100 --drive-chunk-size 2G --ignore-checksum --ignore-size --ignore-times --no-traverse --progress
Transferred: 46.468 MiB / 46.468 MiB, 100%, 7.742 MiB/s, ETA 0s
Transferred: 22 / 22, 100%
Elapsed time: 9.7s
server-1:~/rclone-v1.65.1-linux-amd64$ ./rclone copy gdrive1:testupload ~ ~/localdisk/testdownload --transfers 100 --drive-chunk-size 2G --ignore-checksum --ignore-size --ignore-times --no-traverse --progress
Transferred: 46.468 MiB / 46.468 MiB, 100%, 22.108 MiB/s, ETA 0s
Transferred: 22 / 22, 100%
Elapsed time: 2.8s

Extra Notes:
I ran a disk speed test, this might give more insight:

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=4k --iodepth=64 --size=250M --readwrite=randrw --rwmixread=80

Output:
Jobs: 1 (f=1): [m(1)][100.0%][r=9844KiB/s,w=2396KiB/s][r=2461,w=599 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=3300: Sat Jan 20 09:26:40 2024
read: IOPS=2666, BW=10.4MiB/s (10.9MB/s)(200MiB/19176msec)
bw ( KiB/s): min= 9312, max=43576, per=100.00%, avg=10689.05, stdev=5481.77, samples=38
iops : min= 2328, max=10894, avg=2672.26, stdev=1370.44, samples=38
write: IOPS=670, BW=2684KiB/s (2748kB/s)(50.3MiB/19176msec); 0 zone resets
bw ( KiB/s): min= 2200, max=11496, per=100.00%, avg=2686.32, stdev=1472.79, samples=38
iops : min= 550, max= 2874, avg=671.58, stdev=368.20, samples=38
cpu : usr=1.93%, sys=6.07%, ctx=47963, majf=0, minf=7
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=51133,12867,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=10.4MiB/s (10.9MB/s), 10.4MiB/s-10.4MiB/s (10.9MB/s-10.9MB/s), io=200MiB (209MB), run=19176-19176msec
WRITE: bw=2684KiB/s (2748kB/s), 2684KiB/s-2684KiB/s (2748kB/s-2748kB/s), io=50.3MiB (52.7MB), run=19176-19176msec

Disk stats (read/write):
sda: ios=51113/12889, merge=0/7, ticks=978235/244741, in_queue=1222976, util=99.61%

2

Not much you can do AFAIK. Gdrive is famous for being very slow for small files.

Also make sure you create your own client_id/secret . It won't improve small files transfers but should make google throttling more bearable.

I do not think --transfers 100 makes any sense - default 4 is probably enough.

3

Changed it down to 30, doesn't make much of a difference.

4

You can't.

Google limits you to 2-3 files per second for file creation so they are particularly bad with small files.

The defaults are generally a good starting spot as well as 30 is a bit much for Google too.

5

Yeah I reduced it down to 10, but reading into it it makes no differerence. This absolutely sucks. The only other way is to use multiple remotes and somehow hook them all together, this situation is worse for rclone mount which I'm trying to use. I'm trying to use ChatGPT to try and write a script or app that uses FUSE to distribute the file transfers and reads between remotes connected to the same folder ID but it might be impossible considering how complex it is.

6

If your requirement is down/uploading constantly a lot of small files then you have to chose the right cloud provider - Google Drive is not the best choice here.

And if you prefer to work with Google ecosystem then I suggest to try Google Cloud Storage.

7

I am such an idiot, everything was infront of my eyes this whole time. Rclone has union remotes...I'm so dumb

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.