Help maxing out my connection when transferring a single file to Gdrive

Help and Support
1

Not sure what I am doing wrong, but anything that I try from reading around here is not working. I’m trying to max out my speed on a single file upload to Google Drive. Here’s a speedtest result of what I get on the server I’m transferring from.

Here are the results I am getting with various “drive-chunk-size” settings.

64M
2018/11/11 15:47:46 DEBUG : iso.txt: Sending chunk 4898947072 length 33554432
2018/11/11 15:47:47 DEBUG : iso.txt: Sending chunk 4932501504 length 33554432
2018/11/11 15:47:47 INFO :
Transferred: 4.608G / 54.164 GBytes, 9%, 37.240 MBytes/s, ETA 22m42s
Errors: 0
Checks: 1 / 1, 100%
Transferred: 0 / 7, 0%
Elapsed time: 2m6.7s
Transferring:

  • iso.txt:  8% /54.135G, 42.703M/s, 19m47s

128M
2018/11/11 15:42:41 INFO :
Transferred: 4.457G / 54.164 GBytes, 8%, 37.304 MBytes/s, ETA 22m44s
Errors: 0
Checks: 1 / 1, 100%
Transferred: 0 / 7, 0%
Elapsed time: 2m2.3s
Transferring:

  • iso.txt:  8% /54.135G, 45.381M/s, 18m40s

256M
2018/11/11 15:40:02 INFO :
Transferred: 3.656G / 54.164 GBytes, 7%, 30.071 MBytes/s, ETA 28m39s
Errors: 0
Checks: 1 / 1, 100%
Transferred: 0 / 7, 0%
Elapsed time: 2m4.5s
Transferring:

  • iso.txt:  6% /54.135G, 45.849M/s, 18m47s

512M
2018/11/11 15:37:39 INFO :
Transferred: 5.366G / 54.164 GBytes, 10%, 43.912 MBytes/s, ETA 18m57s
Errors: 0
Checks: 1 / 1, 100%
Transferred: 0 / 7, 0%
Elapsed time: 2m5.1s
Transferring:

  • iso.txt:  9% /54.135G, 45.204M/s, 18m24s

1024M
2018/11/11 15:35:09 INFO :
Transferred: 5.449G / 54.164 GBytes, 10%, 44.545 MBytes/s, ETA 18m39s
Errors: 0
Checks: 1 / 1, 100%
Transferred: 0 / 7, 0%
Elapsed time: 2m5.2s
Transferring:

  • iso.txt: 10% /54.135G, 43.787M/s, 18m58s

I’m using: -vv --stats=1s --drive-chunk-size 128M --transfers 1"
with the latest version of Rclone installed.
Any help or suggestions would be appreciated!

Thanks

2

Your speed test shows upload of 951 Mbit/s which is approx 118 Mbyte/s.

You are acheiving 44 MByte/s with rclone which seems pretty fast to me! I’ve never managed to get drive to work faster than about 20 MByte/s.

I suspect this is a network limitation somewhere between you and drive, but that kind of thing is quite hard to prove.

3

I’ve seen that the v3 vs v2 api is usually the bit that slows it down.

I find that via rclone copy/sync/move, I can only get 20-30MB going but with the vfs mount as it uses the chunks, it maxes out my connection.

rclone copy:

 *                             Unsane (2018).mkv:  1% /24.157G, 21.098M/s, 19m11s

rsync with progress:

felix@gemini:/data$ rsync --progress /GD/Radarr_Movies/Unsane\ \(2018\)/Unsane\ \(2018\).mkv .
Unsane (2018).mkv
    858,292,224   3%   93.49MB/s    0:04
4

That is not what I’d expect! What are the mount parameters you are using? I’ll see if I can replicate.

copy/sync/move should be faster than mount as it is doing less work.

5

I just use the vfs with majority of the defaults:

felix      595     1  0 10:07 ?        00:00:54 /usr/bin/rclone mount gcrypt: /GD --allow-other --bind 192.168.1.30 --dir-cache-time 72h --drive-chunk-size 32M --log-level INFO --log-file /home/felix/logs/rclone.log --umask 002 --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off --rc
6

I did a quick test to see if I could replicate

$ rclone -P copyto 1G TestDrive:1G.1
Transferred:            1G / 1 GBytes, 100%, 25.799 MBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            1 / 1, 100%
Elapsed time:       39.6s

vs

rclone mount TestDrive: /mnt/tmp/ --drive-chunk-size 32M --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off

$ rsync --progress 1G /mnt/tmp/1G.3
1G
  1,073,741,824 100%   30.08MB/s    0:00:34 (xfr#1, to-chk=0/1)

However if you set --drive-chunk-size in the copy then it goes even faster

$ rclone -P copyto 1G TestDrive:1G.2 --drive-chunk-size 32M
Transferred:            1G / 1 GBytes, 100%, 40.214 MBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            1 / 1, 100%
Elapsed time:       25.4s

BTW you can set “chunk_size” in the drive config now-a-days