Rclone downloadspeed inconsistent for Google Drive compared to web

Help and Support
1

I’ve had this problem for several months now.
For my home internet connection 400/50mbit upload works as expected but download jumps up and down like crazy with rclone. Now interestingly the official Google Drive Stream app jumps up and down too for download, similar to rclone. But if I try to download the same file in the Drive web interface I saturate my connection everytime.

To let you get an idea about how this looks here are several screencaps from Windows Resource Monitor while copying the same 10gb file:

rclone_speed
rclone_speed.png630×1130 25.5 KB

This behaviour of rclone doesn’t change neither when I change chunk sizes nor when I use the copy command instead of a mount.

Already tried different Computers at my location. tried running rclone in a linux vm. Always the same problem.

Have a remote dedicated server where I have no problems with my rclone mount.

My guess is that it could have something to do with my ISP or my router (that is owned by my isp)? But why would Google Drive Web work then without problems?

As a test, when I routed all my internet traffic through my remote server over VPN I had a steady 20MByte transfer rate with rclone and drive file stream.

Right now I mount the mounted gdrive from my server with winsshfs. And this works good enough for 4k streaming. But thats just a not so pretty workaround.

2

rclone version shows what?

did you make your own cilent ID/API key?

https://rclone.org/drive/#making-your-own-client-id

What command are you running?

Can you run the command share the log with -vv on it?

3

Currently I use rclone version 1.46.
But I already tried like 10 different versions between 1.36 and 1.46 with no noticable difference.

Yes I use my own client key. In fact I made several to test if it makes a difference. And as expected it didn’t.

The command I used in this example was:
rclone mount gdrive: y: --log-file “D:\appz\rclone\rclone_mount2.log” -vv --read-only --vfs-read-chunk-size=32M --vfs-read-chunk-size-limit=2G --tpslimit 10 --buffer-size=512M --config “C:\Users\4x111.config\rclone\rclone.conf”

But as I said I tried multiple variations of commands with and without mounting and it’s the same problem every time.

Log:
https://drive.google.com/file/d/14QMmlAq6q_aaFjGHRbCkBuypwihqwfFa/view?usp=sharing

4

Are you just running a copy command from the mount?

5

Yes I copied the file from gdrive to my local ssd via the windows file explorer.

And when I download some file with “rclone copy” it behaves the same.

6

If the goal is to copy, probably easier to just use rclone copy and run that command with -vv and let’s see what it does if you can share the log.

7

OK did a
rclone copy -vv gdrive:temp/radarr.tar C:\temp
and speeds were even worse.
rclone_copy

Log:

2019/03/09 15:54:11 DEBUG : rclone: Version “v1.46” starting with parameters [“rclone” “copy” “-vv” “–log-file” “D:\appz\rclone\rclone3.log” “gdrive:temp/radarr.tar” “C:\temp\”]
2019/03/09 15:54:11 DEBUG : Using config file from “C:\Users\4x111\.config\rclone\rclone.conf”
2019/03/09 15:54:11 DEBUG : gdrive: Loaded invalid token from config file - ignoring
2019/03/09 15:54:11 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2019/03/09 15:54:11 DEBUG : gdrive: Saved new token in config file
2019/03/09 15:54:13 DEBUG : radarr.tar: Couldn’t find file - need to transfer
2019/03/09 15:55:13 INFO :
Transferred: 403.996M / 7.369 GBytes, 5%, 6.494 MBytes/s, ETA 18m19s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 1, 0%
Elapsed time: 1m2.2s
Transferring:

  •                                radarr.tar:  5% /7.369G, 8.273M/s, 14m23s

2019/03/09 15:56:13 INFO :
Transferred: 685.996M / 7.369 GBytes, 9%, 5.614 MBytes/s, ETA 20m22s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 1, 0%
Elapsed time: 2m2.1s
Transferring:

  •                                radarr.tar:  9% /7.369G, 2.891M/s, 39m33s

2019/03/09 15:57:13 INFO :
Transferred: 1013.996M / 7.369 GBytes, 13%, 5.565 MBytes/s, ETA 19m33s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 1, 0%
Elapsed time: 3m2.1s
Transferring:

  •                                radarr.tar: 13% /7.369G, 5.439M/s, 20m0s

2019/03/09 15:58:13 INFO :
Transferred: 1.172G / 7.369 GBytes, 16%, 4.955 MBytes/s, ETA 21m20s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 1, 0%
Elapsed time: 4m2.1s
Transferring:

  •                                radarr.tar: 15% /7.369G, 3.057M/s, 34m35s

2019/03/09 15:59:13 INFO :
Transferred: 1.473G / 7.369 GBytes, 20%, 4.990 MBytes/s, ETA 20m10s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 1, 0%
Elapsed time: 5m2.1s
Transferring:

  •                                radarr.tar: 19% /7.369G, 6.407M/s, 15m42s

2019/03/09 16:00:13 INFO :
Transferred: 1.797G / 7.369 GBytes, 24%, 5.080 MBytes/s, ETA 18m43s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 1, 0%
Elapsed time: 6m2.1s
Transferring:

  •                                radarr.tar: 24% /7.369G, 4.900M/s, 19m24s

2019/03/09 16:01:13 INFO :
Transferred: 2.004G / 7.369 GBytes, 27%, 4.860 MBytes/s, ETA 18m50s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 1, 0%
Elapsed time: 7m2.1s
Transferring:

  •                                radarr.tar: 27% /7.369G, 3.732M/s, 24m32s
8

You should increase the drive chunk size:

/usr/bin/rclone move /data/local/ gcrypt: -P --checkers 3 --log-file /home/felix/logs/upload.log -v --tpslimit 3 --transfers 3 --drive-chunk-size 32M --exclude-from /home/felix/scripts/excludes --delete-empty-src-dirs

Is what I use. 32M or 64M is a sweet based on quite a bit of testing.

There is still a bit of cap on the v3 API that I hit when doing a single transfer:

2019-03-09 10:23:03 DEBUG : n0t3rf73m837rcfpfuefg9fs3o: Sending chunk 1073741824 length 33554432
Transferred:   	    1.031G / 3.316 GBytes, 31%, 36.820 MBytes/s, ETA 1m3s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 1, 0%
Elapsed time:       28.6s
Transferring:
 *                                      test.mkv: 31% /3.316G, 38.256M/s, 1m1s^C
[felix@gemini data]$ rclone copy -vv -P test.mkv gcrypt: --drive-chunk-size 32M
9

rclone copy -vv --drive-chunk-size 32M --log-file “D:\appz\rclone\rclone4.log” gdrive:temp/radarr.tar C:\temp\

same problem:
rclone_copy2

Log:

2019/03/09 17:02:13 DEBUG : rclone: Version “v1.46” starting with parameters [“rclone” “copy” “-vv” “–drive-chunk-size” “32M” “–log-file” “D:\appz\rclone\rclone4.log” “gdrive:temp/radarr.tar” “C:\temp\”]
2019/03/09 17:02:13 DEBUG : Using config file from “C:\Users\4x111\.config\rclone\rclone.conf”
2019/03/09 17:02:13 DEBUG : gdrive: Loaded invalid token from config file - ignoring
2019/03/09 17:02:13 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2019/03/09 17:02:13 DEBUG : gdrive: Saved new token in config file
2019/03/09 17:02:14 DEBUG : radarr.tar: Couldn’t find file - need to transfer
2019/03/09 17:03:14 INFO :
Transferred: 325.996M / 7.369 GBytes, 4%, 5.285 MBytes/s, ETA 22m46s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 1, 0%
Elapsed time: 1m1.6s
Transferring:

  •                                radarr.tar:  4% /7.369G, 3.365M/s, 35m45s

2019/03/09 17:04:14 INFO :
Transferred: 610.996M / 7.369 GBytes, 8%, 5.021 MBytes/s, ETA 23m1s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 1, 0%
Elapsed time: 2m1.6s
Transferring:

  •                                radarr.tar:  8% /7.369G, 3.953M/s, 29m14s

2019/03/09 17:05:14 INFO :
Transferred: 877.996M / 7.369 GBytes, 12%, 4.832 MBytes/s, ETA 22m59s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 1, 0%
Elapsed time: 3m1.6s
Transferring:

  •                                radarr.tar: 11% /7.369G, 4.599M/s, 24m9s

2019/03/09 17:06:14 INFO :
Transferred: 1.108G / 7.369 GBytes, 15%, 4.696 MBytes/s, ETA 22m45s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 1, 0%
Elapsed time: 4m1.6s
Transferring:

  •                                radarr.tar: 15% /7.369G, 3.096M/s, 34m30s

2019/03/09 17:07:14 INFO :
Transferred: 1.297G / 7.369 GBytes, 18%, 4.402 MBytes/s, ETA 23m32s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 1, 0%
Elapsed time: 5m1.6s
Transferring:

  •                                radarr.tar: 17% /7.369G, 2.917M/s, 35m31s
10

Oh I misread, you are ‘downloading’ not uploading. Drive-chunk-size does nothing for a download as that’s only for uploading a file.

I wonder if it’s more Windows related as I’m running a simple command on Linux:

rclone copy gcrypt:1G.file /data -P
Transferred:   	     1000M / 1000 MBytes, 100%, 22.651 MBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -

Transferred:            1 / 1, 100%
Elapsed time:       44.1s

Which just copies a 1G file back down for me. Since I was only moving 1 file, I left all the parameters default as they don’t matter much with 1 file.

Is there anything else in the middle that might be slowing things down? Wireless or wired? Able to check your router for other traffic going on?

11

Have wired lan at this pc. No other traffic going on. For every other download be it http, ftp, usenet, speedtest.net, etc… I get fullspeed which is ~400mbit for me. Only rclone and the google file stream app are slow. And as I said when I download a file via web interface from google drive I get ~400mbit too.

It almost seems like traffic is throttled for gdrive api for me?

12

That would be super strange if it was.

Maybe try to spin up a Linux VM and test there?

13

As I explained in my first post I already tried that and it was exactly the same. :slight_smile:

14

Is it up now? I have a few things you can try.

15

Can boot it up right now.
What can i try?

16

Just run a normal mount with something like:

/usr/bin/rclone mount gcrypt: /GD --allow-other --dir-cache-time 96h

and see what you get if you copy from the mount to a local drive.

rsync -P /GD/1G.file /data
1G.file
  1,048,576,000 100%   55.92MB/s    0:00:17 (xfr#1, to-chk=0/1)

and on the Linux side, make sure it’s the latest 1.46.

17

mounted with:

rclone mount gdrive: /mnt/gdrive --allow-other --dir-cache-time 96h

And tried to copy:

rsync -P /mnt/gdrive/temp/radarr.tar /home
radarr.tar
358,383,616 4% 5.05MB/s 0:24:22

same problem
screencap from traffic of vm:
rclone_linux_mount

18

If the rclone is only consuming 5 MB/s or 40 Mb/s, what else is running that is causing it to peak over 100Mb/s for the graph?

19

That’s only rclone and nothing else.
It was not at a steady 5MB/s. While transfering it jumped up and down beween 1MB/s and 25MB/s. That’s why I included the graph. That you can see the fluctations in downloadspeed over time of rclone.

20

Yeah, not sure what else as we’ve ruled out everything rclone related as you’ve tried a different OS.

Either you got a bad host, bad route/peering to the endpoint you are connecting to or your ISP is doing something.