Slow download from gdrive to local


#1

Previously I could download 1 single file at 900Mbps maxing out my gigabit connection.

But now, its 150-200Mbps. Why is this so?

Bear in mind its large single files at 50-100gb. Downloading at this speed is super slow.

Any commands I could try fixing it, seems weird didn’t have this issue last month.


Sudden Slow Download from Google Drive?
#2

Fixed it by changing to 1.36. Did you do something to the latest build? @ncw


#3

V1.41
2018/05/29 15:03:23 INFO :
Transferred: 3.538 GBytes (19.713 MBytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 3m3.7s

V1.36
2018/05/29 15:12:32 INFO :
Transferred: 5.732 GBytes (91.936 MBytes/s)
Errors: 0
Checks: 1
Transferred: 0
Elapsed time: 1m3.8s

Same exact file and remote. And checkout timestamp. Minutes after i changed it, its back to fast speeds.


#4

I managed to replicate this with BAD being about 20 MB/s and good being about 65 MB/s

v1.41 BAD
v1.40 BAD
v1.39 GOOD
v1.36 GOOD

So it happened somewhere in the 1.40 development cycle.

I ran git bisect and it came up with this commit as being the problem one (as I expected!)

07f20dd1fd6f89b9ebdefdc8964a8e9ee9aa9892 is the first bad commit
commit 07f20dd1fd6f89b9ebdefdc8964a8e9ee9aa9892
Author: Fabian Möller <fabianm88@gmail.com>
Date:   Wed Jan 24 00:46:41 2018 +0100

    drive: migrate to api v3

:040000 040000 aad5e99c16dab297c5f79c505bb4d303d2497953 35a6b83de5afb909d3f83c9dfc29b9c2d7c20247 M	backend

The difference appears to be that for v2 vs v3 is that we are downloading files from different endpoints:

FAST 1.39
Host: doc-14-bc-docs.googleusercontent.com
/docs/securesc/XXX/YYY/ZZZ/AAA/PPP?e=download&gd=true

vs

SLOW 1.41
Host: www.googleapis.com
/drive/v3/files/XXX?alt=media HTTP/1.1

rclone is downloading files the recommended way with the v2 API as far as I can see.

@B4dM4n - any ideas why the v3 API should be slower for downloading files?


#5

This is odd, I don’t see a reason why v3 should be slower than v2.

But I can reproduce this behavior. Depending on the device used, there seem to be a limit of 25 to 30 MB/s.

There doesn’t seem to be report of this issue in the Google tracker. Should we open one?
In the meantime I can look into creating a workaround flag like --drive-use-v2-download to use the old links.

I implemented a workaround in v1.41-075-ga193ccdb-drive_v2_download. It adds a flag --drive-v2-download-min-size to specify the minimum size, at which drive v2 download links are used. This adds the overhead of a drive.files.get request for each download to obtain the downloadUrl.

This 25 to 30 MB/s limit seems to also affect uploads for me.


#6

Uploads has always been limited since day 1. Found it weird how only download has full speed, but didn’t thought of it at the time as i do batch uploads but single downloads.

Also, is there a reason to use v3? Seems like v2 is superior, why not use that instead.


#7

From what I can see, I don’t believe this is due to a difference in the API versions.

Download Docs for v2: https://developers.google.com/drive/api/v2/manage-downloads
Download Docs for v3: https://developers.google.com/drive/api/v3/manage-downloads

Both recommend using the alt=media approach. The difference seems to be from the different urls for downloading: the downloadUrl from the file resource and the official method recommended by google.


#8

No, me neither… I think it must be some policy on the endpoints.

That is probably a good idea - want to do the honours?

Nice one :smile:

Vaguely related: this google issue which is related to #2243 and the recommendation to use the v2 API export endpoint there too!


#9

Hi,

I have the same problem. but not with gdrive my problem child is with dropbox all versions above 1.38 are slow to download someone can replicate that?

rclone v1.38-236-g29d34426β

  • os/arch: linux/amd64
  • go version: go1.9.2

#10

I suggest you try a newer rclone v1.41 or the latest beta and if you still have the problem then make a new topic with some more info - thanks


#11

Thank you for the v2 flag! I knew something slowed down but never identified why.


#12

Would it be possible for this branch to be merged into the main branch? Or possibly a flag added --use-drivev2-api or something like that to force it to use the v2 api? It would be nice for uploads to go through the v2 API as well since they are locked at 30 MB/s max on the current rclone version.


#13

just to second that, a flag added --use-drivev2-api would be a nice addition.

And just tried that beta on my dedi and the speeds were awesome…


#14

Any update on whether this flag is going to be added? Seems rclone isn’t pulling files (especially 4K ones) fast enough for them to stream without constant buffering.

I’d like to try the v2 flag and see if this helps


#15

What’s the file you are trying to play? I’ve been playing 4k’s without a problem.


#16

Hmm. OK, maybe it’s not that then. It happens for most, one example was: Red.Sparrow.2018.UHD.BluRay.2160p.TrueHD.Atmos.7.1.HEVC.REMUX-FraMeSToR

to my Sony 4K TV using the native Android Plex app. I have a 76Mb download connection (and do get that full amount)

My server is a Hetzner EX41SSD:

Intel® Core™ i7-6700 Quad-Core
incl. Hyper-Threading Technology
RAM:	32 GB DDR4
Hard Drive:	2 x 500 GB SATA 6 Gb/s SSD 
(Software-RAID 1)
Connection:	1 GBit/s-Port
Guaranteed bandwidth:	1 GBit/s

And my rclone service is set up as a standard crypt:

[Unit]
Description=rclone FUSE mount
Documentation=http://rclone.org/docs/
After=network-online.target

[Service]
Type=simple
User=dan
Group=dan
ExecStart=/usr/bin/rclone mount crypt: /mnt/plex \
   --allow-non-empty \
   --allow-other \
   --dir-cache-time 48h \
   --vfs-cache-max-age 48h \
   --vfs-read-chunk-size 10M \
   --vfs-read-chunk-size-limit 100M \
   --buffer-size 1G \
   --umask 002 \
   --rc \
   -v \
   --log-file=/home/dan/scripts/logs/rclonecrypt.log
ExecStop=/bin/fusermount -uz /mnt/plex/
Restart=always

[Install]
WantedBy=multi-user.target

I may have to look deeper… possibly into the plex logs to see if that can shed some light then.


#17

If that’s not streaming, I’d wonder if you are having peering issues or something goofy.

My mount is just about identical and can stream that movie as I am direct playinng via Infuse on an ATV.


#18

I’m using cloudflare to try and help with peering. Even still, I’m in the UK and the server is in Germany… So not too distant.

Maybe I’ll try another player. May have to get a 4k atv or chromecast


#19

I get google drive issues every single night is, between 7PM PST and 10PM PST. My max download is about 15-16MB/sec which would basically make those large files NOT playable. Buffering etc. It will even buffer on smaller files too. I kept thinking it was my home server. But I just did a simple COPY from the drive to my desktop (hard wired at 300mbit) and i get 15-16mb/sec or roughly 100-150mbit. Which is JUST under what is needed to stream these larger bit rate files. Marked as 1.6hours to transfer the movie file, with the movie being just over that. so Yea, especially since this speed is not constant.

I will test again in the AM to see if my speeds improve. I am wondering if it is like the other thread of the api v2 vs api v3 being used. Someone pointed out that their downloads used to be MAX gigabit speeds, but since upgrading to new rclone they are maxed at 150-200mbit downloads AND uploads.


#20

If you guys have ipv6 configured on your machines, I’d recommend forcing rclone to bind to your ipv4 address instead and having it use that instead for connections. The difference in speed between the two is significant. You can do this by adding --bind I.P.v.4Here to your mount command.