Google Drive Throttling

#1

I have Google Drive mounted with rclone, but it seems that Google Drive is throttling me sometimes. Not quite sure what causes it.

Essentially, plex playback is fine for almost all files and can reach 500+ mbps during playback. Rarely, some files will not playback correctly. I will see traffic that starts at 100 mbps that drops down to 10 mbps. I also noticed that copying a file from the rclone mount will only result in 10 mbps transfer every time regardless of the file. Downloading any file directly from the web interface in Chrome also results in a measly 10 mbps for any file, even those shared by other users.

Here’s the strange thing there are three ways to increase the download speed.

  1. If I begin playback of another file in Plex while copying a file from the mount, the copy speed on another computer immediately increases to 350 + mbps. Even stranger, the playback of a file also affects the download speed in Chrome as well. The Chrome download speeds also increase to 30MB/s. Stopping playback in Plex also results in the speeds decreasing again.

  2. If I download any Google Drive file remotely using a VPN (locally hosted), it will increase the download speeds on my local network. Downloading a file from two different PCs locally or two files on the same PC make no difference in download speed.

  3. Using a public VPN resolves download speeds.

Here are my RClone settings:
–allow-other --buffer-size=2048M --drive-chunk-size=256M --vfs-read-chunk-size=256M --vfs-read-chunk-size-limit=0 --transfers=10 --rc --tpslimit=5 --timeout=1h -vv --dir-cache-time=72h --poll-interval=55s --vfs-cache-poll-interval=1m0s --vfs-cache-max-age=73h

I run RClone in docker using this container that I forked.

TLDR: Google Drive is acting weird and can speed up downloads randomly. Playback speeds up file transfers.

0 Likes

#2

Did you create your own client ID/ API key?

https://rclone.org/drive/#making-your-own-client-id

That docker image you have is a bit rough.

2G of buffer means that it will read up 2GB for every file you touch and use 2GB of memory per file. Lots of waste there.
transfers does nothing on a mount so it should be removed
vfs-cache-poll and max age are useless since you aren’t using any vfs-cache mode.

not sure why they changed the poll interval from 1 m to 55 seconds. Seems just odd.

If you have a specific playback problem with debug log to look at, we’re happy to help. Please post those details.

0 Likes

#3

Thanks for the reply. I have read a lot of your posts in the past and based a lot of my settings off your original vfs settings. I noticed that you adjusted them recently but haven’t changed my settings to match because they had been working fine with playback.

Yes. I have my own API key.

I changed the poll interval because I assumed it had to be smaller than the vfs-cache-poll, but if they are doing nothing, I’ll remove it.

I don’t have any playback problems with Plex really. Everything works smoothly for playback. Its just copying (or downloading a file) that I have issues with. Something makes me feel that Plex has some sort of “magic reading” that speeds up everything else…

0 Likes

#4

What do you mean by copying or downloading? What’s the action you are taking to do that?

0 Likes

#5

Another optimization question, if I am not using the mount to upload, is drive-chunk-size useless as well? I have a separate script that runs.

What buffer size is optimal? 500M? The default 16M?’

Also, are the other vfs options like vfs-read-chunk-size making a difference without any vfs-cache-mode?

0 Likes

#6

The mount is shared using samba in docker. I am using a windows device on the network to copy files down. It copies at slow speeds of 1MB/s and speeds increase when I playback a file in Plex.

0 Likes

#7

Also I’ll add that running the optimize files option in Plex results in slow download speeds as well. These speeds also increase when playing back a file. It seems that there is some specific read method that is required to achieve the full download speed?

0 Likes

#8

Yeah, I wouldn’t imagine trying to debug samba in a docker shared to windows as that sounds awful unfortunately.

I’d put rclone on the device using it personally as I keep things as simple as possible. Trying to figure out what Samba/Windows is doing is tough.

Optimal buffer depends on what you are doing really.

If you have long, sequential reads, larger buffer would make sense. The downside is that once a file is closed, the buffer drops.

That only options related directly to vfs cache mode being anything but ‘off’ are located here:

https://rclone.org/commands/rclone_mount/#file-caching

as they all relate back to file caching.

Think of vfs-read-chunk-size not as a buffer, but how it requests some the provider. It asks for ‘chunks’ of that and you still download those chunks. I leave all of that as the defaults.

That’s these two:

 --vfs-read-chunk-size SizeSuffix         Read the source objects in chunks. (default 128M)
 --vfs-read-chunk-size-limit SizeSuffix   If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)

0 Likes

#9

Hmm thanks. I don’t believe I have to debug samba thankfully though. It has something to do with Google Drive and how RClone requests a file? Running a cp from the mount to another directory results in the same problem.

Its baffling that the downloads from Google Drive in Chrome have the same problems which can be “resolved” by playing a file in Plex?? I really just want to get to the bottom of why any other type of file transfer beside a Plex playback is slow from both the mount and Chrome.

By buffer drops, do you mean it releases the memory? I have 32GB of memory allocated for RClone so I wasn’t terribly worried about memory consumption as long as it improves performance. If its not doing anything performance wise, I would want to remove it though.

0 Likes

#10

If you can reproduce it, put the mount in debug log mode with -vv and recreate the issue and share the log.

Yes, it releases memory but also you consume bandwidth to download it and it’s somewhat of a waste unless you plan to use it.

0 Likes

#11

I created a log that includes annotated pictures of the file transfers.

The log shows:

  1. Played back readfile.mp4 in plex which resulted in high download speed 500 mbps. Scrubbed to different location. Closed file.
  2. Begin copying the copyfile.mp4 at a slow 2MB/s. Start playback in Plex which speeds up the copy on a different machine. Playback eventually finishes buffering to device and stops requesting data. Copy slows down again. Scrub to new location in Plex to force it to download again. Copy speeds up again.

I also included screenshots of the file transfer and the traffic from #2.

Rclone settings were:
–allow-other
–buffer-size=2048M
–drive-chunk-size=256M
–rc
–tpslimit=5
–timeout=1h
-vv
–dir-cache-time=72h
–log-file=/log/read.txt

0 Likes

#12

How are you copying the file? I used your same settings and I’m pushing my gigabit pretty much:

Your settings are a bit off as you have a lot of duplicate things in there if you check the first line of your debug log:

2019/04/13 16:35:34 DEBUG : rclone: Version "v1.46" starting with parameters ["rclone" "mount" "--config=/config/rclone.conf" "--buffer-size" "512M" "--checkers" "16" "--allow-non-empty" "--allow-other" "--uid" "1100" "--gid" "1100" "--umask" "000" "--read-only" "--allow-other" "--buffer-size=2048M" "--drive-chunk-size=256M" "--rc" "--tpslimit=5" "--timeout=1h" "-vv" "--log-file=/log/read.txt" "--dir-cache-time=72h" "direct-decrypt:" "/read-decrypt"]

You have buffer size twice.
checkers does nothing on a mount so can be removed.
Why would you tpslimit it? not sure why that is there.
allow-non-emtpy is just an awful parameter as it allows over mounting and masks things. Just not a good one to ever use imo.

Can you just run the copy by itself and stop it after a few and let’s look at that.

0 Likes

#13

Thanks for catching the duplicates. I removed them, but it made no difference with copying.

In the first log, I used windows to copy from the samba share. This time, I used the same rsync command you used in your screenshot from the machine its mounted on with the same miserable speeds.

The tpslimit was because I was getting API errors in the console when Plex did scans.

Heres the log.

0 Likes

#14

Scans shouldn’t produce any API issues. Was that before or after using your own key? I tested many times and scanned 30-40TB in a few days without any API issues.

0 Likes

#15

The things I looked at:

  • I don’t see any of the reads going out of order. All the offsets look to be sequential
  • The chunksize and length doubles up and moves as I’d expect. Same as my logs when I copy minus a little slower.

You are copying directly from the mount and nothing else like unionfs or anything like that right?

What’s the OS?

0 Likes

#16

I had used rclone since before there was a built in key so I always had my own key. I think it might have been the plex scheduled tasks that happen at night. I would check the api console and there would be errors. The tpslimit got rid of them. Additionally, there were errors that would appear infrequently until I added the tpslimit. Regardless though, removing the tpslimit resulted in the same speeds.

0 Likes

#17

The OS is rancheros. In the logs that I have been running, they have been going through mergerfs. But when you mentioned it, I just tried running it without mergerfs directly but had the same results.

At this point, it really does seem like Google is throttling me, but I am not sure why the speeds would increase when playing back media…

0 Likes

#18

Also, I would like to add that downloading a file in Chrome has the same effect. And a public VPN will fix the issue when I download a file from Chrome.

0 Likes

#19

I’m quite sure google isn’t aware you or exist in terms of our little consumption :slight_smile:

You got a lot of layers going on with dockers/mergerfs and quite some things so it’s more likely it’s something along those lines causing an issue.

If you are using mergerfs, you want to make sure you are using sync_read, but I didn’t see anything out of order in your logs.

Also, downloading via the web uses a completely different way to get files as most people generally report that being fast and rclone being slow as rclone uses drive API and the download does not.

If both are slow, I’d check router/ISP/machine/bad cable/etc.

1 Like

#20

Just to confirm, I am using mergerfs with sync_read.

The reason I bring up the downloads with Chrome is because the plex playback with Rclone also affects the download speeds through Chrome. I get significantly faster speeds when playing back media in Plex. The slow speeds ~1MB/s I get in Chrome also match up to the same slow speeds ~1MB/s I get in Rclone when copying…

I feel like I would have blamed the router or ISP quickly if the speeds were always the same. It just really doesn’t make sense that this is happening.

I also have a ubuntu server machine that I originally had rclone using cache with these settings:
/usr/bin/rclone mount plex: /media/plex
–allow-non-empty
–allow-other
–local-no-check-updated
–buffer-size=0
–cache-chunk-path="/media/cache/chunkcache"
–cache-chunk-size=8M
–cache-db-path="/media/cache/cache.db"
–cache-dir="/media/upload/vfscache"
–cache-info-age=28h
–cache-read-retries=3
–cache-chunk-total-size=10G
–cache-tmp-upload-path="/media/upload/pending"
–cache-tmp-wait-time=“2h”
–cache-workers=5
–cache-writes
–config="/home/rclone/.config/rclone/rclone.conf"
–contimeout=5s
–dir-cache-time=24h
–gid 0
–max-read-ahead=128k
–transfers=3
–tpslimit=3
–timeout=5s
–track-renames
–uid 0
–umask 000
–vfs-cache-mode writes
-vv

When I tried running rsync, I got slightly better speeds, between 3.5 and 5 MB/s without playing any media. Still slow but better?

0 Likes