Google Drive Throttling

That’s the normal polling message you quoted there.

You can either put the mount in debug mode with -vv or you can run something like

rclone ls -vv direct-decrypt:

I just copied a simple file to create the error:

[felix@gemini ~]$ rclone lsf gcrypt: -vv
2019/04/14 23:39:36 DEBUG : rclone: Version "v1.47.0" starting with parameters ["rclone" "lsf" "gcrypt:" "-vv"]
2019/04/14 23:39:36 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2019/04/14 23:39:37 DEBUG : hosts: Skipping undecryptable file name: not a multiple of blocksize
Movies/
Radarr_Movies/
TV/
TV_Ended/
Test/
mounted
2019/04/14 23:39:37 DEBUG : 4 go routines active
2019/04/14 23:39:37 DEBUG : rclone: Version "v1.47.0" finishing with parameters ["rclone" "lsf" "gcrypt:" "-vv"]

For your copy command:

rclone copy --config=/config/rclone.conf --buffer-size 512M --checkers 16

I am not sure why you are setting any buffer size. Google limits to 10 transactions per second so setting 16 checkers is going to 403 a lot as the default transfers is 4. I’d run with 4 and 4 or something along those lines.

drive-chunk-size is useful on copy commands as 32M or 64M is generally the sweet spot for Google.

My command I use is pretty straight forward with an exclude:

/usr/bin/rclone move /data/local/ gcrypt: -P --checkers 3 --log-file /opt/rclone/logs/upload.log -v --transfers 3 --drive-chunk-size 32M --exclude-from /opt/rclone/scripts/excludes --delete-empty-src-dirs

I have some folders I don’t upload so I filter those out.

Yeah I just ran

rclone ls -vv direct-decrypt:

and got nothing. There were no errors except for some 403 rate errors. Nothing even resembling the error I mentioned earlier.

So do you mean I should just leave checkers at the default 4? Why did you adjust your checkers and transfers to 3? How did you know if 32M or 64M is better for chunksize?

rclone copy --config=/config/rclone.conf --drive-chunk-size 32M

Does the directory_name_encryption = true matter? I didn’t have it but have all of my directories encrypted. Not sure what happened.

It’s super hard asking many different questions in the same thread to follow what you are trying to fix or change.

If that isn’t your rclone.conf, what is?

Can you share what your actual config is?
Can you run the same command grab the whole log file and share? Is the error gone now?

The directory names don’t encrypt themselves. If you had it set to true, they would be encrypted. If not, they wouldn’t be.

For uploading, give 32M a test and 64 a test. See what works for your setup. I only wanted to transfer 3 files at a time so I set it to that.

That is my rclone.conf file exactly as is. There was no “directory_name_encryption” and the directory names were being encrypted by itself. Thats what I was trying to say. I think I also just answered my question. It appears that now when editing rclone.conf, it always adds a directory_name_encryption and sets it as either true or false. Mine didn’t have anything, so it might have been because a previous version of rclone didn’t require it.

The error with the files isn’t gone. My mount logs still occasionally show the error. Its not consistent. I believe that it is a bug because I don’t have those file even in the path of the rclone crypt mount.

With uploading, what should I be looking for to know what works better?

If you ran rclone config, it prompts so you have to pick yes or no when you create an encrypted remote:

If your directories are encrypted, it was true.

If you ran the ls command on the same remote as your mount, it would produce the error and capture it.

For uploading, speed would be the thing I’d look for.

I’ve had time to do more testing on this issue. I connected a laptop directly to my modem from Comcast and received a public IP on the device. No router or anything in between.

I then attempted to download files from

  1. Google Drive Web Interface
    I had similar results to before. Speeds were at a steady 2MB/s.

  2. Rclone Rsync copy in Ubuntu VM
    Same results as web interface, slightly slower. Speeds started at 10MB/s then dropped to 1.5MB/s.

  3. Speedtest.net and DSLreports
    Received results of ~950mbps down and ~40mbps up.

I had been searching around and seen this issue but am unsure how to use it?

Should my hostfile look like

172.217.7.234 www.googleapis.com
or
172.217.7.234 googleapis.l.google.com

If its the first one, I tried two different IPs and had no luck.

You were right about Google not throttling, I think. I changed my IP several times and still had these slowdowns. Is this a Comcast issue?

Ok I have been doing some testing using Wireshark to determine the ip that Chrome downloads were coming from to see what I could add to the hosts file. After adding several items to the host file, I was still unable to get it to use the ip that I entered (keep seeing different IPs in Wireshark).

Anyways, I noticed that my downloads from Chrome returned to full speed again when it was the slow speeds yesterday. Copy speeds from Rclone mount have also went to full speed. I made no configuration changes, so I have no clue what happened or if the issue will reappear again.

I would guess you are a victim of throttling either at Comcast or at Google, or maybe just a temporarily congested link.

The drive web API is different to the drive v3 API that rclone uses and it uses different endpoints so can be throttled differently.

If you have trouble again, you could try a VPN - that might help (or might not!)

A VPN does in fact fix it. So does connecting to a different network. It is starting to seem like Comcast more and more. Though my internet searches haven’t brought up anyone with a similar issue.

1 Like

I might have a similar issue (yours was the most similar to mine I could find anyway). I don’t know if it’s the same issue. For me download speeds go to normal when I upload something on my network.

See this Topic:

And this update:

@m1dy
I think I have discovered something new about my setup from your comment. Uploading fixes the issue for me as well. Now that I think about it, I was uploading something the other night when I found that the speeds were “suddenly” fine. It was the uploading that made the speeds go back to normal… This also fixed my download speeds from the web ui as well! (though I would really rather not constantly be uploading files to Gdrive…) Also wanted to add that I have the same results where it doesn’t matter what computer I am uploading from as long as I am uploading somewhere on my local network (same as you).

I’ve also discovered that the uploading speed has to reach a certain minimum speed. I tried speeds of 10kbps and found that it didn’t affect speeds at all. Only when I increased it to 2MB/s did it allow everything to go back to normal. 1MB/s wasn’t enough. Can you try to see if you get the same results with uploading?

I see that a VPN also fixed the issue for you as well… Are you using Comcast? What region are you in? I am in the Chicagoland area.

I’ve discovered another incredible thing about the uploading. It doesn’t even have to be to Gdrive! Running a remote Plex stream, I can see the mount pulling data at my maximum connection speed. Only when I run a local Plex stream does it go back to the slow speeds.

This answers why #2 in the op was occuring. It seems that just uploading in general causes it to work normally again. It also answers #1 in the op as well. My Plex playbacks have always been remote playbacks which means it was always uploading…

When running an upload speedtest while reading a file from mount, I can see the download speeds instantly shoot up and drop back down after the test is complete.

So the conclusion is that it has something to do with uploading files (potentially a modem issue??)

@ayao
I’m happy to know that I am not the only one with this rather mysterious issue :slight_smile:
As I was describing in my update post in the topic I created I also discovered that it doesn’t matter what gets uploaded and whereto. It even doesn’t matter if it is another computer in my local network that uploads to get full speed download on my machine.

In regard to how different upload speeds affects download speed my observations where very similar to yours.

I’m in germany and I’m using the cable internet provider Vodafone. My router (provided by my provider) is a “Fritz!Box 6490 Cable”. So we can at least rule out that it has something to do with our location or our router (as you most probably have another router).

But as I see it everything points to either the router, the integrated modem or provider. Which is confusing as we have (semi-) ruled these things out?

I find this all just.. interesting. Sure sounds like internet provider throttling. :roll_eyes: Set up a 1MB constant upload with rclone with a bw-limit to some vps or Google compute (ingress is free so it'll cost you pennies) and just let it run.

For me, my network setup (sophos xg vm > pfsense vm > gateway in bridge mode Comcast xb6) is very complicated, but I connected a device directly to the modem bypassing my router and still had the issue.

I'll see if calling them helps at all or see if they will let me swap out their modem with my old modem.

If Comcast is throttling Gdrive, wouldn't other people with Comcast notice? I don't understand why I would seemingly be the lone person. Strange...

Good question but it sure smells like it. Comcast and Xfinity do a lot of "interesting" things.

I'm pretty sure that my internet provider doesn't throttle anything. Also it wouldn't make sense that uploading disables the throttling.

Then why does a VPN fix things?

Thats what I'm questioning as well. Though it seems strange that if our ISPs were throttling, why would they stop throttling when uploading? Also it seems strange that we would both have the same experiences with two completely different ISPs. Is there some throttling that gets disabled by uploading? I find that hard to imagine.

The only thing I conceivably say for why the VPN fixes things is that it bypasses something the modem is doing? Maybe the modem/router is at fault? Or its unintentional throttling? I did find this which is very recent but a rep said that they don't throttle.