Rclone mount random slow speeds

cg0 · July 11, 2022, 2:59pm

yes. just edit the script and change the instances in which it refers to KiB and MiB for KB and MB, respectively.

cg0 · July 11, 2022, 3:07pm

this is very much a backend issue. config-wise, there's currently not much that can be done via rclone to remedy the issue. the problem is that if your request ends up with the address of a slow server, you will be stuck with it and then download of 1GB+ files will take way too long finish. so far, the best solution that has been proposed is to test servers ahead of time and pre-select the ones that pass a performance criterion.

antigravity · July 11, 2022, 6:56pm

Thanks yeah I tried that didn’t work- I’ll send through the info you need in the GH. Thanks for your help.

What speeds are everyone experiencing? I’m noticing a limit of around 2.5Mb/s or around 20mbps.

So smaller files seem to play/transcode fine- but larger files are an issue. Plex buffers either direct play or transcoding.

I’m surprised there’s no option to multi thread on mounts? I know you can on the copy command.

asdffdsa · July 11, 2022, 7:16pm

https://forum.rclone.org/t/rclone-1-59-0-release/31808/11

cg0 · July 11, 2022, 7:27pm

that's what I've been doing as well. script-wise, I think it is better to test the current host first to make sure it is actually necessary to run the more comprehensive tests (from the set of IPs returned by dig).

same results here in south america.

el_pedriyo · July 11, 2022, 9:21pm

Well, turning mount into multithread as rclone copy, could be the solution, since rclone copy is performing much better than mount at this moment, as it uses multiple of the routes, no matter if they are quick or not.

antigravity · July 12, 2022, 3:50am

Some pretty poor performance coming from the US. 3 of the 8 endpoints provide sufficient speeds for 4k playback.

Couple of quick questions out of curiosity:

Does RClone only use the whitelisted IP's added to the hosts file?
Do whitelisted IP's get deleted if they test poorly next time the script is run?
If an IP is removed from the hosts file, will it impact any active streams running through that endpoint?

******** Google Endpoint Scanner (GES) ********

[GES] [INFO] The application started on Tue Jul 12 13:35:34 AEST 2022.
[GES] [INFO] Hosts backup file found. Restoring it.
[GES] [INFO] Please wait. Downloading the test file from 142.250.189.170...
[GES] [INFO] Parsing connection with 142.250.189.170.
[GES] [INFO] 212.7 MBytes. Above criterion endpoint. Whitelisting IP '142.250.189.170'.
[GES] [INFO] Please wait. Downloading the test file from 142.251.46.234...
[GES] [INFO] Parsing connection with 142.251.46.234.
[GES] [INFO] 5.097 MBytes. Below criterion endpoint. Blacklisting IP '142.251.46.234'.
[GES] [INFO] Please wait. Downloading the test file from 142.251.214.138...
[GES] [INFO] Parsing connection with 142.251.214.138.
[GES] [INFO] 5.290 MBytes. Below criterion endpoint. Blacklisting IP '142.251.214.138'.
[GES] [INFO] Please wait. Downloading the test file from 172.217.164.106...
[GES] [INFO] Parsing connection with 172.217.164.106.
[GES] [INFO] 225.3 MBytes. Above criterion endpoint. Whitelisting IP '172.217.164.106'.
[GES] [INFO] Please wait. Downloading the test file from 142.250.189.234...
[GES] [INFO] Parsing connection with 142.250.189.234.
[GES] [INFO] 180.4 MBytes. Above criterion endpoint. Whitelisting IP '142.250.189.234'.
[GES] [INFO] Please wait. Downloading the test file from 142.250.191.42...
[GES] [INFO] Parsing connection with 142.250.191.42.
[GES] [INFO] 4.355 MBytes. Below criterion endpoint. Blacklisting IP '142.250.191.42'.
[GES] [INFO] Please wait. Downloading the test file from 142.251.46.170...
[GES] [INFO] Parsing connection with 142.251.46.170.
[GES] [INFO] 4.919 MBytes. Below criterion endpoint. Blacklisting IP '142.251.46.170'.
[GES] [INFO] Please wait. Downloading the test file from 142.251.46.202...
[GES] [INFO] Parsing connection with 142.251.46.202.
[GES] [INFO] 4.878 MBytes. Below criterion endpoint. Blacklisting IP '142.251.46.202'.
[GES] [INFO] The whitelisted IP '142.250.189.234' will be added to the hosts file.
[GES] [INFO] The whitelisted IP '142.250.189.170' will be added to the hosts file.
[GES] [INFO] The whitelisted IP '172.217.164.106' will be added to the hosts file.

cg0 · July 12, 2022, 2:35pm

rclone uses https://www.googleapis.com to interface with the gdrive backend. by default, your machine will first check the hosts file for an address that matches www.googleapis.com and will return the first address that matches it (e.g., 142.250.189.170).

yes. I made a change to mine yesterday that allows running a precheck (USE_PRECHECK) to ensure that the current host is bad before running the more comprehensive tests (from the several ip candidates returned via dig). this reduces bandwidth usage and allows you to run the script more frequently, which is something that I've been doing for the past few days.

no. it will impact future requests to resolve www.googleapis.com.

ashlar · July 12, 2022, 4:39pm

If anybody has the skill to translate the script to a Windows batch, it would be greatly appreciated.

antigravity · July 12, 2022, 4:43pm

Beautiful thanks. If you have multiple whitelisted address it can return any one of these?

Excellent- and this works with multiple whitelisted addresses?

You mention “current host”, I thought the host changed every seperate connection attempt and uses whatever address it can? (Limited to what is in the host file)

Perfect!

Sorry for the 21 questions just trying to better understand how rclone works with these addresses.

Thank you for the script- I’ve already noticed a substantial improvement in streaming performance with it running every 4 hours.

ashlar · July 12, 2022, 4:50pm

Apart from the Windows script request (if anyone is capable of creating it), it would be good for the scripts to allow for IPs not returned from dig/nslookup. From my country/city I get a fixed set of IPs but I've discovered that IPs from other countries in Europe provide, currently, much better performance, for instance.

PS
I am trying to escalate this with Google customer support.

cg0 · July 12, 2022, 5:30pm

by default, it returns only the first match (starting from the beginning of the file until eof). anything fancier than that requires an specialized application (e.g., dnsmasq).

no. it works with the first match in the hosts file, per my previous comment.

I agree but this has not been implemented yet. one idea is for some of us to sponsor this implementation, which would justify bumping its priority dev-wise.

antigravity · July 12, 2022, 5:35pm

So rclone only ever uses the first whitelisted address in the host file- for all connections?

Interesting…

Nebarik · July 13, 2022, 5:06am

Sorry I don't. But if you feel like learning some Linux. Windows has a inbuilt Ubuntu VM you can play with. You'll have to manually update your Windows hosts file with the chosen IP address. But at least the script will run and do the testing for you (assuming you installed rclone).

Considering how fast foreign servers are working, particularly Singaporean ones for me. I've updated my script a little to accept manually included IPs that might not have been found by the local dig.

Create a file called .whitelist-apis in the same folder, and fill it in with some IPs that you like the look of. 1 IP per line. No extra writing please.

So what happens is dig will find a list of IPs that Google gives out for your region.
Then anything in the .whitelist-apis will be added into the mix.
Then anything in the .blacklist-apis will be removed from the apis list. This includes any whitelisted IPs.
Then the script tests the APIs that are left, and any that have turned bad will be added to the .blacklist-apis file.

github.com

Nebarik/mediscripts-shared/blob/main/googleapis.sh


#!/bin/bash

#---------#
# Read me #
#---------#

# This script will check the download speed from the different Google endpoints by downloading a test file. And then it will set your Hosts file with it.
# Any speed measured in KiB/s it will blacklist so it doesnt check again next time, improving script speed. Might be worth clearing out the .blacklist-apis file every now and then in case one of those IPs gets fixed.
# If you have any servers you prefer, particularly from a foreign country that the dig may not find. Add them to a file in this folder, one line each. Default filename is .whitelist-apis
# Example .whitelist-apis file contents
# 123.456.789
# 222.333.444
# The blacklisting will overwrite and take priority over any whitelisted IPs

# For the testfile, if it's too small it will download too quickly for the rclone.log to get a good read on it.
# Bigger the better as Google Drive ramps up speed as it goes, however slow endpoints will take forever. I recommend roughly 50MB.
# Create a file with this command: "fallocate -l 50M dummythicc" and then copy it to your Gdrive somewhere rclone can get at it.

# Troubleshooting:

This file has been truncated. show original

mrlgm007 · July 13, 2022, 10:13am

Also from South Africa and thought it was ISP glad I am not alone on this.

ashlar · July 13, 2022, 10:26am

I was thinking about this yesterday. So I would basically install rclone in the virtual machine, transfer the rclone.conf file in order for the VM installation to use it and run the script. The result would be copied to the hosts file in the VM and I would need to manually adjust it for the Windows hosts file, is that it?

STARK_LINE · July 13, 2022, 1:59pm

download files slow speed - Google Drive Community
this is trending in support community but still they are not giving any attention to this and they are one of the biggest tech gaint in the world, forgot about solving this they don't even reply to this trending problem in community. form my point of view they are just avoiding it.

VBB · July 13, 2022, 3:48pm

The guy starts the topic with "i have shared drive to download 4k movies and web series". Of course that's not gonna get the same attention, if any, as a legit business ticket. Unfortunate, but understandable.

Nebarik · July 14, 2022, 1:00am

Yep, exactly that.
Mine and cg0's scripts announce what they're doing and what IPs have what speeds so you'll have a good idea of which IPs are the good ones.

Cookie-Monster · July 14, 2022, 1:32am

How would I run this if I run rclone in a docker container that has a different network from the host? And would I have to restart the container every time the ip's update?