Oh yeah absolutely. I mentioned it before, bigger is better to let the speed ramp up. The downside is the slow endpoints take a super long time to complete the test.
However my main concern isn't so much trying to get the highest speed results and compare the top endpoints against each other. What I want is to ensure im not using one of the clearly broken 300kb/s endpoints.
Maybe do a small 50MB file to blacklist the clearly broken ones. And then swap the file for a 500MB or something file and try again to get the good endpoints to compete if that's important to you.
Ok i have created the dummyfile and updated the script to reflect my rclone config however the script is getting stuck on
Hosts backup file not found - backing up
Checking my hosts file it has created a single entry which looks wrong
142.250.71.74\twww.googleapis.com
If i ping www.googleapis.com however it seems it is using this IP even though the entry doesnt seem correct to me. The download speed form this server is 300KB/s when testing.
Edit: Worked it out, i had a hosts.backup file already created from another plugin,just had to delete it.
copied from github and only edited my mount
#!/bin/bash
#---------#
# Read me #
#---------#
# This script will check the download speed from the different Google endpoints and then set your Hosts file with it.
# For the testfile, if it's too small it will download too quickly for the rclone.log to get a good read on it.
# Bigger the better as Google Drive ramps up speed as it goes, however slow endpoints will take forever. I recommend roughly 50MB.
# Create a file with this command: "fallocate -l 500M dummythicc"
# If you get the error "tmpapi/speedresults is a directory", this means the didn't find a speed measured in MiB/s, generally safe to ignore unless you are expecting GiB/s.
#-----------------#
# Edit these bits #
#-----------------#
testfile='gdrive_vfs:/temp/dummythicc'
api=www.googleapis.com
#-------------------#
# Hosts file backup #
#-------------------#
for f in /etc/hosts.backup; do
if [ -f "$f" ]; then
printf "Hosts backup file found - restoring\n"
sudo cp $f /etc/hosts
break
else
printf "Hosts backup file not found - backing up\n"
sudo cp /etc/hosts $f
break
fi
done
#-----------------#
# Diggity dig dig #
#-----------------#
mkdir tmpapi
mkdir tmpapi/speedresults/
mkdir tmpapi/testfile/
dig +answer $api +short > tmpapi/api-ips-fresh
#------------------------#
# Backlist Known Bad IPs #
#------------------------#
mv tmpapi/api-ips-fresh tmpapi/api-ips-progress
touch .blacklist-apis
blacklist=.blacklist-apis
while IFS= read -r bip; do
grep -v "$bip" tmpapi/api-ips-progress > tmpapi/api-ips
mv tmpapi/api-ips tmpapi/api-ips-progress
done < "$blacklist"
mv tmpapi/api-ips-progress tmpapi/api-ips
#--------------#
# Colour codes #
#--------------#
RED='\033[1;31m'
YEL='\033[1;33m'
GRN='\033[0;32m'
NC='\033[0m'
#------------------#
# Checking each ip #
#------------------#
input=tmpapi/api-ips
while IFS= read -r ip; do
hostsline="$ip\t$api"
sudo -- sh -c -e "echo '$hostsline' >> /etc/hosts"
printf "Please wait, downloading the test file from $ip... "
rclone copy --log-file tmpapi/rclone.log -v "${testfile}" tmpapi/testfile
if grep -q "KiB/s" tmpapi/rclone.log; then
speed=$(grep "KiB/s" tmpapi/rclone.log | cut -d, -f3 | cut -c 2- | cut -c -5 | tail -1)
printf "${RED}$speed KiB/s${NC} - Blacklisting\n"
rm -r tmpapi/testfile
rm tmpapi/rclone.log
echo "$ip" >> .blacklist-apis
sudo cp /etc/hosts.backup /etc/hosts
else
speed=$(grep "MiB/s" tmpapi/rclone.log | cut -d, -f3 | cut -c 2- | cut -c -5 | tail -1)
printf "${GRN}$speed MiB/s${NC}\n"
echo "$ip" >> tmpapi/speedresults/$speed
rm -r tmpapi/testfile
rm tmpapi/rclone.log
sudo cp /etc/hosts.backup /etc/hosts
fi
done < "$input"
#-----------------#
# Use best result #
#-----------------#
ls tmpapi/speedresults > tmpapi/count
max=$(sort -nr tmpapi/count | head -1)
macs=$(cat tmpapi/speedresults/$max)
printf "${YEL}The fastest IP is $macs at a speed of $max | putting into hosts file\n"
hostsline="$macs\t$api"
sudo -- sh -c -e "echo '$hostsline' >> /etc/hosts"
#-------------------#
# Cleanup tmp files #
#-------------------#
rm -r tmpapi
Wont do it just yet, script doesnt seem to make the directories on Unraid and it also doesnt update the hosts file correctly, the speedtests run fine and it outputs the fastest server but it adds \t the host file
Hosts backup file found - restoring
Please wait, downloading the test file from 142.250.71.74... e[0;32m10.50 MiB/se[0m
Please wait, downloading the test file from 142.250.66.170... e[0;32m7.004 MiB/se[0m
Please wait, downloading the test file from 142.250.66.234... e[1;31m335.0 KiB/se[0m - Blacklisting
e[1;33mThe fastest IP is 142.250.71.74 at a speed of 10.50 | putting into hosts file
Script Finished Jul 04, 2022 17:47.04
the "e[0;32m10" stuff are colour codes. Guessing unraid doesnt support colour output. Dont stress it's only for the text output and wont affect testing.
The "\t" in the hosts file is a bash thing for a tab gap. guessing unraid also doesnt support that.
Sorry man, I've never used unraid before so you might have to refactor the code a little to work better for you.
Or maybe it works fine and just displays poorly for you. Who knows.
yeah i worked out the "\t" thing fine and got it to insert correctly.
The only issue im having is thatUnraid doesn't support file paths without a preceding /. I've tried playing around with your script but im not sure where i need to add /. Typically temporary files are created in unraid on most other scripts inside of /tmp so your mkdir cmd should be something like /tmp/tmpapi etc but im not sure how to edit the rest to make it work.
Ok so on Linux, the root directory is /. And then from there there are nested folders, so the full path to say my home folder would be /home/nebarik/, or unraid's tmp folder /tmp/.
Without the preceding slash that means it's a relative path, referring to a file or folder that exists alongside where you are. I wrote it to use relative paths for maximum compatibility, so it just runs in place whereever you like to keep your scripts.
So for your example if you downloaded my script to /tmp/ and then change directory'd (cd /tmp) into that folder and ran it. it would create a folder called tmpapi/, that has a full path of /tmp/tmpapi.
Again I'm not super familiar with what limitations unraid has over vanilla Linux. But it looked like you had it running eariler? The fact it's outputting speeds means those files in tmpapi/ are being written correctly (and then destroyed in the last step). Assuming you're on the right track and you need to type in full paths. Maybe put /tmp/ before every relative path I wrote to make it a full path?
Luckily I only have 2. tmpapi/ and .blacklist-apis. Make every mention of those into a full path of where you would like them and you should be good.
Did some testing and got a list of five "good" IP addresses from my location. At least during this session (flushing DNS cache every time to change IP) results seemed consistent. Fast IPs were always fast, slow IPs were always slow.
Added the IPs to the hosts file in Windows 10, apparently only the first entry is used, always.
Still don't understand what's going on with Google, to be honest. This seems a problem on their end. Throttling would be done differently, I'm sure.
thanks! I've forked your repo and made a few changes to the script (mediscripts-shared/googleapis.sh at main · cgomesu/mediscripts-shared · GitHub). it is mostly the same thing. just reorganized a bunch, made it posix shell compliant, and included other minor stuff (validations, better termination/cleanup, option to select based on speed rather than 'MiB/s' alone, etc.).
I ran a few tests today as well but in Brazil instead. Here is the last results:
***********************************************
******** Google Endpoint Scanner (GES) ********
***********************************************
[GES] [INFO] The application started on Mon Jul 4 19:17:18 -03 2022.
[GES] [INFO] Hosts backup file found. Restoring it.
[GES] [INFO] Please wait. Downloading the test file from 142.250.79.202...
[GES] [INFO] Parsing connection with 142.250.79.202.
[GES] [INFO] 7.799 MiB/s. Above criterion endpoint. Whitelisting IP '142.250.79.202'.
[GES] [INFO] Please wait. Downloading the test file from 142.251.128.106...
[GES] [INFO] Parsing connection with 142.251.128.106.
[GES] [INFO] 8.006 MiB/s. Above criterion endpoint. Whitelisting IP '142.251.128.106'.
[GES] [INFO] Please wait. Downloading the test file from 142.250.218.10...
[GES] [INFO] Parsing connection with 142.250.218.10.
[GES] [INFO] 7.598 MiB/s. Above criterion endpoint. Whitelisting IP '142.250.218.10'.
[GES] [INFO] Please wait. Downloading the test file from 142.251.128.42...
[GES] [INFO] Parsing connection with 142.251.128.42.
[GES] [WARNING] 615.5 KiB/s. Abnormal endpoint. Blacklisting IP '142.251.128.42'.
[GES] [INFO] Please wait. Downloading the test file from 142.250.219.202...
[GES] [INFO] Parsing connection with 142.250.219.202.
[GES] [INFO] 8.167 MiB/s. Above criterion endpoint. Whitelisting IP '142.250.219.202'.
[GES] [INFO] Please wait. Downloading the test file from 142.251.128.74...
[GES] [INFO] Parsing connection with 142.251.128.74.
[GES] [WARNING] 708.7 KiB/s. Abnormal endpoint. Blacklisting IP '142.251.128.74'.
[GES] [INFO] Please wait. Downloading the test file from 142.250.219.234...
[GES] [INFO] Parsing connection with 142.250.219.234.
[GES] [INFO] 8.001 MiB/s. Above criterion endpoint. Whitelisting IP '142.250.219.234'.
[GES] [INFO] Please wait. Downloading the test file from 142.251.128.10...
[GES] [INFO] Parsing connection with 142.251.128.10.
[GES] [WARNING] 647.2 KiB/s. Abnormal endpoint. Blacklisting IP '142.251.128.10'.
[GES] [INFO] Please wait. Downloading the test file from 142.251.132.42...
[GES] [INFO] Parsing connection with 142.251.132.42.
[GES] [WARNING] 657.1 KiB/s. Abnormal endpoint. Blacklisting IP '142.251.132.42'.
[GES] [INFO] Please wait. Downloading the test file from 216.58.222.10...
[GES] [INFO] Parsing connection with 216.58.222.10.
[GES] [INFO] 7.500 MiB/s. Above criterion endpoint. Whitelisting IP '216.58.222.10'.
[GES] [INFO] Please wait. Downloading the test file from 142.251.132.10...
[GES] [INFO] Parsing connection with 142.251.132.10.
[GES] [INFO] 6.998 MiB/s. Above criterion endpoint. Whitelisting IP '142.251.132.10'.
[GES] [INFO] Please wait. Downloading the test file from 142.250.218.170...
[GES] [INFO] Parsing connection with 142.250.218.170.
[GES] [INFO] 7.600 MiB/s. Above criterion endpoint. Whitelisting IP '142.250.218.170'.
[GES] [INFO] Please wait. Downloading the test file from 142.251.129.42...
[GES] [INFO] Parsing connection with 142.251.129.42.
[GES] [WARNING] 627.0 KiB/s. Abnormal endpoint. Blacklisting IP '142.251.129.42'.
[GES] [INFO] Please wait. Downloading the test file from 142.251.128.138...
[GES] [INFO] Parsing connection with 142.251.128.138.
[GES] [INFO] 7.834 MiB/s. Above criterion endpoint. Whitelisting IP '142.251.128.138'.
[GES] [INFO] Please wait. Downloading the test file from 142.250.219.170...
[GES] [INFO] Parsing connection with 142.250.219.170.
[GES] [INFO] 8.199 MiB/s. Above criterion endpoint. Whitelisting IP '142.250.219.170'.
[GES] [INFO] Please wait. Downloading the test file from 142.250.218.74...
[GES] [INFO] Parsing connection with 142.250.218.74.
[GES] [WARNING] 580.1 KiB/s. Abnormal endpoint. Blacklisting IP '142.250.218.74'.
[GES] [INFO] The fastest IP is 142.250.219.170. Putting into the hosts file.
***********************************************
* Finished Google Endpoint Scanner (GES)
* Message: Reached EOF without errors
***********************************************
which shows 5 problematic Google servers in my case. (8MB/s is as far as my connection goes. the bad servers are usually transferring at 700KB/s tops.)
@Nebarik (and anyone else running scripts to update the hosts file), how often are you running the script to make sure the addresses remain valid? Have you had any issues so far?
Ok, so... pardon my ignorance here. In the script you state This is done by adding the best GDrive server available at the time of testing to this hosts /etc/hosts file.
So I guess in Linux it works as I see it working in Windows, that is you can just put one IP to point to www.googleapis.com. All the other entries are basically ignored.
So the script tests all the IPs and then puts the fastest one in the hosts file, correct?
How do you get the IP list to test (I think I can use nslookup in Windows)? How do you force rclone to download from a specific IP? The only way I have found in Windows is flushing the DNS cache and trying to get a different IP (sometimes it works, sometimes you're directed to the same one).
This makes much more sense to me now, and i was able to find the .blacklistapi file in my scripts folder. I wasn’t understanding where the files where being created. Thanks for the lesson
Don't know about others but my experiments today are giving me strange results... this afternoon, out of 8 IPs that appear to be pointing at www.googleapis.com, three were slow (20Mbps) and five were fast (in excess of 700Mbps). Tonight I am experimenting some more and I see IPs that this afternoon were slow being fast (the ones that were fast remain fast).
Older systems apparently did. But modern Linux you can have multiple IPs in your host file, and it will act no different to having multiple A records on a domain.
Also I don't think we need to flush the dns as often as Windows requires. At least not in my testing.
I'm personally only bothering with 1 entry for the host file, because I'm worried one of the good ones might turn bad. But theoretically instead of blacklisting bad ips, we could whitelist the good ones and shove them all into the hosts file.
Wow that looks a bit more advanced than mine. I rely too heavily on expecting everything working perfectly with no error handling at all
