Unable to upload to Gsuite as of a couple hours ago

Sadly no. Here's what I've got:

removed

I'm still getting those 429s. I guess they've got my number - might be time to start researching other cloud storage.

EDIT 18:25 - Strangely its now back... maybe the changes needed a bit to take effect? Nevertheless this is weird because I am not hitting my API limits at all when I check.

Damn it - the DNS trick for my HETZNER cloud server at nuernberg stopped working after one day :frowning:

I changed the DNS entry at

  • /etc/systemd/resolved.conf
  • /etc/netplan/50-cloud-init.yaml

Did a "sudo systemd-resolve --flush-caches" and also changed the rclone copy command to:

rclone copy "/sync/"  "Gdrive:" --drive-chunk-size=256M --tpslimit 5 --drive-upload-cutoff 1000T --drive-acknowledge-abuse=true --bwlimit=50M --use-mmap --transfers=4 --checkers=8 --user-agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36 Edg/109.0.1518.70" --drive-stop-on-upload-limit

And get now the same error message from google drive :frowning: " Failed to copy with 10 errors: last error was: googleapi: got HTTP response code 429 with body:"

What can I do? Please help me THX :slight_smile:

are you sure your dns's didnt revert to hetzners?

i had this problem after a day.. then saw that my dns's had reverted to hetzners without even a reboot.. changed them to cloudflare again and all has been well for days....

i keep wondering if the people whio say its stopped working have also had the dns revert..?

when you use a Hetzner image you have to remove the hardlink for /etc/resolve.conf or netplan will override it.then just create a new one with your nameservers
rm /etc/resolve.conf
nano /etc/resolve.conf

thank you - that was the solution :slight_smile:

But the file has no be named "resolv.conf"

THX

Im still getting error on moving files.

I have edited "sudo nano /etc/systemd/resolved.conf"
to:

[Resolve]
DNS=1.1.1.1 1.0.0.1
FallbackDNS=8.8.8.8

I have also added this to "sudo nano /etc/resolv.conf"


### Hetzner Online GmbH installimage
# nameserver config
nameserver 185.xx.xx.2
nameserver 2a01:xxx:x::add:1
nameserver 185.xx.xx.1
nameserver 2xxxxxx00::add:2
      nameservers:
        addresses:
          - 1.0.0.1
          - 1.1.1.1

I then:
sudo systemd-resolve --flush-caches

I tried to use:
rclone move --no-traverse --delete-empty-src-dirs -v --tpslimit 5 --transfers 1 --bwlimit 35m --max-transfer 740G --ignore-existing --drive-upload-cutoff 1000T --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" '/home/xxxxxx/storage/down/move/other02' remote:

But i still getting this error:
"Failed to copy: googleapi: got HTTP response code 429 with body"

What am i missing??
Please, can someone help me?

I think in /etc/resolv.conf you need to update the name servers in red not add new rows.

thank you for answering me!
That didnt help either :frowning:

What’s your OS by any chance?

Can you do a nslookup to a website to check what DNS is being used?

Thanks!
Im using:
Linux Debian-1101-bullseye-amd64-base 5.10.0-16-amd64 #1 SMP Debian 5.10.127-2 (2022-07-23) x86_64

if i run nslookup www.yahoo.com

Server:         185.12.64.2
Address:        185.12.64.2#53

If sudo nano /etc/resolv.conf i added it like this:

### Hetzner Online GmbH installimage
## nameserver config
nameserver 185.12.64.2
nameserver 2a01:x::adxxxd:1
nameserver 185.12.64.1
nameserver 2a01:xxxxx::add:2
nameservers 1.0.0.1
nameservers 1.1.1.1

I tried it like this but my remote to google didnt work after this?

### Hetzner Online GmbH installimage
## nameserver config
nameservers 1.0.0.1
nameservers 1.1.1.1

Still using the old dns. Remove the ones starting with 185 as you did, but rename from nameservers to nameserver for both rows and should work.

Ohhh i did miss the "S"
Thank you soooo much!

Will try this as soon as possible!

It's on and off for me. Some days will be smooth and others the 429 error will be back.

I just want to thank you for taking time and help me!
Its now working.

Thanks to the good tips here with the change to the cloudflare DNS servers and the DNS flush, the uploads here are running again.

Servers are in Falkenstein, Germany, running Ubuntu Server 22.04. We haven't changed anything in our rclone commands.

Our DNS change commands:

sudo nano /etc/netplan/01-netcfg.yaml

Change the nameservers in this file:

       name servers:
         addresses:
           - 1.1.1.1

First try whether the network is still reachable after the changes:

sudo netplan try

If everything works:

sudo netplan apply

Clear DNS cache:

sudo systemd-resolve --flush-caches

Check if the cache has been cleared:

resolvectl statistics

If the answer there is 0 under size:

cache
Current cache size: 0

the DNS cache was emptied.

Even without a reboot it has been working for us ever since:

Transferred: 38,780 GiB / 38,780 GiB, 100%, 62,606 MiB/s, ETA 0s
Checks: 8 / 8, 100%
Transferred: 8 / 8, 100%
Elapsed time: 8m15.6s

I.e. the uploads run more or less normally again with approx. 600-700 MBit (although they almost always ran with 1 GBit before the upload lock, but it doesn't matter).

really helpful post, for me thank you for sharing this with us

Incredible effort by the community. Thank you to everyone.

It's never DNS amiright?! haha

Is anyone else experiencing the 429 errors sporadically after the DNS changes? For example, if my server is downloading things at the same time multiple users are watching stuff... Plex will stop playing, circle of death, or kick them off completely. Looking at logs, the 429 errors come back in what appears to be when I'm uploading to drive and other users are trying play from drive.

I can see the error when uploading while something is being streamed or downloaded from Google, but its like an information only, without any impact.

Although I changed my DNS to Google DNS, and confirmed it's working (nslookup is reporting), I still get 429s. The workaround might work with bwlimited uploads, but for me with Gigabit and two files I get a 429 thrown after a few gb of upload.I really hope Hetzner is in touch with Google to sort this out. Until that is fixed, I guess I have to upload via http proxy.