Google drive uploads failing, http 429

Help and Support
1

What is the problem you are having with rclone?

For almost 12 months we've been using rclone to copy backup files to google drive.
Yesterday the larger files started failing with:
Failed to copy: googleapi: got HTTP response code 429 with body: your computer or network may be sending automated queries (I'll add the full output below)

We have 8 files that get copied up each day

  • 4 < 100 M
  • 3 < 4 G
  • 1 33G

The three largest (2.3G, 3.6G, 33G) are the ones failing (others succeed)
If I change --drive-chunk-size from 256M to 2G I can get the 2.3G, 3.6G ones up
The 33G one though always seems to fail slightly after about 2G has been uploaded, no matter what drive-chunk-size I use (the other two were failing about the same point before I changed to a bigger chunk)

I've tried different --drive-pacer-min-sleep and --drive-pacer-burst settings to see if I could get it to slow down thinking maybe that would help, It didn't
Tried different --drive-chunk-size sizes including really tiny ones, also didn't help
This is running from an AWS ec2 instance so I tried a different elastic IP, didn't help (also tired a different zone, didn't help)
Tried a different --drive-impersonate user@company.com.au, didn't help
The only thing I've found that works is splitting the file into 2G chunks, 'split -b 2G upload_test.gz "upload_test.gz.part", and uploading that way (this works with --drive-chunk-size 256M)

The only thing I haven't tried that I can think of is setting up a new service account, but nothing in the google console for the account suggests its limited in any way, generally we only reach a rate of 13 api requests per 100 seconds

If anyone has any thoughts I'm open to suggestions

Run the command 'rclone version' and share the full output of the command.

Been using
rclone v1.55.1

  • os/type: linux
  • os/arch: amd64
  • go/version: go1.16.3
  • go/linking: static
  • go/tags: none

Also tried the latest
rclone v1.60.0

  • os/version: centos 7.5.1804 (64 bit)
  • os/kernel: 3.10.0-693.2.2.el7.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.19.2
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -vv --drive-impersonate user@company.com.au copy --error-on-no-transfer upload_test.gz sharedGDrive:test100 --log-file=rclone.105.txt  --drive-chunk-size 256M

The rclone config contents with secrets removed.

[sharedGDrive]
type = drive
client_id = 1234-xxxxx.apps.googleusercontent.com
client_secret = 1a2b3c
scope = drive.file
root_folder_id = 0z9y9x-AbCd
service_account_file = /etc/cron.custom/sqdrive-43e2dba69310.json

A log from the command with the -vv flag

2022/11/17 10:26:52 DEBUG : rclone: Version "v1.60.0" starting with parameters ["rclone" "-vv" "--drive-impersonate" "user@company.com.au" "copy" "--error-on-no-transfer" "upload_test.gz" "sharedGDrive:test100" "--log-file=rclone.105.txt" "--drive-chunk-size" "256M"]
2022/11/17 10:26:52 DEBUG : Creating backend with remote "upload_test.gz"
2022/11/17 10:26:52 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2022/11/17 10:26:52 DEBUG : fs cache: adding new entry for parent of "upload_test.gz", "/backups"
2022/11/17 10:26:52 DEBUG : Creating backend with remote "sharedGDrive:test100"
2022/11/17 10:26:52 DEBUG : sharedGDrive: detected overridden config - adding "{7tj_T}" suffix to name
2022/11/17 10:26:53 DEBUG : fs cache: renaming cache item "sharedGDrive:test100" to be canonical "sharedGDrive{7tj_T}:test100"
2022/11/17 10:26:53 DEBUG : upload_test.gz: Need to transfer - File not found at Destination
2022/11/17 10:26:56 DEBUG : upload_test.gz: Sending chunk 0 length 268435456
2022/11/17 10:27:07 DEBUG : upload_test.gz: Sending chunk 268435456 length 268435456
2022/11/17 10:27:18 DEBUG : upload_test.gz: Sending chunk 536870912 length 268435456
2022/11/17 10:27:29 DEBUG : upload_test.gz: Sending chunk 805306368 length 268435456
2022/11/17 10:27:40 DEBUG : upload_test.gz: Sending chunk 1073741824 length 268435456
2022/11/17 10:27:52 DEBUG : upload_test.gz: Sending chunk 1342177280 length 268435456
2022/11/17 10:27:53 INFO  :
Transferred:        1.264 GiB / 68.079 GiB, 2%, 22.462 MiB/s, ETA 50m46s
Transferred:            0 / 1, 0%
Elapsed time:       1m1.2s
Transferring:
 *                                upload_test.gz:  1% /68.079Gi, 22.461Mi/s, 50m46s

2022/11/17 10:28:03 DEBUG : upload_test.gz: Sending chunk 1610612736 length 268435456
2022/11/17 10:28:15 DEBUG : upload_test.gz: Sending chunk 1879048192 length 268435456
2022/11/17 10:28:26 DEBUG : upload_test.gz: Sending chunk 2147483648 length 268435456
2022/11/17 10:28:31 ERROR : upload_test.gz: Failed to copy: googleapi: got HTTP response code 429 with body: <html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"/><title>Sorry...</title><style> body { font-family: verdana, arial, sans-serif; background-color: #fff; color: #000; }</style></head><body><div><table><tr><td><b><font face=sans-serif size=10><font color=#4285f4>G</font><font color=#ea4335>o</font><font color=#fbbc05>o</font><font color=#4285f4>g</font><font color=#34a853>l</font><font color=#ea4335>e</font></font></b></td><td style="text-align: left; vertical-align: bottom; padding-bottom: 15px; width: 50%"><div style="border-bottom: 1px solid #dfdfdf;">Sorry...</div></td></tr></table></div><div style="margin-left: 4em;"><h1>We're sorry...</h1><p>... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.</p></div><div style="margin-left: 4em;">See <a href="https://support.google.com/websearch/answer/86640">Google Help</a> for more information.<br/><br/></div><div style="text-align: center; border-top: 1px solid #dfdfdf;"><a href="https://www.google.com">Google Home</a></div></body></html>
2022/11/17 10:28:31 ERROR : Attempt 1/3 failed with 1 errors and: googleapi: got HTTP response code 429 with body: <html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"/><title>Sorry...</title><style> body { font-family: verdana, arial, sans-serif; background-color: #fff; color: #000; }</style></head><body><div><table><tr><td><b><font face=sans-serif size=10><font color=#4285f4>G</font><font color=#ea4335>o</font><font color=#fbbc05>o</font><font color=#4285f4>g</font><font color=#34a853>l</font><font color=#ea4335>e</font></font></b></td><td style="text-align: left; vertical-align: bottom; padding-bottom: 15px; width: 50%"><div style="border-bottom: 1px solid #dfdfdf;">Sorry...</div></td></tr></table></div><div style="margin-left: 4em;"><h1>We're sorry...</h1><p>... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.</p></div><div style="margin-left: 4em;">See <a href="https://support.google.com/websearch/answer/86640">Google Help</a> for more information.<br/><br/></div><div style="text-align: center; border-top: 1px solid #dfdfdf;"><a href="https://www.google.com">Google Home</a></div></body></html>
2022/11/17 10:28:32 DEBUG : upload_test.gz: Need to transfer - File not found at Destination
2022/11/17 10:28:33 DEBUG : upload_test.gz: Sending chunk 0 length 268435456
2022/11/17 10:28:44 DEBUG : upload_test.gz: Sending chunk 268435456 length 268435456
2022/11/17 10:28:53 INFO  :
Transferred:        2.463 GiB / 70.092 GiB, 4%, 20.791 MiB/s, ETA 55m30s
Transferred:            0 / 1, 0%
Elapsed time:       2m1.2s
Transferring:
 *                                upload_test.gz:  0% /68.079Gi, 21.993Mi/s, 52m28s

2022/11/17 10:28:56 DEBUG : upload_test.gz: Sending chunk 536870912 length 268435456
2022/11/17 10:29:07 DEBUG : upload_test.gz: Sending chunk 805306368 length 268435456
2022/11/17 10:29:18 DEBUG : upload_test.gz: Sending chunk 1073741824 length 268435456
2022/11/17 10:29:29 DEBUG : upload_test.gz: Sending chunk 1342177280 length 268435456
2022/11/17 10:29:41 DEBUG : upload_test.gz: Sending chunk 1610612736 length 268435456
2022/11/17 10:29:52 DEBUG : upload_test.gz: Sending chunk 1879048192 length 268435456
2022/11/17 10:29:53 INFO  :
Transferred:        3.775 GiB / 70.092 GiB, 5%, 21.912 MiB/s, ETA 51m39s
Transferred:            0 / 1, 0%
Elapsed time:       3m1.2s
Transferring:
 *                                upload_test.gz:  2% /68.079Gi, 22.247Mi/s, 50m52s

2022/11/17 10:30:04 DEBUG : upload_test.gz: Sending chunk 2147483648 length 268435456
2022/11/17 10:30:15 DEBUG : upload_test.gz: Sending chunk 2415919104 length 268435456
2022/11/17 10:30:22 ERROR : upload_test.gz: Failed to copy: googleapi: got HTTP response code 429 with body: <html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"/><title>Sorry...</title><style> body { font-family: verdana, arial, sans-serif; background-color: #fff; color: #000; }</style></head><body><div><table><tr><td><b><font face=sans-serif size=10><font color=#4285f4>G</font><font color=#ea4335>o</font><font color=#fbbc05>o</font><font color=#4285f4>g</font><font color=#34a853>l</font><font color=#ea4335>e</font></font></b></td><td style="text-align: left; vertical-align: bottom; padding-bottom: 15px; width: 50%"><div style="border-bottom: 1px solid #dfdfdf;">Sorry...</div></td></tr></table></div><div style="margin-left: 4em;"><h1>We're sorry...</h1><p>... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.</p></div><div style="margin-left: 4em;">See <a href="https://support.google.com/websearch/answer/86640">Google Help</a> for more information.<br/><br/></div><div style="text-align: center; border-top: 1px solid #dfdfdf;"><a href="https://www.google.com">Google Home</a></div></body></html>
2022/11/17 10:30:22 ERROR : Attempt 2/3 failed with 1 errors and: googleapi: got HTTP response code 429 with body: <html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"/><title>Sorry...</title><style> body { font-family: verdana, arial, sans-serif; background-color: #fff; color: #000; }</style></head><body><div><table><tr><td><b><font face=sans-serif size=10><font color=#4285f4>G</font><font color=#ea4335>o</font><font color=#fbbc05>o</font><font color=#4285f4>g</font><font color=#34a853>l</font><font color=#ea4335>e</font></font></b></td><td style="text-align: left; vertical-align: bottom; padding-bottom: 15px; width: 50%"><div style="border-bottom: 1px solid #dfdfdf;">Sorry...</div></td></tr></table></div><div style="margin-left: 4em;"><h1>We're sorry...</h1><p>... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.</p></div><div style="margin-left: 4em;">See <a href="https://support.google.com/websearch/answer/86640">Google Help</a> for more information.<br/><br/></div><div style="text-align: center; border-top: 1px solid #dfdfdf;"><a href="https://www.google.com">Google Home</a></div></body></html>
2022/11/17 10:30:23 DEBUG : upload_test.gz: Need to transfer - File not found at Destination
2022/11/17 10:30:24 DEBUG : upload_test.gz: Sending chunk 0 length 268435456
2022/11/17 10:30:35 DEBUG : upload_test.gz: Sending chunk 268435456 length 268435456
2022/11/17 10:30:46 DEBUG : upload_test.gz: Sending chunk 536870912 length 268435456
2022/11/17 10:30:53 INFO  :
Transferred:        4.930 GiB / 72.354 GiB, 7%, 21.912 MiB/s, ETA 52m30s
Transferred:            0 / 1, 0%
Elapsed time:       4m1.2s
Transferring:
 *                                upload_test.gz:  0% /68.079Gi, 22.483Mi/s, 51m10s

2022/11/17 10:30:57 DEBUG : upload_test.gz: Sending chunk 805306368 length 268435456
2022/11/17 10:31:09 DEBUG : upload_test.gz: Sending chunk 1073741824 length 268435456
2022/11/17 10:31:20 DEBUG : upload_test.gz: Sending chunk 1342177280 length 268435456
2022/11/17 10:31:32 DEBUG : upload_test.gz: Sending chunk 1610612736 length 268435456
2022/11/17 10:31:43 DEBUG : upload_test.gz: Sending chunk 1879048192 length 268435456
2022/11/17 10:31:53 INFO  :
Transferred:        6.263 GiB / 72.354 GiB, 9%, 21.912 MiB/s, ETA 51m28s
Transferred:            0 / 1, 0%
Elapsed time:       5m1.2s
Transferring:
 *                                upload_test.gz:  2% /68.079Gi, 23.258Mi/s, 48m29s

2022/11/17 10:31:54 DEBUG : upload_test.gz: Sending chunk 2147483648 length 268435456
2022/11/17 10:32:00 ERROR : upload_test.gz: Failed to copy: googleapi: got HTTP response code 429 with body: <html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"/><title>Sorry...</title><style> body { font-family: verdana, arial, sans-serif; background-color: #fff; color: #000; }</style></head><body><div><table><tr><td><b><font face=sans-serif size=10><font color=#4285f4>G</font><font color=#ea4335>o</font><font color=#fbbc05>o</font><font color=#4285f4>g</font><font color=#34a853>l</font><font color=#ea4335>e</font></font></b></td><td style="text-align: left; vertical-align: bottom; padding-bottom: 15px; width: 50%"><div style="border-bottom: 1px solid #dfdfdf;">Sorry...</div></td></tr></table></div><div style="margin-left: 4em;"><h1>We're sorry...</h1><p>... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.</p></div><div style="margin-left: 4em;">See <a href="https://support.google.com/websearch/answer/86640">Google Help</a> for more information.<br/><br/></div><div style="text-align: center; border-top: 1px solid #dfdfdf;"><a href="https://www.google.com">Google Home</a></div></body></html>
2022/11/17 10:32:00 ERROR : Attempt 3/3 failed with 1 errors and: googleapi: got HTTP response code 429 with body: <html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"/><title>Sorry...</title><style> body { font-family: verdana, arial, sans-serif; background-color: #fff; color: #000; }</style></head><body><div><table><tr><td><b><font face=sans-serif size=10><font color=#4285f4>G</font><font color=#ea4335>o</font><font color=#fbbc05>o</font><font color=#4285f4>g</font><font color=#34a853>l</font><font color=#ea4335>e</font></font></b></td><td style="text-align: left; vertical-align: bottom; padding-bottom: 15px; width: 50%"><div style="border-bottom: 1px solid #dfdfdf;">Sorry...</div></td></tr></table></div><div style="margin-left: 4em;"><h1>We're sorry...</h1><p>... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.</p></div><div style="margin-left: 4em;">See <a href="https://support.google.com/websearch/answer/86640">Google Help</a> for more information.<br/><br/></div><div style="text-align: center; border-top: 1px solid #dfdfdf;"><a href="https://www.google.com">Google Home</a></div></body></html>
2022/11/17 10:32:00 INFO  :
Transferred:        6.288 GiB / 6.288 GiB, 100%, 21.912 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:       5m8.5s

2022/11/17 10:32:00 DEBUG : 3 go routines active
2022/11/17 10:32:00 Failed to copy: googleapi: got HTTP response code 429 with body: <html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"/><title>Sorry...</title><style> body { font-family: verdana, arial, sans-serif; background-color: #fff; color: #000; }</style></head><body><div><table><tr><td><b><font face=sans-serif size=10><font color=#4285f4>G</font><font color=#ea4335>o</font><font color=#fbbc05>o</font><font color=#4285f4>g</font><font color=#34a853>l</font><font color=#ea4335>e</font></font></b></td><td style="text-align: left; vertical-align: bottom; padding-bottom: 15px; width: 50%"><div style="border-bottom: 1px solid #dfdfdf;">Sorry...</div></td></tr></table></div><div style="margin-left: 4em;"><h1>We're sorry...</h1><p>... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.</p></div><div style="margin-left: 4em;">See <a href="https://support.google.com/websearch/answer/86640">Google Help</a> for more information.<br/><br/></div><div style="text-align: center; border-top: 1px solid #dfdfdf;"><a href="https://www.google.com">Google Home</a></div></body></html>
2 Likes
2

That's generally a problem at Google from past experience. I'm not sure anything to do other than wait it out.

You can try adjust the user-agent and see if that helps as that was an issue in the past.

3

Thanks for the tip Animosity022. I forgot to mention I did try a few different user agents while I was trying a different IP, but just like changing the IP it didn't help

I didn't really want to play the waiting game but it seems I may have no choice

4

Usually if it is more wide spread, there would be more posts. I migrated away quite some time ago so I’m can’t test much myself.

5

I got exactly the same issue since yesterday. Looks like there is something changed on google's end.

I have tried service accounts, but it still has the same issue anyway.

6

I am getting this error too today.
I can upload nfo or jpg files to gdrive without issues, but cannot upload mkv files (>3GB), got the exact 429 errors.

7

After a bunch of testing I found as long as I'm using version v1.60.0 adding --drive-upload-cutoff 1000T fixes it
(doesn't fix it on v1.55.1, I didn't try the versions in between)

--drive-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 8Mi)
Setting it to 1000T effectively disables the chunked uploading

2 Likes
8

The error message says this

I suspect that is some new Google rate limiting kicking in.

I think it is breaking this legitimate use of the API so hopefully they will roll it back soon.

That will mean rclone just uses a single HTTP connection to upload your data which is evading the new check.

Multipart uploads are more reliable as the individual parts can be retried, but it is a decent workaround until Google fixes their problem.

1 Like
9

Confirmed that it works.

Hopefully they will roll it back.

10

I've also been seeing this issue w/ Google Drive over the last few days. Prior to that, rclone has been rock solid for multiple years -- thanks for that!

I just attempted using the --drive-upload-cutoff 1000T workaround but am still getting the 429 errors on files over ~1.5GB. I've tried copying the same file from multiple client addresses with mixed results:

I've tried uploading to Drive from two different AWS EC2 instances with different client IPs. This consistently results in a 429 error. The amount of data written in Drive varies per copy attempt by up to 100MB. Sometimes 1.6GB will be written to Drive, other times 1.7GB or 1.8GB will be written.

I've also tried uploading the same source file to the same Drive account from my home internet (about ~20Mb upload speed) and the file copied up without issue.

This left me with at least two possible explanations:

  1. Google has targeted a set of IPs addresses from AWS and applied some sort of limitation. This seems unlikely to me and could be invalidated by testing on another client w/ a fast upload pipe. I haven't had time to do this yet.

  2. As you suggest, Google could have deployed new rate limiting rules and clients with faster upload speeds might be disproportionately affected. This could explain why uploading from AWS instances fail but uploading from my slower home connection works.

11

Hmm...

It might be worth experimenting with --tpslimit to slow rclone down a little. Try 5 and increase/decrease if successful/unsuccessful.

12

I tried " --tpslimit 5 --drive-upload-cutoff 1000T" and it works!
I am uploading files from an azure VM, files are larger than 3GB.

13

I just tried --drive-upload-cutoff 1000T --tpslimit 1 (also tried --tpslimit 5) and still getting 429 errors from AWS hosts.

14

@andym I'm also using AWS (Australia region).
Your symptoms sound a little different, my uploads were consistently dying slightly after the 2G mark across EC2's. Since adding --drive-upload-cutoff 1000T my crons have been working successfully for the past two days

Based on other topics Google sends 429 errors for other things as well, make sure the message is the same

I had some success adding --drive-chunk-size 2G for flies between 2G - 5G but it was inconsistent, and didn't work for the really large files but might be worth a shot

--drive-upload-cutoff 1000T only worked when I was on the latest version of rclone, so double check your on 1.60

15

I upgraded to 1.60.0 shortly after seeing the 429 errors for the first time 5 or so days ago,

The 429 error message I'm getting includes an HTML payload in the response with the following text message (HTML markup removed for clarity): We're sorry... ... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.

I just tried several different --drive-chunk-size values (64M and 512M) -- both produce the 429 errors. The instance I'm testing with only has 1G of total ram, so I can't try anything higher. These attempts also included the --drive-upload-cutoff 1000T option as well.

I've also tried modifying the user agent to protect against any kind of application-specific filtering Google might be doing. That didn't provide any relief.

I've used rclone to talk to Drive from this specific EC2 instance/IP address for 4+ years now. None of the other software on the instance or the overall usage pattern has changed. Last week was the first time I've seen a 429 error.

16

Same problem. Started on Nov 15. Happens on an AWS server, not from my home server.
Added --drive-upload-cutoff 1000T and it worked to upload a 3.3GB file without issues.

17

@GBoudreau Your problems with Drive started the same time mine did. It's weird that the --drive-upload-cutoff 1000T has worked for you and others but not for me. Would you mind sharing what other options (if any) that you're passing to rclone? Prior to November 15, I was passing no other options to rclone.

For what it's worth, I am wrapping my Drive remote with a Crypt remote.

18

I use no other options: rclone copy --drive-upload-cutoff 1000T source target

19

--drive-upload-cutoff 1000T works for me with copy, but not with mount.

Anyone else experiencing this?

20

Oh! --drive-upload-cutoff 1000T does in fact work for me w/ rclone copy but does not work with rclone mount. Good catch!

cc @ncw