GDrive Error 403 The download quota for this file has been exceeded

Help and Support
1

What is the problem you are having with rclone?

I've been running rclone for months (years?) just fine but the last couple days I've been having repeated problems with Google Drive. Is anyone else experiencing download exceeded 403 errors?

ReadFileHandle.Read error: low level retry 8/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded

My gdrive size is about 70TB, and it's mounted locally as well as a remote server (I don't have the best upload speeds). I use separate api credentials and applications for the local vs remote mount (but still connect to the same accounts storage). I don't have a ton of people accessing my plex server. The most is about 5-6 concurrent streams and it's usually around 1-2. Is this just a GDrive bug/problem? I've been running the same commands/config for at least a year without trouble.

Here's the last week from console

image
image1632×559 52.7 KB

image

What is your rclone version (output from rclone version)

rclone v1.46

  • os/arch: linux/amd64
  • go version: go1.11.5

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04
Codename: bionic

(64 bit)

Which cloud storage system are you using? (eg Google Drive)

GSuite Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

This is my mount command

rclone mount gsuite: /data/gsuite.encrypt --rc --allow-other --log-level ERROR --syslog --umask 022 --allow-non-empty --fuse-flag sync_read --tpslimit 10 --tpslimit-burst 10 --dir-cache-time=160h --buffer-size=64M --attr-timeout=1s --vfs-read-chunk-size=128M --vfs-cache-max-age=5m --vfs-cache-mode=writes --vfs-read-chunk-size-limit=off

I use encfs for encryption and mergerfs to fuse remote and local cache

The rclone config contents with secrets removed.

[gsuite]
type = drive
client_id = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com
client_secret = xxxxxxxxxxxxxxxxxxxxxxx
scope = drive
token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","token_type":"Bearer","refresh_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","expiry":"2020-09-05T18:22:37.869946632-07:00"}

A log from the command with the -vv flag

Sep  5 08:26:16 domain-tld rclone[998]: ,qTgiHVExY3EvJtodbHVWVUL/NY9PS-SeI2G6pAIhtR6PUfJc/,tjoHa1RBnvp6-eYwlwiHACw,SmCHRlyu-VDtDGqd,iE0-/GZz3v-rTFl4PrqnthqTzLnOKbkBNU2R0Jol148bv09uk4PmTlSxTQDvk6ItcM78EjkA: ReadFileHandle.Read error: low level retry 1/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
Sep  5 08:26:16 domain-tld rclone[998]: ,qTgiHVExY3EvJtodbHVWVUL/RvRZJOoIqlQZruUvcMRcBuC1/FYpxlrt71V75pTL7PXagWNH078-c0zrn32Nqkuc7ssoo1,/FG,fAcG6QEpVcmw5FmPWtBc2/1Em7AwGHYOl,f-mGn0LPQluR0A2bkvJiOOimmTXkt2g6wVer38KxLNLsua-1fbBtwKg31aDFqDQo1Hz-r9VjrkLiVQBLx51xjxSQLpkU,XrF9tsfGxx6peYzHFaXHZngSn4: ReadFileHandle.Read error: low level retry 1/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
Sep  5 08:26:16 domain-tld rclone[998]: ,qTgiHVExY3EvJtodbHVWVUL/NY9PS-SeI2G6pAIhtR6PUfJc/,tjoHa1RBnvp6-eYwlwiHACw,SmCHRlyu-VDtDGqd,iE0-/GZz3v-rTFl4PrqnthqTzLnOKbkBNU2R0Jol148bv09uk4PmTlSxTQDvk6ItcM78EjkA: ReadFileHandle.Read error: low level retry 2/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
Sep  5 08:26:16 domain-tld rclone[998]: Qkv4qksNQCqgz3JLyTKFLYPL/TkVRZggLJ4Nj8e0RJHX7QRce9thrVISDHocjlODhVQM780/CrucoR0Za,jm7-B0NyOumyo-/VwwdYww0v0WBjVRUbAVmXhTAeYfqRKvlNeMK-B2c,DVFbQ2,CyyGIvmLyblHeWkjFt,K1tXoudSiUWQTE7PqOLslojlRchBd2cNqR5hfOyk0g-: ReadFileHandle.Read error: low level retry 1/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
Sep  5 08:26:16 domain-tld rclone[998]: ,qTgiHVExY3EvJtodbHVWVUL/RvRZJOoIqlQZruUvcMRcBuC1/FYpxlrt71V75pTL7PXagWNH078-c0zrn32Nqkuc7ssoo1,/FG,fAcG6QEpVcmw5FmPWtBc2/1Em7AwGHYOl,f-mGn0LPQluR0A2bkvJiOOimmTXkt2g6wVer38KxLNLsua-1fbBtwKg31aDFqDQo1Hz-r9VjrkLiVQBLx51xjxSQLpkU,XrF9tsfGxx6peYzHFaXHZngSn4: ReadFileHandle.Read error: low level retry 2/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
Sep  5 08:26:16 domain-tld rclone[998]: ,qTgiHVExY3EvJtodbHVWVUL/NY9PS-SeI2G6pAIhtR6PUfJc/,tjoHa1RBnvp6-eYwlwiHACw,SmCHRlyu-VDtDGqd,iE0-/GZz3v-rTFl4PrqnthqTzLnOKbkBNU2R0Jol148bv09uk4PmTlSxTQDvk6ItcM78EjkA: ReadFileHandle.Read error: low level retry 3/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
Sep  5 08:26:16 domain-tld rclone[998]: ,qTgiHVExY3EvJtodbHVWVUL/RvRZJOoIqlQZruUvcMRcBuC1/FYpxlrt71V75pTL7PXagWNH078-c0zrn32Nqkuc7ssoo1,/FG,fAcG6QEpVcmw5FmPWtBc2/1Em7AwGHYOl,f-mGn0LPQluR0A2bkvJiOOimmTXkt2g6wVer38KxLNLsua-1fbBtwKg31aDFqDQo1Hz-r9VjrkLiVQBLx51xjxSQLpkU,XrF9tsfGxx6peYzHFaXHZngSn4: ReadFileHandle.Read error: low level retry 3/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
Sep  5 08:26:17 domain-tld rclone[998]: ,qTgiHVExY3EvJtodbHVWVUL/NY9PS-SeI2G6pAIhtR6PUfJc/,tjoHa1RBnvp6-eYwlwiHACw,SmCHRlyu-VDtDGqd,iE0-/GZz3v-rTFl4PrqnthqTzLnOKbkBNU2R0Jol148bv09uk4PmTlSxTQDvk6ItcM78EjkA: ReadFileHandle.Read error: low level retry 4/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
Sep  5 08:26:17 domain-tld rclone[998]: ,qTgiHVExY3EvJtodbHVWVUL/RvRZJOoIqlQZruUvcMRcBuC1/FYpxlrt71V75pTL7PXagWNH078-c0zrn32Nqkuc7ssoo1,/FG,fAcG6QEpVcmw5FmPWtBc2/1Em7AwGHYOl,f-mGn0LPQluR0A2bkvJiOOimmTXkt2g6wVer38KxLNLsua-1fbBtwKg31aDFqDQo1Hz-r9VjrkLiVQBLx51xjxSQLpkU,XrF9tsfGxx6peYzHFaXHZngSn4: ReadFileHandle.Read error: low level retry 4/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
Sep  5 08:26:17 domain-tld rclone[998]: Qkv4qksNQCqgz3JLyTKFLYPL/TkVRZggLJ4Nj8e0RJHX7QRce9thrVISDHocjlODhVQM780/CrucoR0Za,jm7-B0NyOumyo-/VwwdYww0v0WBjVRUbAVmXhTAeYfqRKvlNeMK-B2c,DVFbQ2,CyyGIvmLyblHeWkjFt,K1tXoudSiUWQTE7PqOLslojlRchBd2cNqR5hfOyk0g-: ReadFileHandle.Read error: low level retry 2/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
Sep  5 08:26:17 domain-tld rclone[998]: ,qTgiHVExY3EvJtodbHVWVUL/NY9PS-SeI2G6pAIhtR6PUfJc/,tjoHa1RBnvp6-eYwlwiHACw,SmCHRlyu-VDtDGqd,iE0-/GZz3v-rTFl4PrqnthqTzLnOKbkBNU2R0Jol148bv09uk4PmTlSxTQDvk6ItcM78EjkA: ReadFileHandle.Read error: low level retry 5/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
Sep  5 08:26:17 domain-tld rclone[998]: ,qTgiHVExY3EvJtodbHVWVUL/RvRZJOoIqlQZruUvcMRcBuC1/FYpxlrt71V75pTL7PXagWNH078-c0zrn32Nqkuc7ssoo1,/FG,fAcG6QEpVcmw5FmPWtBc2/1Em7AwGHYOl,f-mGn0LPQluR0A2bkvJiOOimmTXkt2g6wVer38KxLNLsua-1fbBtwKg31aDFqDQo1Hz-r9VjrkLiVQBLx51xjxSQLpkU,XrF9tsfGxx6peYzHFaXHZngSn4: ReadFileHandle.Read error: low level retry 5/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
Sep  5 08:26:17 domain-tld encfs[987]: 2020-09-05 08:26:17,894 WARN  read failed at offset 16392 for 1024 bytes: Input/output error
Sep  5 08:26:17 domain-tld rclone[998]: ,qTgiHVExY3EvJtodbHVWVUL/NY9PS-SeI2G6pAIhtR6PUfJc/,tjoHa1RBnvp6-eYwlwiHACw,SmCHRlyu-VDtDGqd,iE0-/GZz3v-rTFl4PrqnthqTzLnOKbkBNU2R0Jol148bv09uk4PmTlSxTQDvk6ItcM78EjkA: ReadFileHandle.Read error: low level retry 6/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
Sep  5 08:26:18 domain-tld rclone[998]: ,qTgiHVExY3EvJtodbHVWVUL/RvRZJOoIqlQZruUvcMRcBuC1/FYpxlrt71V75pTL7PXagWNH078-c0zrn32Nqkuc7ssoo1,/FG,fAcG6QEpVcmw5FmPWtBc2/1Em7AwGHYOl,f-mGn0LPQluR0A2bkvJiOOimmTXkt2g6wVer38KxLNLsua-1fbBtwKg31aDFqDQo1Hz-r9VjrkLiVQBLx51xjxSQLpkU,XrF9tsfGxx6peYzHFaXHZngSn4: ReadFileHandle.Read error: low level retry 6/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
Sep  5 08:26:18 domain-tld rclone[998]: Qkv4qksNQCqgz3JLyTKFLYPL/TkVRZggLJ4Nj8e0RJHX7QRce9thrVISDHocjlODhVQM780/CrucoR0Za,jm7-B0NyOumyo-/VwwdYww0v0WBjVRUbAVmXhTAeYfqRKvlNeMK-B2c,DVFbQ2,CyyGIvmLyblHeWkjFt,K1tXoudSiUWQTE7PqOLslojlRchBd2cNqR5hfOyk0g-: ReadFileHandle.Read error: low level retry 3/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
Sep  5 08:26:18 domain-tld encfs[987]: 2020-09-05 08:26:18,308 WARN  read failed at offset 8 for 1024 bytes: Input/output error
Sep  5 08:26:18 domain-tld rclone[998]: ,qTgiHVExY3EvJtodbHVWVUL/NY9PS-SeI2G6pAIhtR6PUfJc/,tjoHa1RBnvp6-eYwlwiHACw,SmCHRlyu-VDtDGqd,iE0-/GZz3v-rTFl4PrqnthqTzLnOKbkBNU2R0Jol148bv09uk4PmTlSxTQDvk6ItcM78EjkA: ReadFileHandle.Read error: low level retry 7/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
Sep  5 08:26:18 domain-tld rclone[998]: ,qTgiHVExY3EvJtodbHVWVUL/RvRZJOoIqlQZruUvcMRcBuC1/FYpxlrt71V75pTL7PXagWNH078-c0zrn32Nqkuc7ssoo1,/FG,fAcG6QEpVcmw5FmPWtBc2/1Em7AwGHYOl,f-mGn0LPQluR0A2bkvJiOOimmTXkt2g6wVer38KxLNLsua-1fbBtwKg31aDFqDQo1Hz-r9VjrkLiVQBLx51xjxSQLpkU,XrF9tsfGxx6peYzHFaXHZngSn4: ReadFileHandle.Read error: low level retry 7/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
Sep  5 08:26:18 domain-tld rclone[998]: ,qTgiHVExY3EvJtodbHVWVUL/NY9PS-SeI2G6pAIhtR6PUfJc/,tjoHa1RBnvp6-eYwlwiHACw,SmCHRlyu-VDtDGqd,iE0-/GZz3v-rTFl4PrqnthqTzLnOKbkBNU2R0Jol148bv09uk4PmTlSxTQDvk6ItcM78EjkA: ReadFileHandle.Read error: low level retry 8/10: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2

That's an old version of rclone as you'd want to update that.

You should take a look at the new vfs-cache-mode full as well as that will help out with streaming as well.

3

This is essentially the way my system works with my local cache and mergerfs. I guess the rclone version should say something about how long I've been using this. I'll try it out though as it could help with older content that's suddenly streamed more frequently. I'll try the updated rclone and see if there are any changes.

4

That's not how the VFS cache works as it keeps files in cache that are being used and only grabs the pieces.

Example would be if you play something in Plex, it opens and closes the same file 3 times before it plays it and you'd have 3 requests for those pieces. With vfs-cache-mode full, you get 1 request and it greatly reduces API hits.

5

Right, but when files are new, they're always grabbed from my local storage anyways. Still, I see the benefit so I'll give it a shot.

6

updated to rclone v1.53.0 and I'm using the following vfs flags
--vfs-read-chunk-size=128M --vfs-cache-max-age=48h --vfs-cache-max-size=64G --vfs-cache-mode=full --vfs-read-chunk-size-limit=off

I guess I'll find out in a day or so if it's working. I plan to slowly turn on services on each server so I don't overburden the mounts.

7

--vfs-read-chunk-size=128M and --vfs-read-chunk-size-limit=off are the defaults.
this sould work good. Would prefer the --buffer-size defaults too.

8

I have had the exact same issue in the past 5 days, banned twice (don't remember when was the last time before these ones).

Screenshot 2020-09-07 at 10.50.19

I thought it was related to updating to the latest v1.53.0 and adding the VFS full but might be something else, I'm using @Animosity022 recommended mount flags.

9

@D34DC3N73R do you have the option "Generate intro video markers" enabled as a scheduled task by any chance? I have checked my logs on both of the servers I have (1x dedicated for family, 1x Raspberry for myself at home) and the errors start at the exact same time, they both run Scheduled Tasks every 3 days at 2am.

10

I do, (because I enjoy the feature) but I have mine set when media is added and as a scheduled task. My bans seemed to happen around 5-10pm Pacific time, so I don't think that was the problem but may have contributed to the number of pulls. I've also had that enabled since it was released and never ran into an issue until around a week ago.

Right now, I'm running 2 plex instances, and 2 emby instances, (with no other *arr apps running) and it's been good for almost 2 days. I'm going to start enabling the other apps 1 day at a time and see where that takes me.

11

Update: I'm still up and running with no other bans yet, but I don't have my full stack up yet either. Looking back on my quotas, something definitely changed on Sept 4th, but I'm not sure what exactly that was.

image
image2343×796 44 KB

12

I think Google changed something about accessing files, several people having problems to copy/acess files, public links and your own files. In a group that I participate in, the peope started to suspect that something is wrong. I'm having the same problem to sync a folder

13

Since moving to 1.53 with --vfs-cache-mode FULL I have been experiencing these 403 errors every day.

I am going to go back to my old rclone config with --vfs-cache-mode WRITES, and see if the behavior changes. I use Sonarr/Radarr/Bazarr and Plex. Plex is not configured to do anything crazy and none of my app config has changed when I upgraded to 1.53. I had done some testing with the beta in a test environment and I had not experienced this problem, but I can't be any more specific than that.

14

I stream quite a bit from my gdrive rclone mount and I have not seen any 403's in my logs. I also have backup jobs from my gdrive mount to jotts cloud, and they also don't seem to be affected. I've been on 1.53 since it was released.

15

I'm also having 403 error for recently uploaded files, never touched before and definitely not used enough to trigger 403 errors :frowning:

Been trying for 3 days to download a 5 TB folder to my disk with no sucess. And those files aren't used anywhere

16

I have been having exactly the same problems since switching to 1.53 with the same config i used before...

, I will switch back to 1.52.3 and confirm if I also have got the 403 errors.

17

For what it's worth I have a machine (CentOS 8) still running rclone 1.51.0 (don't ask). No mount; no cache or vfs-cache - just sync - and it had a flurry of 403 errors three days ago. I had never seen them on that account before. Maybe something changed or happened Google end. Maybe that end, and vfs changes.

18

I didn't upgrade to 1.53 or use the new --vfs-cache-mode full until after these problems started. There really doesn't seem to be much of a difference in terms of requests. Sept 10th was the least amount of requests since I started having these issues (235K), but still close to double what I was typically getting before these issues started (120K). I'm still not running bazarr or ombi yet, but the rest of my stack is up.

19

do any of you guys service accounts? I think google is doing something with them

20

Mine was a standard, not service account.