I/O Error when I am reading from drives after 10TB blown

Help and Support
1

What is the problem you are having with rclone?

I/O error when trying to read files.

Blew my 10TB read limit from drives. Google console showing this:

I am not sure what other quota to check for but so far seeing those two led to me believing it has been reset.

I even made a new oauth secret and then gotten a new access token.

When I do

dd if=file of=/dev/null

the result is I/O error, or input output error.

Even with refreshed access token I still cannot download/read from drives. (speed issue from last time has been solved, I have been reading at 200MiB/s with my app and that is why it was blown.)

What other graphs / ways do I have to check to see if my quotas have been reset? Or is the 10TB for the month's time instead of the day's?

Journalctl shows this, might now be the exact error but should be close enough:

open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded

iirc, there was one that says --low-level-retry or smth along those lines.

What is your rclone version (output from rclone version)

- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.4.0-77-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.16.5
- go/linking: static
- go/tags: none

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 20.04

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

#!/bin/bash

rclone mount gd-3: /mnt/point \
--read-only \
--allow-other \
--umask 002 \
--poll-interval 10s \
--no-modtime \
--transfers 10

The rclone config contents with secrets removed.

[main]
type = drive
client_id = xxx
client_secret = xxx
scope = drive
token = {"access_token":"xxx>
team_drive = xxx
root_folder_id =

A log from the command with the -vv flag

Sorry no logs as shown above
2

You can't check quotas for upload or download anywhere.

You wait until it resets.

Those errors are expected if you can't download the file.

3

Oh lol so none of us know when google decides to reset the api?

Great, so I think what I will do is this:

  1. limit read rate after reset, so it doesn't exceed the 10TB
  2. pause the file sharing apps I am running until a dd test shows regular reading.

How does that sound to you?

4

It's not published.

I really don't know what your workflow is doing to download 10TB in a day so I can't offer much assistance.

5

Yeah that sounds fair.

So every oauth secret, unlike service accounts share one quota?

Since I did delete the old one and use a new one, tho within the same project.

6

Quotas are per user.

I'm not aware of / nor would I recommend anyway to try to get around it or abuse Google's quotas.

2 Likes
7

So here is my conclusion for those in the same situation:

  1. wiki for setting up SA is a bit outdated, you have to google when the steps get confusing
  2. SA uses its own oauth and each SA has its own quota
  3. oauth's quota is per project based, so no matter how many oauth you have within a project, they share the same quota. SA are different tho.
8

Feel free to make any updates if you see fit. Beauty of open source if you find something.

You are confusing API quota and upload/download quotas.

Confusing API quota and upload/download quota.

1 Like
9

Regarding the upload/download quota resets:

The timer starts with your first upload within the 24 hour time frame. So, let's say you start uploading something at 3:20 pm on day 1. Then your last upload within the 750GB limit can be at 3:19 pm on day 2. At 3:20 pm on day 2, the quota will reset, and you will be able to upload again. If you don't upload 750GB within 24 hours, or you start uploading later after a reset, then the next reset will be pushed to that later time the next day. I can only assume the same applies to downloads.

TLDR: There is no fixed reset time.

Hope that all makes sense :laughing:

1 Like
10

That makes a lot of sense, as I was just thinking that possibility too!

So assuming that is how uploads/writing to the drives work, is it also safe to assume downloads are also the same?

Apparently it seems like this isn't a widely documented knowledge. My initial guess was midnight EST, that is why I came here like why ain't my quota reset yet! :upside_down_face:

11

There's more than just a download quota as there are per file quotas as well as that also depends on a regular drive or a team drive and if the file is shared and accessed via that.

There isn't any sure fire way to figure out exactly when it resets because it isn't published.

There are a lot of correlations on when and how.

So best bet is when you can download again and it's working, it has reset.

2 Likes
12

You're absolutely right. I tend to forget about team drives, as I don't use them. I also don't download from GDrive :stuck_out_tongue: (outside of streaming, of course)

13

Indeed, just waiting it out is a sure way.

14

hi, can I ask where exactly do you see this output from your screenshot?

15

iirc,

systemctl status rclone
16

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.