I think that's more expensive. List type hits are basically infinite (billions allowed) but each open of a file counts towards download quota whether the file was fully downloaded or not. Say you do an ffprobe on a file which only requires you to grab the beginning portion of it, read headers and close it. To Google, you downloaded that full file even though you only really opened it, read a few bytes and closed it.
Mergerfs came up as a possible viable workaround to the fact that rclone can't re-auth a mount while it's mounted but I don't think there's a need to overcomplicate things meaning that I believe it's probably better to log this as an enhancement request to rclone to handle this and that'd require no changes on the mergerfs side and would reduce complexity. Note that I have not run into the errors described in the github issue with normal use, only with my attempts to work around the original issue by including multiple remotes and having mergerfs randomize reads from them.
That's not right as that's exactly why HTTP range requests were implemented a few years back as you don't use a range request, it does count it as a full download. I've done thousands of ffprobe on 100GB in a few minutes to invalidate that.
There's no debating that Drive supports range requests but if you go look at the audit logs you will see any hit on a file just says the file was downloaded, there are no mentions of partial downloads anywhere.
I've hit max download quota with plex scanning without deep analysis enabled while downloading at about 200 Mbps which is literally impossible since you'd need to have sustained gigabit for 24 hours to hit the 10 TB cap. That either means that they count partial hits as full downloads or the quota window is not 24h (which is known not to be the case since the reset time shows up in everyone's developer console and is always the same).
Here is the Google Bug that shows partial downloads was an issue was fixed back in 2016 and supports my statement that partial downloads work and are counted properly per Google.
The 10TB down and 750GB up per user are the known quotas that they publish.
They are all the 'unpublished' quotas per file and it gets dicer with Shared drives/Team Drives and Shared with me content as those seem to have stricter downloads per file.
In a quick example, here is me running ffprobe 200 times on an 80GB file in a loop, which would equal 16TB if it all counted.
It shows up as single line items per ffprobe in the audit logs for your GSuite:
It seems to me that the out of band changing of branches is probably the best solution if there isn't a practical way for rclone to error in a way that is more compatible with mergerfs' way of working.
As I mentioned I could create do a few things to manage this. The most basic would be to add an open call to a policy as part of its filtering. That's not ideal but may work.
Alternatively, I could in effect create a loop where it will run a policy as it does now and if the open fails it would remove that branch from the list of possible branches and try again. Not hard to do but not pretty. If I go down that route though I should probably audit every function to see if they need something similar. create, mkdir, mknod, etc. A "HA" mode of sorts.
To do it optimally though might be tricky as it'd require rewriting all policies to return sorted lists rather than the individual entries so it wouldn't need to recalculate.
By failing stat you mean that rlcone would continue to return the files in directory listings but if you ran stat() on them it would return an error...
I guess that would be possible.
The main problem is that rclone doesn't really know when you are out of quota or not. Google doesn't return a definitive error saying you are out of quota it just tells you to slow down indefinitely. Also rclone doesn't know when you will be back in quota either...
The google quotas aren't documented either so it is all a bit hit and miss.
We recently discovered a way of telling one of the quota erorrs apart form the others and implemented a --drive-stop-on-upload-limit which causes rclone to quit if it gets that error. That is an upload quota message so it isn't quite the right thing.
Rclone goes to some effort not to actually open the cloud storage file, so if you just open the file and close it again, you won't get an error! If you were to read data from each file then that would be really expensive as it will be doing HTTPS round-trips etc...
Yes, I mean that stat would return some error. EIO or similar.
Ugh. OK. Welp that's out then.
Seems like the out of band reconfiguration of mergerfs might be the best then. Unless just guessing that a "slow down" error is a quota error and allowing people to set some timeout. It'd be super clunky but it might work. But then it's not much different from the out of band method.
I'll just note that @darthShadow is working on the out of band reconfiguration of rclone to use a different service account file which would fix the problem from a different direction.
Just to add my tests I have been doing. I suffer from this issue everyday from 8 AM to 12 AM so I tried 1 servers with --union-search-policy=eprand and 1 with --union-search-policy=epff and both servers still had the same issue.
Without being able to tell what remote was being used for each try to open the file, I can't be sure they are working.
It's also possible that the errors still happened because not all my servers were using a random policy...or It's also possible that the errors happened but if users tried to kept opening the file it would eventually succeed because of the random search.
Regardless... Tonight I plan on switching all servers to eprand so in theory, I shouldn't have 403 errors anymore because the hits will be distributed across 3 different remotes
@random404 testing with a union and search policy eprand (from latest beta) appears to be showing promising results at the obvious overhead of requiring multiple remotes to be joined but that should only be a one time setup.
hmm maybe I was wrong, took a look at the code and only see set and get commands added so i'm guessing this doesn't reauth so it'll still be using the old SA account when mounted until the remote is remounted right?
