Yea, but doesn't that risk the file being lost in the cache?
-move operation from OS to cache
-file in cache queued for upload
-error from rclone (which the user can very easily miss)
=> file never gets uploaded, and depending on how the VFS tracks "files that should be uploaded but never did" then it either stays indefinitely until manual purge, or is cleaned out when VFS cache size or time limits are reached.
If these are the only two options then an error is better than nothing, but I'm not sure if jamming up the drive or risking losing the file entirely is worse. Both aren't great outcomes.
BTW, I haven't had the energy to perform the confirmation test to make certain this is issue is correct yet (dealing with some health issues currently). I think it is, but I don't like asserting that as true based on a single unexpected event. Needs repeating to be sure. Maybe we ought to put this on the backburner until I get that done so you don't end up doing unnecessary changes. I should get it done in a few days. I can ping you about it when I have conclusive results.
Per the linked chat, it was saying if you created a file that's greater than 100GB, it would restrict you from creating another file that size until the next reset.
That chat was from 2017 as well so it's a bit dated so it's always hard to tell if that is still a thing or something has changed.
Animosity is right in pointing out that larger sizes can be created, but (according to that source) new uploads after that in the same quota-timeframe will fail. This seemed pretty much what happened to me.
Don't take this as verified until I actually do verify it. So far it only happened once and the hypothesis came after the event. That's pretty different to testing the hypothesis
I will run a test on this tomorrow if all goes well and report the findings here.
Just for reference, did you test with Gdrive or Teamdrive?
I wouldn't expect different behavior from them, but should we end up getting different results then it might be worth noting.
Ok, I've done some more testing and troubleshooting and I'm sure I finally solved the issue.
Firstly, I've confirmed that an 100GB file "ban" does not seem to be a thing. Either that info was inaccurate, or out of date. My test: uploaded 2x 125GB files - waited until upload finished - then uploaded 100x 10MB files and 2x 1GB files. No problems uploading, and drive responds normally afterwards. With both me and Animosity getting the same result here I think we can put this hypothesis to bed as debunked.
So what was the problem? Well - it was ironically exactly what the error said, an API rate-limit.
The thing is, I had separate Oauth client IDs on several unrelated disks, so I was sure this was not an issue to even look into... but there was something I didn't realize until now: Oauth client IDs created inside of the same project apparently share quota - or at least share some form of burst quota. The solution was simply to recreate Oauth client IDs, each in their separate projects. I did this one by one with stress-testing in between and it was really obvious that the separate-project IDs had way more API quota at their disposal. Not only did they finish heavy list operations much much faster, but unlike the other drives they didn't get a lot of 403 rate-limit errors in their logs. As I moved more drives to separate-project IDs the pressure lessened on the remainder and I saw them speed up and get less errors.
Now, maybe this is already known to some of you, but this is news to me, and I never came across this information in either the docs or many hours absorbing really long threads specifically about 403 errors. I think it would be well worth putting this (and some of the other confirmed limitations) in the docs. It only applies to people with multiple google drives of course, but I doubt I am the only one. @ncw I can make a pull request and make those changes if you'd like.
Right now I am at a point where I have made a fully multithreaded sync and maintainance script that operates on all drives at once, and this runs great now - compared to before where I was having severe issues even running these tasks sequentially.
Finally, I made an observation about the Gdrive pacer. It seems like it may potentially be vulnerable to getting "deadlocked" (in a not strictly programming sense) in situations where it starts and API-intensive task after the API quota has already been heavily taxed. In my case it resulted in syncs that never seemed to progress at all. I presume this may be because API calls that result in 403 error still count against the quota (probably on purpose by google as it rewards designing software that doesn't hammer the API). In these cases it seems like it maybe doesn't manage to throttle sufficiently to dig itself out of the hole and you end up with 99% of requests resulting in 403 errors. This is what I misinterpreted as maybe being a >100GB temp ban. I have not done extensive testing to find the exact limits, but I speculate that this could happen in spesific circumstances even on a single drive if you execute several API-intensive tasks in rapid sequence from a script - because the pacer throttling is not persistent across commands in this way. Possibly the pacer could be improved to be allowed to throttle even harder if it gets this badly rate-limited (haven't looked at the code for that yet).
Hope this was useful info for someone out there It was certainly worth the troubleshooting time for me to get to the bottom of this.
rclone exponentially backs off it's API hits if you get errors.
The quotas are documented on the console:
In general, it was tuned to 10 per second as that's the default quota limit per user. You can request more quota per se and some folks have gotten it and some not.
In general, if you are not trying to abuse the system they have, the defaults and such in rclone will not result in 403 rate limiting. Some 403s are good as it means are trying to push for maximum performance as well.
Yea, I was aware of those quota limits. I had just assumed that multiple Client IDs inside of the same project counted as separate "users" as that seemed intuitive - but that is apparently not the case. The Client IDs have to be from separate projects to not interfere with each other.
The default quotas don't seem overly restrictive for a single drive, but when it gets shared over multiple it became a problem.
I think this is a new development. client IDs used to have their own quota, now it is the project that has the quota. I'm not sure when that changed, but it became very clear when I recently asked google for more quota (which they granted!).
Yes please, I'd love some more docs.
I think you are right about this. After you've had some sort of ban, the pacer will retry the 403 errors increasing the timeout (as per the google spec) until it is 16s per request.
What I'd really like to be able to do is tell the "normal" 403 message, the back off a bit mate messages from the you've been banned messages.
It would be really helpful to have some examples of each captured with -vv --dump responses - maybe there is something I can tell apart in the messages.
Here is what the response codes look like for the built in client ID, so pretty much all 200 or 403. I was wondering if those responses are ever 429 (I'm pretty sure I might have seen such).
I did a quick test with --files-from --no-traverse which drive seems to hate to generate some messages.
That quite quickly gives these messages (I pretty printed the JSON). I'd love to see more examples especially from when one of the quotas has been exceeded.
2019/08/08 10:24:39 DEBUG : HTTP RESPONSE (req 0xc000579700)
2019/08/08 10:24:39 DEBUG : HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Alt-Svc: quic=":443"; ma=2592000; v="46,43,39"
Cache-Control: private, max-age=0
Content-Type: application/json; charset=UTF-8
Date: Thu, 08 Aug 2019 09:24:39 GMT
Expires: Thu, 08 Aug 2019 09:24:39 GMT
Server: GSE
Vary: Origin
Vary: X-Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block
3c8
{
"error": {
"errors": [
{
"domain": "usageLimits",
"reason": "userRateLimitExceeded",
"message": "User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=XXX",
"extendedHelp": "https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=XXX"
}
],
"code": 403,
"message": "User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=XXX"
}
}
