[Is it possible to automatticly kill rclone once rate limit as reached?(Gdrive)]

Hey i was wondering if it is possible to automaticly kill/ stop clone once the 403 rate limit has reached?
Or is it only possible via --max-transfer ?

As far as I know there does not exist any "rate-limit detection" mechanism yet (although one should be possible in theory and would be neat)

--max-transfer is one options.
--bwlimit 8700k (about 8,5Mbit/sec assuming 750GB/day quota)

That kind of bwlimit would never go past 750GB at any time - but it also assumes you are not in a hurry...

I know that a timeout flag is also in the works, but I think that's a blind time-based kill of the instance.

1 Like

No, it really is not possible as unfortunately, the same message is used for a normal rate limit and when you reach your upload quota.

I use max transfer and only do a single upload each night.

It seems that way, which is a shame since it removes the possibility of a really clean and easy detection.
That said, I still think it would be fairly easily doable.

All you would really need is a small monitoring script running internally that would keep an eye on the transfers.

It would have a simple timeout value and look at the return codes for all operations that might return a 403 due to hitting the max upload quota. If a period is detected (say 120sec for illustration) in which 100% of the quota-affected commands are 403 then we can fairly assume that a max quota has been hit and we are now in a full stall. rclone can then exit to allow the next script to run or do some other appropriate action like notify admin (email?).

Assuming that the timeout period is not trivially small and that there isn't some other unusual service-interruption at work then it should be very unlikely for just API-limit throttling to trigger this while it should pick up that you hit the quota wall fairly fast. This is essentially the same way you or I would differentiate the two problems when looking at a log-file. API limit gives 403 intermittently while max upload quota is a hard wall of nothing but 403's.

This is one of those many ideas I've had rattling around in the brainbox for a while but haven't had the time to make an issue for yet. @Cubuss Feel free to "steal" this and make an issue for it on your own if you want to encourage development of such a feature. It's not a terribly complex idea, so if you just copypaste the explanation here to go with your use-case description then that should be most of the work done already...

1 Like

rclone already exponentially starts to back off when it hits 403s via the pacer settings.

It's already very easy to mitigate the issue by using a bandwidth limit / max transfers.

Trying to watch for 403s is just problematic and not a very viable programatic solution.

It's probably better to keep something that tallies up the upload and resets the limit at 00:00 UTC when the GDrive limits tend to reset. The challenge is these are not published times and such so it's a bit of a guessing game.

I'm not sure I see how the pacer factors into this. Yes, it throttles down, but as long as the timeout duration is far longer than the longest possible pacer level (which 2 minutes would be by far compared to the 16 or so seconds I think is the current pacer roof) then the pacer shouldn't be capable of creating any false positives.

It's not a perfect hard-data solution, but that just isn't open to us at the moment, and I think it would be relatively accurate given reasonable parameters.

I don't feel like --max-transfer and --bwlimit fully solve the issue. They can help - but they are only really useful under very spesific situations and under spesific limitations. More than good enough for your daily-upload type usecase for sure, but I don't necessarily think everyone has dedicated server systems for their storage to have that luxury of time.

Logging the transfers would certainly be a viable option too - and one I've considered also. I would just disregard time completely and make it a running time-frame. Use timestamps and tally up the score from the last 24hrs at intervals of X. It won't matter when the actual reset happens as long as the system makes sure that you never pass 750GB (or whatever limit) in any given 24-hour interval. If you hit the limit you can pause for a bit, exit with an error or do some other appropriate action.

Should work. Assuming you could get the correct data you could actually eliminate the guesswork properly. That said - getting the correct data may not be as trivial as it seems...
You need to start factoring in other mounts connecting to the same drive/user adding up in the same pool. That starts to get complicated if we are talking about access from different systems. Then you need persistence for it to cover stops and starts.
Seems like a lot of work compared to looking at the return errors.

Can you share a situation where setting the bwlimit would not work with capping the 750GB upload limit per day?

Any use-case involving having a lot of bandwidth, but no dedicated home-server to handle uploads leisurely over time. In those cases it would obviously be far more optimal to get it done in the minimal amount of time and be done with it so you don't have to leave your workstation or gaming-rig going all night just to keep up with some large uploads.

It would will "work" in terms of not failing, but it would be far from ideal. It's only ideal if time is of no issue (like if you have a 24/7 server that can just do this in the background for you). Me and you have that, but I don't think most average users do.

Also, any situation in which it's not easy or feasible to let a single instance do all the uploading for you. You can work around that to a large extent by having scripts use the RC as their common point of entry, but that's added complexity and probably infeasible in some setups.

if the daily limit is 750GB, it doesn't matter what you have as you can only upload that much. If I can only upload 750GB per day, doing it in 30 minutes or 5 hours doesn't really matter.

Do you have a scenario that makes it not work?

Well there is our disagreement.
I think that (time) matters and I think many others would tend to agree, but you don't. That's fair enough, and I can't say your opinion is "wrong". I just disagree.

But no, it would never outright fail (as long as you can funnel the traffic through a single instance), which I didn't mean to imply to be clear. It's a pretty workable solution for any 24/7 system.

24 hours is fixed for a reset and regardless if you upload in 5 minutes or 23 hours, you wait 24 hours.

So why does time matter for the issue?

Let's say I only have a single PC and I'm at work until late in the evening.
I have 500GB to upload for the day on a gigabit connection, but only 2 hours until I need to go to bed.

Don't you agree it's a (potentially quite big) benefit to be able to do that task in 30 minutes rather than leave the system running overnight - or even worse - 24/7 - just to be able to use your full upload quota? It's bothersome, wasteful and not technically required to achieve the goal.

In that scenario, wouldn't you just upload and turn the PC off when done? If my goal was a PC, I'd probably just drag and drop in the Drive Interface and avoid the upload quota all together.

Well that would work IF you were sure that you didn't already use some of the quota from this 24hr period, which would be really hard to track manually... that has to be considered an unknown.

And what if it's 900GB instead?
What if you want to have this run in a script where after the upload-sync is done, another script runs to sync that data to a backup-volume - or any other script task, like deduping or other maintenance.
That won't work because with no control the script just indefinitely stalls and you have to manually check it - or if you use --bwlimit as the control then it will take 24hrs to do.

You could use --max-limit 750GB and hope that you have your full quota to work with, but since nobody knows when exactly that resets it's impossible to say unless you used nothing for the previous 24hrs. If you overshoot the script stalls. If you undershoot you aren't as efficient as you can be.

In your scenario you described, I'd upload and turn my PC off before I went to bed if it failed or not.

Most scenario reflect around folks running servers/plex/some media install and have something on so bwlimit works for those scenarios.

If you have a PC you turn off and off, hitting a quota really doesn't matter as you have interaction with the machine so I'm trying to get a scenario that makes some form of quota tracking / useful valuable.

Yea but since it would stall all the scripts after it now also stalled - and you have no choice but to babysit the system and adapt to it's needs rather than the other way around (isn't that what computers are for?).

I'd much rather have a single script that can reliably do everything as instructed with no extra interactions - but also not artificially performance/time restricted. I have made exactly that - scripts that perform full sync and maintenance of all systems (be it rclone-related or otherwise) that I can run and forget about, but the only thing that isn't /can't be properly handled currently is quota-based stalling. Because there is no feedback/reaction from rclone you can't program the wanted the result. I can't imagine I'm the only one that finds this unnecessarily restrictive.

I've since set up a fairly complex system that works for my needs, but I think that's too optimistic to realistically expect from the average user - and I do think it could potentially be as easy as an optional switch.

I think we're rambled enough on this now and made our points clear (and derailed the thread sufficiently, as usual :wink: ). Let's draw a line here and agree to disagree unless you have any important last comment to make.

It's not about agree or disagreeing as I'm trying to understand a use case for the switch which is why I'm asking questions and so far, there isn't really a good one.

In your case you've given, you'd just run and turn the PC off, which I really don't think most folks are doing these days. The goal being to flesh out a scenario and application for the switch.

In short - any time you want to run upload scripted with minimal manual input, and time is at least somewhat a factor.

Any upload that isn't limited to 24hrs in bandwidth will have an unpredictable result and may never halt, depending on largely unknown circumstances relating to the unknowns around the quota - thus jamming up the entire script-chain.

I don't feel that constantly babysitting the system in case it halts is an adequate solution, nor do I think resorting to artificial BWlimiting is ideal as that's not really targeting the root of the problem and wastes an inordinate amount of time if you have a broad upload pipe.

I don't think I am able to put it more succinctly than that, and the limitation is fairly obvious at least in my mind. I don't think it is coincidental that we see this same question repeating in the forum. I accept that you have a different viewpoint, but I feel like may be not appreciating usecases outside your own - however typical that might be amongst the forum veterans here in our little community.

I think I'm gonna let this rest for now at least - because I feel like I'm just repeating myself, and that's not productive for either of us.

Not really obvious or I wouldn't be asking as I'm trying to understand.

If I wasn't appreciating use cases outside of my own, why am I bothering to ask?

Google poorly choose to have the same error for a rate limit and the upload limit so it's not an easy 1 + 2 code to exit out once the limit is reached.

With that being said, I'm trying to understand the use case and trying to help figure out different solutions as they can be documented and shared for other people. As for me, it doesn't apply but others could find those solutions viable and helpful to share so let's not assume intent and just ask me.

Same for me, I have my own system for now that adequately serve my spesific needs, but that's a custom fit an complex setup that just circumvents the problem rather than dealing with it basically. I don't think that is a good solution for an average user (and I wouldn't mind it not being a limitation in my own setup design either honestly).

I'm not assuming intent, and meant no offense. I tried to word that as carefully as I could, but nuance can be difficult in the written word. If you felt that was a stab at you, I can assure you that was not my aim, and I apologize if I accidentally offended.

I just feel I have detailed the problems several times by now, and from my viewpoint I don't see that your responses adequately solves them - each tool we have at our disposal currently has it's own set of significant problems and that is kind of the point. As I don't really have anything more to add I'm not sure what else to do except rehashing what I already said, which I don't feel is productive. Obvious from my standpoint doesn't need to equal obvious for everyone. That's fine - but I seem unable to bridge that between you and me - and concede that could be due to my explanations rather than your understanding of them.

If anything I said was not clear then I am happy to elaborate on something spesific. Otherwise I do not know how to proceed.