Gdrive limits you to 750GB a day. If one has a lot to upload (so will be multiple ~750GB chunk a day, one might want to upload many uploads in parallel (say becuse their single transfer upload speed isn't great, but 20-30 parallel uploads will saturate the connection).
now what happens if I set a 750GB (I actually haven't tested this recently, but going by memory), it kills the connection when 750GB total have been transfered, even if 20-30 are in progress (i.e. the 750GB includes their data). this is bad, as I just wasted time uploading that data.
The way gdrive works (From observation) is that it places quota checks at 2 points. at file start and at file completion. a file only counts to one's quota once its completed, therefore if one has (For instance) 740GB of files to upload, one can add a 50-100GB file to upload in parallel with them. It will start in a safe area (quota hasn't been hit) and will end in a safe area (until it completes, only 740GB could have been completed as well).
therefore, it be nice to have a "limit" that says, stop "adding" active transfers after I cross that 750GB mark (i.e. it might be 800GB total, as I described, but once I add a file that will take me past 750GB total transferred, don't start any more new transfers from the queue).
does this make sense? am I missing something that enables that already am I misunderstanding the current limit option?
its not quite what I want (or I think last I tested). lets give an example.
imagine I have 30 files to transfer, 10 are 20GB and 20 are 40GB.
if I do a order by sized mixed with transfers = 20, what would be the ideal scenario?
it picks 10 of the 40GB and 10 of the 20GB. after the 10 of the 20GB are done, about 400GB total would hve been transfered (assuming all the transfers are proceeding about the same rate). that means, it would think it can add all the rest of 40GB files (at least my experience, as it did it based on the amount currently transferred, not what it will have transferred if all the in process one's complete). and we know it can't complete all 10 as it will hit a quota error.
What I am arguing for is a mode that says only 4 of the rest of the 40GB files should be able to be transferred as then I'll have transferred 760GB total (4014+1020)
is that a clear example? perhaps there is a way to do this, but last I checked, soft wasn't it.
I think that is what the soft mode is supposed to do, more-or-less.
--cutoff-mode=hard|soft|cautious
This modifies the behavior of --max-transfer Defaults to --cutoff-mode=hard.
Specifying --cutoff-mode=hard will stop transferring immediately when Rclone reaches the limit.
Specifying --cutoff-mode=soft will stop starting new transfers when Rclone reaches the limit.
Specifying --cutoff-mode=cautious will try to prevent Rclone from reaching the limit.
So rclone should queue transfers until the actual transferred is > your limit. The existing transfers won't be stopped.
That sounds quite clever - a bit more clever than what rclone does right now.
The order that the files are sent in is completely specified by the --order-by mode and this doesn't interact with the --cutoff-mode. So all the cutoff-mode can do is look at the file due to be sent and a) send it or b) decide the cutoff has been reached and send no more.
what I am saying is something like this. Every files added to "active transfer" gets added to a running total. Once that total goes over the provided limit, rclone stops adding files to active transfers.
its not that clever, I think? i.e. we dont kill the processes, we just let it run its course.