Hi,
box.com, idrive, google they all have monthly download or upload bandwidth quotas / limits. We need a feature in rclone to respect that and to handle these correctly without errors.
I think this feature is needed by huundreds of people here.
Thanks
EDIT:
I talk from features like that, for a mount point.
It already exists.
max transfers or setting a bandwidth limit.
If there would be a different error msg, something like this for Gdrive could be implemented
Google drive
(--drive-stop-on-download-limit)
But currently I guess there is no “extra” error msg from Box
You receive only the 429 error
Documentations exist for a reason.
Here you can find most options. Some backends e. g. Drive have specific flags such as stop on upload/download limit in case the server responds with rate limit exceeded so that the transfers will be (fatally) stopped.
If you want something else maybe specify it more.
Guys what are you talking. This feature does not exist. There is only one option for google drive.
The option -max-transfer and --bwlimit does not fit the requirements!
IT IS NOT POSSIBLE TO SET BANDWITH quota limits for a MOUNT POINT on a monthly base. So please stop talking.
I talk from features like that, for a mount point.
This feature would be great and EXTREMELY useful, and I agree it's absolutely NOT the same as --max-transfer and --bwlimit because the only way to use these and using the full quota is basically doing a single transfer working 24/7, I think a solution would be a simple local file like quota.rclone or something like that which contains the quota info maybe managed also by rclone config (setting the quota up at time of remote creation for example) or even using the rclone.conf file itself to store the updated info.
What do you think @alneven and @M_R ?
@Animosity022 and @blackjack4494 We really appreciate your work on the forum a lot and again I come in peace but please can you try to partecipate to the discussions in a more open and positive way? I don't understand why lately you bash or worse censor some threads closing it, opened by people who just ask for features, the risk is just driving people away since most don't want to deal with thick rubber walls 
Thank you @dia3olik for understanding the problem. Most users here use RClone for mounting and there are no useful bandwidth settings for that.
I like your idea with the separate file and think that is the best solution from a developer's point of view if you don't want to store the data in a database. I would not save the consumption in the config. Only the settings, i.e. the limits, should be able to be set in the config.
And in a mount point it is easy to identify wich is upload and download traffic. So i think its not to hard to implement.
Again, on the mount just like other commands, you can use max-transfer, you can use bwlimit on the mount, you can use bw limit per file.
Feel free to implement it then and submit a PR to do what you need done.
I don't know what you mean.
You let it sound the answers were written in an unhelpful and maybe disrespectful manner which I cannot see at all. I even asked if he could specify the request. At no point did he mention mount in the starting post.
But as Animosity already said there is --max-transfer.
If you want it to reset monthly on Linux and the likes you just setup a cronjob to restart rclone each month and for Windows you can use scheduler.
If you still want some option to log the traffic stats in a file natively you would need to open an issue on github to request this feature or implement it yourself.
I read up on max-transfer, and it doesn't seem like the request here is possible with rclone itself. Rclone is session based, so from my understanding if the token were to reset in the middle of the month that would render max-transfer inaccurate for this use-case. bwlimit is still possible to use, although I doubt anyone wants to voluntarily limit their speed to the point where a 1GB file would take multiple hours to upload.
Therefore it seems there needs to be some sort of aggregate log of the total transfer amounts from each session that is then checked at a specified interval by some external script.
Not exactly. The max transfer is something you'd configure when you'd start the mount. You'd restart the mount when you wanted to 'reset' your quota. The keys or whatnot do not matter.
This is a particularly complex and difficult to implement as if you wanted it across everything, you have to write the stats somewhere across the board and keep the statistics somewhere that 'all' your stuff can access. How does a retry work if you retry a failed file? This is an extremely complex and edge case ask.
On Google as an example, you just stop uploading when you get the error and you reset the next day. Box seems to have a monthly limit per user or something.
Can it be done? Most definitely, but the complexity, requirements and number of folks wanting it don't exist as generally, if it was critical and useful for a large number of folks, it would get done. Sometimes things happen quickly because a developer/programmer want it and they do it.
The other way, sponsor it and pay for it to get done if it is that important to you and you can financially support it. I find all this stuff 'easy' to me since I don't program and it magically happens by others so I just consume it and everything is easy from my perspective 
So then therefore a full server reboot would reset the quota since it is restarting the mount, correct?
Just for terminology, there isn't anything in rclone about a quota per se.
The max-transfers tracks a mount/single command/single action so anything that restarts it, would reset the count as it's not stateful.
A reboot would reset it. Stopping it and restarting it would reset it.
Ok, thanks. That is what I mean by "it doesn't seem like the request here is possible with rclone itself." Granted it is possible, it's just not reliable.
I have looked over the docs, but I wouldn't say I'm fluent with them but I wonder if there is already a flag to limit output to a date and successful transfer size, to allow an external script to parse over it. That would eliminate the inherit unreliability of max-transfer. Just a thought.
There are for other providers.
Google Drive is popular and has a lot of us so someone implemented both directions to stop on download when you hit quota and stop on upload.
The OP is asking for a very complex stateful thing that counts and resets on a calendar. Odds are, that never happens unless it is sponsored or someone with development skills really want it as it's just really not in that much demand.
Having more of a 'stateful' cache for a mount is something I've been dying for as the cache backend had it, but when that hasn't come yet either and that has quite a lot of interest.
I'm not saying it's a bad idea or anything like that as it's a neat one, but the want and will to get it done is where it'll sit much like the bazillion backends everyone thinks are 'simple/easy' to add.
I wouldn't say it's very simple but it's not particularly complex. I don't know how RClone is structured, but the basic process would be.
Add 5 new options to rclone config
Feature is only available for MOUNT command.
That would be my approach to doing that. Perhaps we can think about that.
The most difficult part to implement is already done - traffic accounting used by --bwlimit flag. It means that we have number of bytes sent/received counters in place.
Also traffic cut off logic when limit is reached is available.
Stateful --bwlimit flag variation would require periodic writing of used limit to a file - also nothing new for rclone - for example rclone bisync stores two paths states between runs. It means that code where and how to write such file in different operating systems is also present and can be easily reused.
Rest is just some code needed to handle new flags and some logic glue. Definitely nothing complex - not like e.g. improved stateful VFS cache @Animosity022 mentioned.
But easy or not it still requires somebody to spent time to implement it. And indeed it will only happen when somebody needs it and is willing to do it.
Until it happens I would suggest very easy though a bit crude approach. Periodically write to log --bwlimit counter. It is literally few lines of code to make it working.
Having that I could run my rclone mount --bwlimit X command and calculate value of X based on sum of recorded bwlimit counters values (a bit of bash/grep/sed kung-fu on previous logs). Minus is that it implies mount restart at least once at the end of "billing" cycle. If it is monthly - should not be an issue though.
This is how I would do it if I needed it urgently.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.