monthly API limits

What is the problem you are having with rclone?

I'm trying to start uploading ~600K files (350TB total) to an Enterprise account. Enterprise accounts are limited to 100K API calls per month. I have not purchased the Box account yet because I wanted to check if it was even viable for my use case first. If the API limiter caps me after checking the timestamps on 100K files, I'll have to find another solution.

Run the command 'rclone version' and share the full output of the command.


Which cloud storage system are you using? (eg Google Drive)


The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync

that is a large amount of data, i would assume box would work with you on that?

where are the source files located?

can run rclone ls > file.lst one-time on the source and then
rclone copy ./source box: --no-check-dest --retries=1

if box supports --fast-list, such as many providers, one api call would return the details on 1,000 files, so 600 api calls.

i would get a free/cheap account and test on that, monitor the api calls.

Yes, I've confirmed with my Box sales rep that they're fine with this amount of data. The source files are located on local storage on the same LAN as the host that will run rclone (storage is mounted on that host via SMB).

I'm not sure if Box supports --fast-list, I can check with them though (unless someone here happens to know). That does seem like a viable option though.

looks like rclone on box, does not support ListR and thus does not support --fast-list

i could be wrong, maybe some-else knows better.

you plan to upload 350TiB within 30days at approx. 130MiB/s?
else if the upload was spread over 30+ days, then 200,000 api calls.

Right but once everything is uploaded and Rclone runs checks to see if there are updates, it’ll hit the API limit after checking 1/6th of my files.

well, there are multiple workarounds for that, such as filters.

let's take this simplified example.
--- the initial transfer has completed.
--- rclone is run once a day.
--- within the last 24 hours, a single source file is modified.

in that case, only one source file needs to be compared and copied to the dest, at the cost of a few api calls.

rclone copy ./source box: --max-age=24h
with that command, rclone will list the source, find files modified within the last 24 hours and check+copy only those modified files against

rclone copy ./source box: --max-age=24h --no-check-dest --retries=1
with that command, reduce api calls further by not checking dest, simply copy the file

Interesting idea. What if that new file (or set of files) took 50 hours to upload and at the 24 hour mark, even more new files were created? I imagine Rclone would then miss those new files on its next run because they’re 26 hours old. Is that correct or am I missing something?

well, rclone would be missing something ;wink.

you would need to adjust the time period based on your specific use case and total api usage.
perhaps --max-age=72h would work.
and there are some possible edge cases that rclone might miss.

for that, need to run a full rclone sync

I’m going to follow the advice in your first post and make a trial account to see how things look. I’ve heard reports that normal Rclone operations might not count against your normal api usage so I want to experiment a bit. I’ll report back here for the benefit of future adventurers.

Thanks for all of the help on this!

1 Like

It appears that normal rclone operations do not count against the monthly API limit. I created a 14 day trial account, uploaded a few thousand files, and ran a Platform Activity Report to check API call usage and the report came back empty indicating no API calls had been made.

It seems that limit is only for custom applications that need a specific API key to work. Not sure I totally understand the difference between that and what rclone is doing, but hey, problem solved!

1 Like

in my experience, such statistics are not real-time but aggregated over a time period.
and based on the link you shared, that seems to be the case with box.
so i would wait 24+ hours and check again.

Just ran another report, still empty, things are looking good :slight_smile:

I am the same as you. I also get a blank report, but I don't trust it. I think it is impossible that with what I have uploaded, no API calls have been made.

I've been running my upload job on a Box Business Plus eval plan (50K API calls per month). The admin console indicates that rclone has nearly 70K "activities" and the upload is still running with no errors:

1 Like

I kinda worry that they'll enforce the limit anytime later seeing massive data growth

This is why I was worried about even trying, I looked at my API calls for my rclone Google Projects and they average 6-7 million calls/month. I have about the same amount of data as you, moved ~228TB to Dropbox so far.

The other thing about is they have that 5GB max file size limit. Are you getting around that with rclone's new feature to break apart the files? Last I knew, that feature was still in beta, I'm interested in knowing if it works well for streaming 4k files.

According to the reply from this forum user, if Box exceeds the number of API calls, it will not stop, but there may be additional fees. Have you encountered this situation?

I'm interested in Box as well, but noticed the low API and the factor of the file size limits. Any update on your adventures with them so far?

So for should I put my chunker inside my crypt? or my crypt inside my chunker?

both will work. but IMO:


is more logical

the reason being is to encrypt everything possible in remotes' chain.

when you use;


I do not think you gain anything but you leak information about chunking

1 Like