I'm trying to start uploading ~600K files (350TB total) to an Enterprise Box.com account. Box.com Enterprise accounts are limited to 100K API calls per month. I have not purchased the Box account yet because I wanted to check if it was even viable for my use case first. If the API limiter caps me after checking the timestamps on 100K files, I'll have to find another solution.
Run the command 'rclone version' and share the full output of the command.
Which cloud storage system are you using? (eg Google Drive)
The command you were trying to run (eg rclone copy /tmp remote:tmp)
Yes, I've confirmed with my Box sales rep that they're fine with this amount of data. The source files are located on local storage on the same LAN as the host that will run rclone (storage is mounted on that host via SMB).
I'm not sure if Box supports --fast-list, I can check with them though (unless someone here happens to know). That does seem like a viable option though.
Interesting idea. What if that new file (or set of files) took 50 hours to upload and at the 24 hour mark, even more new files were created? I imagine Rclone would then miss those new files on its next run because they’re 26 hours old. Is that correct or am I missing something?
I’m going to follow the advice in your first post and make a trial account to see how things look. I’ve heard reports that normal Rclone operations might not count against your normal api usage so I want to experiment a bit. I’ll report back here for the benefit of future adventurers.
It appears that normal rclone operations do not count against the monthly API limit. I created a 14 day trial account, uploaded a few thousand files, and ran a Platform Activity Report to check API call usage and the report came back empty indicating no API calls had been made.
It seems that limit is only for custom applications that need a specific API key to work. Not sure I totally understand the difference between that and what rclone is doing, but hey, problem solved!
I've been running my upload job on a Box Business Plus eval plan (50K API calls per month). The admin console indicates that rclone has nearly 70K "activities" and the upload is still running with no errors:
This is why I was worried about even trying Box.com, I looked at my API calls for my rclone Google Projects and they average 6-7 million calls/month. I have about the same amount of data as you, moved ~228TB to Dropbox so far.
The other thing about Box.com is they have that 5GB max file size limit. Are you getting around that with rclone's new feature to break apart the files? Last I knew, that feature was still in beta, I'm interested in knowing if it works well for streaming 4k files.