We are currently uploading huge data from our fileservers,to BOX.com using rclone.
We have around 20 sites, spread over the world, that are sending data through rclone.
However, despite having data sent from multiple locations, it seems that we are still capped to around 350k/380k uploads per day (refering to BOX.com activity graphs), no matter how many different “senders” are uploading data at the same time.
Please note that all rclone instances are linked to the same box user account.
Thank you in advance if you have any clues on this issue!
Limiting transfer is quite common for cloud providers in general and it sounds like you’ve hit one of those limits. They may be able to raise it for you, especially if you are a paying customer!
Thank you for your answer.
A ticket has already been opened at Box support, they asked me to open a thread here in the meantime.
Updates will be posted here
Thus if you connect that many sites to one account most of your api calls will bounce with a 429 Too Many Requests.
In order to get around this you should utilize a JWT authentication and work via service accounts that act “as-user”. This way each site will count as it’s own user, thus multiplying the rate limit by the amount of sites, users.
@Nick Now I don’t know two things here: Is rclone able to do a Json Web Token Authentication, and if, does it support our “As-User” header?
I’ve seen this, not exactly. JWT is basically signed packages. This works passwordless, you use a private key to sign a claim that is then send via a post request.
It is very popular as it does not need manual password entries, this site contains also a link to a go lib.
From the library example code:
// Create a new token object, specifying signing method and the claims
// you would like it to contain.
token := jwt.NewWithClaims(jwt.SigningMethodHS256, jwt.MapClaims{
"foo": "bar",
"nbf": time.Date(2015, 10, 10, 12, 0, 0, 0, time.UTC).Unix(),
})
// Sign and get the complete encoded token as a string using the secret
tokenString, err := token.SignedString(hmacSampleSecret)
fmt.Println(tokenString, err)
It’s actually easier to use than OAuth2, as you can get a json file from box which contains all the necessary parts, which you can easily parse. The only difference to OAuth2 is there is no refresh token, you just send another claim to get a new access token. This is why it works with multiple clients as the token only expires by explicit invalidation or expiration.
For the as-user header a command line parameter that takes a json file with custom headers would be the easiest option enabling not only box but other cloud provider extensions.
I have no experience in go, but have done jwt already, with a library it is really easy.
Sure, yes you can. If you mail me your login, I can check if any resources are missing and supply them.
I recently had written an example in python here:
and a colleague of mine has written some examples in node:
A lot of cloud providers do support JWT so this will have a larger benefit.
I’d also like to know what your opinion is on adding a custom header parameter. This way a lot of additional features could be supported, e.g. the as-user header with minimal coding.
Zapier has a good article when to use what kind of authentication:
In general OAuth2 is a great way for humans to authenticate, but if you want passwordless access JWT is a great extension to OAuth2.
Simplified you could say it is working similar to an ssh key authentication, with you holding the private key and the other side holding the public key.
