BOX.com - Capped upload?


#1

Hi there,

We are currently uploading huge data from our fileservers,to BOX.com using rclone.

We have around 20 sites, spread over the world, that are sending data through rclone.

However, despite having data sent from multiple locations, it seems that we are still capped to around 350k/380k uploads per day (refering to BOX.com activity graphs), no matter how many different “senders” are uploading data at the same time.

Please note that all rclone instances are linked to the same box user account.

Thank you in advance if you have any clues on this issue!


#2

I suspect this is one for box support.

Limiting transfer is quite common for cloud providers in general and it sounds like you’ve hit one of those limits. They may be able to raise it for you, especially if you are a paying customer!


#3

Hi ncw,

Thank you for your answer.
A ticket has already been opened at Box support, they asked me to open a thread here in the meantime.
Updates will be posted here


#4

Hi,

this is Johannes from Box support. I’ve asked to raise this question as it might be of general interest on why you are seeing this behavior.

Box has a rate limit of 10 API calls per second per user and a transfer rate limit of 4 files per second per user. https://developer.box.com/v2.0/reference#rate-limiting

Thus if you connect that many sites to one account most of your api calls will bounce with a 429 Too Many Requests.

In order to get around this you should utilize a JWT authentication and work via service accounts that act “as-user”. This way each site will count as it’s own user, thus multiplying the rate limit by the amount of sites, users.

@Nick Now I don’t know two things here: Is rclone able to do a Json Web Token Authentication, and if, does it support our “As-User” header?

Thanks you

Johannes


#5

Is this equivalent to using a custom “client_id” and “client_secret”? Rclone can do that.

I suspect there might be more to it than that though (I don’t know much about JWT).

rclone doesn’t use the As-User header either, though it would be easy to add I suspect.


#6

I’ve seen this, not exactly. JWT is basically signed packages. This works passwordless, you use a private key to sign a claim that is then send via a post request.

It is very popular as it does not need manual password entries, this site contains also a link to a go lib.

From the library example code:

// Create a new token object, specifying signing method and the claims
// you would like it to contain.
token := jwt.NewWithClaims(jwt.SigningMethodHS256, jwt.MapClaims{
    "foo": "bar",
    "nbf": time.Date(2015, 10, 10, 12, 0, 0, 0, time.UTC).Unix(),
})

// Sign and get the complete encoded token as a string using the secret
tokenString, err := token.SignedString(hmacSampleSecret)

fmt.Println(tokenString, err)

It’s actually easier to use than OAuth2, as you can get a json file from box which contains all the necessary parts, which you can easily parse. The only difference to OAuth2 is there is no refresh token, you just send another claim to get a new access token. This is why it works with multiple clients as the token only expires by explicit invalidation or expiration.

For the as-user header a command line parameter that takes a json file with custom headers would be the easiest option enabling not only box but other cloud provider extensions.

I have no experience in go, but have done jwt already, with a library it is really easy.


#7

Very useful thanks :slight_smile:

Can I use JWT on a free personal box account (which is what I use for testing?)

It looks like I can but I’m not 100% sure.


#8

Sure, yes you can. If you mail me your login, I can check if any resources are missing and supply them.

I recently had written an example in python here:


and a colleague of mine has written some examples in node:

A lot of cloud providers do support JWT so this will have a larger benefit.

I’d also like to know what your opinion is on adding a custom header parameter. This way a lot of additional features could be supported, e.g. the as-user header with minimal coding.


#9

Thank you :smiley:

I use my normal email for my test account nick@craig-wood.com.

Nice examples thank you.

Interesting, It has passed me by somehow!

Do you have examples of other cloud providers which support it?

Custom header is very easy to add.

I made a new issue here: https://github.com/ncw/rclone/issues/2582

I’d love some feedback on that issue and links to any extra docs or things I should know!

Thanks

Nick


#10

JWT is widely supported.
Amazon supports it via Amazon Cognito: https://aws.amazon.com/cognito/

Google Cloud Storage supports it:

Auth0 support it:

Azure supports it as well:

Zapier has a good article when to use what kind of authentication:

In general OAuth2 is a great way for humans to authenticate, but if you want passwordless access JWT is a great extension to OAuth2.

Simplified you could say it is working similar to an ssh key authentication, with you holding the private key and the other side holding the public key.


#11

Thank you for that bit of research. I’ve stuck that text in the issue too :smile: