Box.com Mult-Part Multiple Uploads

Help and Support
1

I understand that if I'm copying say a single file, and do --transfers=50, it will do multiple multi-part uploads on that single file, and speeds up uploads significantly. This is using box.com

How would I go about upping the amount of multi-part uploads, without increasing the amount of files that are copied?
In a folder that has say 100 files in it, and I'd like to upload say 4 at a time, but allow up to XX part uploads, what would be the options to do this? I understand this increases RAM usage. I'd rather have single files finish faster, then it working on multiple files at once.

2

Not sure if it will work for box but you can try:

rclony copy src dst --multi-thread-cutoff 100M --multi-thread-streams 10 --transfer 10 -vv

--multi-thread-cutoff sets minimum file size to use multi threads and --multi-thread-streams defines number of streams

3

Thanks for the attempt! Unfortunately --multi-thread-streams did not do anything in this case.

4

So maybe not possible for box. Maybe somebody else knows box better and have some way to do it.

1 Like
5

--multi-thread is for downloads only at the moment.

However box does support multipart uploads

 --box-upload-cutoff SizeSuffix   Cutoff for switching to multipart upload (>= 50 MiB) (default 50Mi)

So any file bigger than 50M will be uploaded in chunks and these chunks will be uploaded concurrently. See the box docs for more info.

If you turn on -vv you'll see a DEBUG message for each chunk so you can see how it works.

1 Like
6

Thanks Nick, I believe that portion is working correctly, but when using the --transfers option, it is doing multiple files, instead of multiple pieces.

How do I limit it to upload only X amount of files at a time, but multiple pieces?

7

This gets so much easier if you use the template and share the information in it.

We keep guessing as what you might be doing and/or seeing.

8

--transfers controls both according to the docs. Use --transfers 10 to send with a maximum of 10 uploads at once. So this could be 1 file split into 10 chunks uploading in parallel. Or 10 files uploading with 1 chunk in parallel.

1 Like
9

Nick I think you're understanding now what I'm trying to do. Is there a way to force it to prioritize chunks over files?

11

whoever implemented box remote decided to use --transfers to control both at the same time - so yes it is possible if you re-implement it in more clever way. You are more than welcomed to give it a try if this is something you need.

1 Like
12

Understood, so not currently available, but could be added.

13

In the meantime you can use workaround - wrap rclone in shell script and process files in the loop one by one and then:

rclone copy src:file1 box:file1 --transfers 100

given that box docs are correct it will result with file being uploaded in 100 chunks.

1 Like
14

I'm not why you neglected you to use the template as I'm positive we wouldl have all saved some time. By not using the template, you make things more complicated and make the re-use of other people searching for things. That's why we ask for the template and ignoring and making a comment about me being 'positive' is rather rude but here we are.

1 Like
15

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.