Uploading to box.com seems capped at 50MiB/s

MB089 · July 19, 2023, 12:28pm

What is the problem you are having with rclone?

Hello,

we are currently getting familiar with uploading with Rclone to box.com.
Due to the nature of our business, we are dealing with large amounts of data.
These are folders with sometimes 4-5000 files, the largest files here are between 20-30GB in size.
Initially we now have to upload several hundred TB to BOX. Accordingly, we have opted for the Enterprise Plus variant of BOX.
However, the upload to BOX is currently somewhat difficult, since we cannot exceed 50MiB/s, despite a 10 GB connection (The routing was also checked with my network colleagues, there is no error or detour here).

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.1
- os/version: Microsoft Windows Server 2019 Datacenter 1809 (64 bit)
- os/kernel: 10.0.17763.4499 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.20.6
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

https://www.box.com/

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy "Source" Box:Test-Upload -P --transfers 32 --checkers 8

The rclone config contents with secrets removed.

[Box]
type = box
client_id = XX
client_secret = XX
box_sub_type = enterprise
token = {"access_token":"XX"}

[Overlay]
type = chunker
remote = Box
chunk_size = 49G

A log from the command with the -vv flag

PS C:\Windows\system32> rclone copy "SOURCE" Box:Test-Upload -P --transfers 32 --checkers 8 -vv
2023/07/19 14:04:26 DEBUG : rclone: Version "v1.63.1" starting with parameters ["C:\\Users\\"USER"\\Documents\\rClone\\rclone.exe" "copy" "\\\\"SOURCE" "Box:Test-Upload" "-P" "--transfers" "32" "--checkers" "8" "-vv"]
2023/07/19 14:04:26 DEBUG : Creating backend with remote "\\\\"SOURCE"
2023/07/19 14:04:26 DEBUG : Using config file from "C:\\Users\\"USER"\\AppData\\Roaming\\rclone\\rclone.conf"
2023/07/19 14:04:26 DEBUG : fs cache: renaming cache item "\\\\"SOURCE" to be canonical "//?/UNC/"SOURCE""
2023/07/19 14:04:26 DEBUG : Creating backend with remote "Box:Test-Upload"
2023-07-19 14:04:28 DEBUG : Thumbs.db: Sizes differ (src 12800 vs dst 220160)
2023-07-19 14:04:28 DEBUG : box root 'Test-Upload': Waiting for checks to finish
2023-07-19 14:04:28 DEBUG : box root 'Test-Upload': Waiting for transfers to finish
2023-07-19 14:04:29 DEBUG : XX: Multipart upload session started for 23 parts of size 8Mi
2023-07-19 14:04:29 DEBUG : XX: Uploading part 1/23 offset 0/179.246Mi part size 8Mi
2023-07-19 14:04:29 DEBUG : XX: Multipart upload session started for 15 parts of size 16Mi
2023-07-19 14:04:29 DEBUG : XX: Multipart upload session started for 21 parts of size 8Mi
2023-07-19 14:04:29 DEBUG : XX: Uploading part 1/21 offset 0/165.432Mi part size 8Mi
2023-07-19 14:04:29 DEBUG : XX: Uploading part 2/23 offset 8Mi/179.246Mi part size 8Mi
2023-07-19 14:04:29 DEBUG : XX: Uploading part 1/15 offset 0/231.125Mi part size 16Mi
2023-07-19 14:04:29 DEBUG : XX: Uploading part 2/21 offset 8Mi/165.432Mi part size 8Mi
2023-07-19 14:04:29 DEBUG : XX: Uploading part 3/23 offset 16Mi/179.246Mi part size 8Mi
2023-07-19 14:04:29 DEBUG : XX: Uploading part 3/21 offset 16Mi/165.432Mi part size 8Mi
2023-07-19 14:04:29 DEBUG : XX: Uploading part 4/23 offset 24Mi/179.246Mi part size 8Mi
2023-07-19 14:04:29 DEBUG : XX: Uploading part 4/21 offset 24Mi/165.432Mi part size 8Mi
2023-07-19 14:04:29 DEBUG : XX: Uploading part 2/15 offset 16Mi/231.125Mi part size 16Mi
2023-07-19 14:04:29 DEBUG : Thumbs.db: sha1 = 5e601a786ae02f845d854da7854a0f2a5cefd702 OK
2023-07-19 14:04:29 INFO  : Thumbs.db: Copied (replaced existing)
2023-07-19 14:04:30 DEBUG : XX: Uploading part 5/23 offset 32Mi/179.246Mi part size 8Mi
2023-07-19 14:04:30 DEBUG : XX: Uploading part 5/21 offset 32Mi/165.432Mi part size 8Mi
2023-07-19 14:04:30 DEBUG : XX: sha1 = 02c6e05244a76035360533ac36f9b7a484d87ce0 OK
2023-07-19 14:04:30 INFO  : XX: Copied (new)
2023-07-19 14:04:30 DEBUG : XX: Uploading part 6/23 offset 40Mi/179.246Mi part size 8Mi
2023-07-19 14:04:30 DEBUG : XX: Uploading part 3/15 offset 32Mi/231.125Mi part size 16Mi
2023-07-19 14:04:30 DEBUG : XX: Uploading part 6/21 offset 40Mi/165.432Mi part size 8Mi
2023-07-19 14:04:30 DEBUG : XX: Uploading part 7/23 offset 48Mi/179.246Mi part size 8Mi
2023-07-19 14:04:30 DEBUG : XX: Uploading part 7/21 offset 48Mi/165.432Mi part size 8Mi
2023-07-19 14:04:30 DEBUG : XX: Uploading part 8/23 offset 56Mi/179.246Mi part size 8Mi
2023-07-19 14:04:30 DEBUG : XX: Uploading part 4/15 offset 48Mi/231.125Mi part size 16Mi
2023-07-19 14:04:30 DEBUG : XX: Uploading part 8/21 offset 56Mi/165.432Mi part size 8Mi
2023-07-19 14:04:30 DEBUG : XX: Uploading part 9/23 offset 64Mi/179.246Mi part size 8Mi
2023-07-19 14:04:31 DEBUG : XX: Uploading part 9/21 offset 64Mi/165.432Mi part size 8Mi
2023-07-19 14:04:31 DEBUG : XX: Uploading part 10/23 offset 72Mi/179.246Mi part size 8Mi
2023-07-19 14:04:31 DEBUG : XX: Uploading part 5/15 offset 64Mi/231.125Mi part size 16Mi
2023-07-19 14:04:31 DEBUG : XX: Uploading part 10/21 offset 72Mi/165.432Mi part size 8Mi
2023-07-19 14:04:31 DEBUG : XX: Uploading part 11/23 offset 80Mi/179.246Mi part size 8Mi
2023-07-19 14:04:31 DEBUG : XX: Uploading part 11/21 offset 80Mi/165.432Mi part size 8Mi
2023-07-19 14:04:31 DEBUG : XX: Uploading part 12/23 offset 88Mi/179.246Mi part size 8Mi
2023-07-19 14:04:31 DEBUG : XX: Uploading part 12/21 offset 88Mi/165.432Mi part size 8Mi
2023-07-19 14:04:31 DEBUG : XX: Uploading part 6/15 offset 80Mi/231.125Mi part size 16Mi
2023-07-19 14:04:31 DEBUG : XX: Uploading part 13/23 offset 96Mi/179.246Mi part size 8Mi
2023-07-19 14:04:32 DEBUG : XX: Uploading part 13/21 offset 96Mi/165.432Mi part size 8Mi
2023-07-19 14:04:34 DEBUG : XX: Uploading part 14/23 offset 104Mi/179.246Mi part size 8Mi
2023-07-19 14:04:34 DEBUG : XX: Uploading part 7/15 offset 96Mi/231.125Mi part size 16Mi
2023-07-19 14:04:34 DEBUG : XX: Uploading part 14/21 offset 104Mi/165.432Mi part size 8Mi
2023-07-19 14:04:35 DEBUG : XX: Uploading part 15/23 offset 112Mi/179.246Mi part size 8Mi
2023-07-19 14:04:35 DEBUG : XX: Uploading part 8/15 offset 112Mi/231.125Mi part size 16Mi
2023-07-19 14:04:35 DEBUG : XX: Uploading part 15/21 offset 112Mi/165.432Mi part size 8Mi
2023-07-19 14:04:35 DEBUG : XX: Uploading part 16/23 offset 120Mi/179.246Mi part size 8Mi
2023-07-19 14:04:35 DEBUG : XX: Uploading part 9/15 offset 128Mi/231.125Mi part size 16Mi
2023-07-19 14:04:36 DEBUG : XX: Uploading part 16/21 offset 120Mi/165.432Mi part size 8Mi
2023-07-19 14:04:36 DEBUG : XX: Uploading part 17/23 offset 128Mi/179.246Mi part size 8Mi
2023-07-19 14:04:36 DEBUG : XX: Uploading part 10/15 offset 144Mi/231.125Mi part size 16Mi
2023-07-19 14:04:36 DEBUG : XX: Uploading part 17/21 offset 128Mi/165.432Mi part size 8Mi
2023-07-19 14:04:36 DEBUG : XX: Uploading part 18/23 offset 136Mi/179.246Mi part size 8Mi
2023-07-19 14:04:36 DEBUG : XX: Uploading part 11/15 offset 160Mi/231.125Mi part size 16Mi
2023-07-19 14:04:36 DEBUG : XX: Uploading part 18/21 offset 136Mi/165.432Mi part size 8Mi
2023-07-19 14:04:36 DEBUG : XX: Uploading part 19/23 offset 144Mi/179.246Mi part size 8Mi
2023-07-19 14:04:36 DEBUG : XX: Uploading part 19/21 offset 144Mi/165.432Mi part size 8Mi
2023-07-19 14:04:36 DEBUG : XX: Uploading part 12/15 offset 176Mi/231.125Mi part size 16Mi
2023-07-19 14:04:36 DEBUG : XX: Uploading part 20/23 offset 152Mi/179.246Mi part size 8Mi
2023-07-19 14:04:36 DEBUG : XX: Uploading part 20/21 offset 152Mi/165.432Mi part size 8Mi
2023-07-19 14:04:36 DEBUG : XX: Uploading part 21/23 offset 160Mi/179.246Mi part size 8Mi
2023-07-19 14:04:37 DEBUG : XX: Uploading part 21/21 offset 160Mi/165.432Mi part size 8Mi
2023-07-19 14:04:37 DEBUG : XX: Uploading part 22/23 offset 168Mi/179.246Mi part size 8Mi
2023-07-19 14:04:37 DEBUG : XX: Uploading part 13/15 offset 192Mi/231.125Mi part size 16Mi
2023-07-19 14:04:37 DEBUG : XX: Uploading part 23/23 offset 176Mi/179.246Mi part size 8Mi
2023-07-19 14:04:37 DEBUG : XX: Uploading part 14/15 offset 208Mi/231.125Mi part size 16Mi
2023-07-19 14:04:38 DEBUG : XX: Uploading part 15/15 offset 224Mi/231.125Mi part size 16Mi
2023-07-19 14:04:41 DEBUG : XX: commit multipart upload failed 1/100 - trying again in 2 seconds (not ready yet)
2023-07-19 14:04:43 DEBUG : XX: commit multipart upload failed 1/100 - trying again in 2 seconds (not ready yet)
2023-07-19 14:04:44 DEBUG : XX: sha1 = 74a41b1d6825087d050147415d2030b6bd9d8d6f OK
2023-07-19 14:05:01 INFO  : XX: Copied (new)
2023-07-19 14:05:05 DEBUG : XX: sha1 = c4a465a22918334a3c0a45fa8790898849e05ff4 OK
2023-07-19 14:05:05 INFO  : XX: Copied (new)
2023-07-19 14:05:05 DEBUG : XX: sha1 = 6ed230e93008958876eebc7caf3be7d3945e7b78 OK
2023-07-19 14:05:05 INFO  : XX: Copied (new)
Transferred:      576.871 MiB / 576.871 MiB, 100%, 9.317 MiB/s, ETA 0s
Checks:                 1 / 1, 100%
Transferred:            5 / 5, 100%
Elapsed time:        38.9s
2023/07/19 14:05:05 INFO  :
Transferred:      576.871 MiB / 576.871 MiB, 100%, 9.317 MiB/s, ETA 0s
Checks:                 1 / 1, 100%
Transferred:            5 / 5, 100%
Elapsed time:        38.9s

I tried to setup the Chunker from another thread, but looking at the test command, it looks like its not working and he splits up ~250MB files into several parts.
Is something fundamentally wrong with my command or config? Does BOX limit the upload rate?

Greetings and thanks in advance

Turner · July 19, 2023, 2:15pm

There is no limiting that I have seen, been going strong at ~250MiB/s - though I think this may have to do with the multi-part uploads being limited by --transfers.

kapitainsky · July 19, 2023, 3:49pm

You are not using chunker as first your chunker config has error, missing : after remote name. It should look like this:

[Overlay]
type = chunker
remote = Box:
chunk_size = 49G

and second - chunker is used when you actually use it:) so your copy command should look like this:

rclone copy "Source" Overlay:Test-Upload -P --transfers 32 --checkers 8

As you are on enterprise plus your max file size is 150G so maybe you do not need chunker at all. You have to decide.

This is question to BOX. But on this forum many people claim 200MiB/s+ speeds. Of course speed depends on many factors specific to every case.

system · August 18, 2023, 3:49pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.