Chunker not working?

Help and Support
1

Hi,

I am trying to upload to box.com and there is a limitation of 5GB in size per file. I've setup crypt and chunker. While all files are being encrypted, the files are not being split. How do i verify what is wrong?

rclone v1.62.2

  • os/version: ubuntu 22.04 (64 bit)
  • os/kernel: 5.19.0-46-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.20.2
  • go/linking: static
  • go/tags: none

Currently i have the following remotes setup:

box:
box-chunker:
box-crypt:
dropbox:
dropbox-crypt:
dropbox-upload:
google:
teamdrive:

I am using cloudplow to transfer the files automatically however, whenever it does,it fails with the error message saying the file size exceeded.

2

Maybe somebody can help you when you post all data required.

Start with content of your rclone.config file removing all secrets. For your question only box related remotes are important.

Also which box plan you have?

PS. Update your rclone to the latest version

3

Thank you for your reply.

Account Type: Business
Max File Size
5.0 GB

[box]
type = box
token = {"access_token":"X"}

[box-crypt]
type = crypt
remote = box:encrypt
password = X
password2 = X
filename_encoding = base32768

[box-chunker]
type = chunker
remote = box-crypt:
chunk_size = 4.800Gi
hash_type = md5quick
4

Ok.

I think is a problem...

4.8GiB = 5.15GB (GiB unit is based on power of 2 and GB on power of 10)

play it safe and set it to:

chunk_size = 4Gi

or do correct maths

5

here it is better to use sha1 hash as it is what Box supports natively

6

Thank you. I am still getting the error:

Error "file_size_limit_exceeded" (403): File size exceeds the folder owner's file size limit

I've done the update and made the change as suggested.

rclone v1.63.1
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.19.0-46-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.6
- go/linking: static
- go/tags: none

[box]
type = box
token = {"access_token":"X"}

[box-crypt]
type = crypt
remote = box:encrypt
password = X
password2 = X
filename_encoding = base32768

[box-chunker]
type = chunker
remote = box-crypt:
chunk_size = 4Gi
hash_type = sha1
7

and what is rclone command you use to transfer files? I suspect you do not use chunker at all...

8

I am using cloudplow to do it..

A part of the config reads:

{
    "core": {
        "dry_run": false,
        "rclone_binary_path": "/usr/bin/rclone",
        "rclone_config_path": "/home/seed/.config/rclone/rclone.conf"
    },
     "remotes": {
"box-crypt": {
            "hidden_remote": "box-chunker:",
            "rclone_excludes": [
                "**partial~",
                "**_HIDDEN~",
                ".unionfs/**",
                ".unionfs-fuse/**"
            ],
            "rclone_extras": {
                "--checkers": 16,
                "--drive-chunk-size": "64M",
                "--stats": "60s",
                "--transfers": 8,
                "--verbose": 1
            },
 "rclone_sleeps": {},
            "sync_remote": "box-crypt:/Media",
            "upload_folder": "/mnt/local/Media",
            "upload_remote": "box-crypt:/Media"
        }
       },
 "syncer": {},
    "uploader": {
        "box-crypt": {
            "check_interval": 3,
            "exclude_open_files": true,
            "max_size_gb": 3,
            "opened_excludes": [
                "/downloads/"
            ],
            "size_excludes": [
                "downloads/*"
            ]
        }
    }
}

    }
}
9

Ok I have no clue about cloudplow.

but I can see that your remotes' setup is as follows:

box <-- box-crypt <-- box-chunker

So in order to use all chain you should use box-chunker e.g.

rclone copy src box-chunker:

You have to make sure that your cloudplow software is using box-chunker remote and only this remote.

You can test it by running e.g.:

rclone copy /path/to/5GplusSize.file box-chunker:test/5GplusSize.file

if it works than it is your other software configuration problem.

10

Thank you. I tried that but it failed

2023-08-10 21:13:19 NOTICE: largefile.qasucin6.partial: Removing partially written file on error: close /mnt/box/test/largefile.qasucin6.partial: input/output error
2023-08-10 21:13:19 ERROR : largefile: Failed to copy: close /mnt/box/test/largefile.qasucin6.partial: input/output error
2023-08-10 21:13:19 ERROR : Attempt 1/3 failed with 1 errors and: close /mnt/box/test/largefile.qasucin6.partial: input/output error
Transferred:        6.185 GiB / 12 GiB, 52%, 3.481 MiB/s, ETA 28m30s
Transferred:            0 / 1, 0%
Elapsed time:     31m14.5s
Transferring:
 *                                     largefile:  3% /6Gi, 4.119Mi/s, 24m5s
11

I can not see what you tried.

You just post some error

12

I tried this

rclone copy /home/user/largefile box-chunker:test/5GplusSize.file -P

this is currently the progress

Transferred:      228.312 MiB / 6 GiB, 4%, 2.268 MiB/s, ETA 43m28s
Transferred:            0 / 1, 0%
Elapsed time:      1m46.6s
Transferring:
 *                                     largefile:  3% /6Gi, 2.268Mi/s, 43m28s

Is this intended or should it split the 6GB to a smaller file?

13

It will split large file. But looks like it will take time:) ETA 43m28s

To make it faster try to increase transfers (unless you know that your uplink connection is slow 30 mbps?):

rclone copy /home/user/largefile box-chunker:test/5GplusSize.file -P --transfers 32

when finished you can see if file is split:

rclone ls box-crypt:test

chunker should show one big file:

rclone ls box-chunker:test

1 Like
14

Thank you.

It seems that rclone chunker is working as intended and thank you for the tip on increasing the speed of transfer. My upload is about 500Mbps therefore that helped to quicken the transfer. Thank you for your assistance and super fast responce.

1 Like
15

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.