Avngl
August 10, 2023, 10:57am
1
Hi,
I am trying to upload to box.com and there is a limitation of 5GB in size per file. I've setup crypt and chunker. While all files are being encrypted, the files are not being split. How do i verify what is wrong?
rclone v1.62.2
os/version: ubuntu 22.04 (64 bit)
os/kernel: 5.19.0-46-generic (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.20.2
go/linking: static
go/tags: none
Currently i have the following remotes setup:
box:
box-chunker:
box-crypt:
dropbox:
dropbox-crypt:
dropbox-upload:
google:
teamdrive:
I am using cloudplow to transfer the files automatically however, whenever it does,it fails with the error message saying the file size exceeded.
Maybe somebody can help you when you post all data required.
Start with content of your rclone.config file removing all secrets. For your question only box related remotes are important.
Also which box plan you have?
PS. Update your rclone
to the latest version
Avngl
August 10, 2023, 11:10am
3
Thank you for your reply.
Account Type: Business
Max File Size
5.0 GB
[box]
type = box
token = {"access_token":"X"}
[box-crypt]
type = crypt
remote = box:encrypt
password = X
password2 = X
filename_encoding = base32768
[box-chunker]
type = chunker
remote = box-crypt:
chunk_size = 4.800Gi
hash_type = md5quick
Ok.
Avngl:
chunk_size = 4.800Gi
I think is a problem...
4.8GiB = 5.15GB
(GiB unit is based on power of 2 and GB on power of 10)
play it safe and set it to:
chunk_size = 4Gi
or do correct maths
Avngl:
hash_type = md5quick
here it is better to use sha1
hash as it is what Box
supports natively
Avngl
August 10, 2023, 11:40am
6
Thank you. I am still getting the error:
Error "file_size_limit_exceeded" (403): File size exceeds the folder owner's file size limit
I've done the update and made the change as suggested.
rclone v1.63.1
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.19.0-46-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.6
- go/linking: static
- go/tags: none
[box]
type = box
token = {"access_token":"X"}
[box-crypt]
type = crypt
remote = box:encrypt
password = X
password2 = X
filename_encoding = base32768
[box-chunker]
type = chunker
remote = box-crypt:
chunk_size = 4Gi
hash_type = sha1
and what is rclone
command you use to transfer files? I suspect you do not use chunker at all...
Avngl
August 10, 2023, 11:57am
8
I am using cloudplow to do it..
A part of the config reads:
{
"core": {
"dry_run": false,
"rclone_binary_path": "/usr/bin/rclone",
"rclone_config_path": "/home/seed/.config/rclone/rclone.conf"
},
"remotes": {
"box-crypt": {
"hidden_remote": "box-chunker:",
"rclone_excludes": [
"**partial~",
"**_HIDDEN~",
".unionfs/**",
".unionfs-fuse/**"
],
"rclone_extras": {
"--checkers": 16,
"--drive-chunk-size": "64M",
"--stats": "60s",
"--transfers": 8,
"--verbose": 1
},
"rclone_sleeps": {},
"sync_remote": "box-crypt:/Media",
"upload_folder": "/mnt/local/Media",
"upload_remote": "box-crypt:/Media"
}
},
"syncer": {},
"uploader": {
"box-crypt": {
"check_interval": 3,
"exclude_open_files": true,
"max_size_gb": 3,
"opened_excludes": [
"/downloads/"
],
"size_excludes": [
"downloads/*"
]
}
}
}
}
}
Ok I have no clue about cloudplow
.
but I can see that your remotes' setup is as follows:
box <-- box-crypt <-- box-chunker
So in order to use all chain you should use box-chunker
e.g.
rclone copy src box-chunker:
You have to make sure that your cloudplow
software is using box-chunker
remote and only this remote.
You can test it by running e.g.:
rclone copy /path/to/5GplusSize.file box-chunker:test/5GplusSize.file
if it works than it is your other software configuration problem.
Avngl
August 10, 2023, 1:14pm
10
Thank you. I tried that but it failed
2023-08-10 21:13:19 NOTICE: largefile.qasucin6.partial: Removing partially written file on error: close /mnt/box/test/largefile.qasucin6.partial: input/output error
2023-08-10 21:13:19 ERROR : largefile: Failed to copy: close /mnt/box/test/largefile.qasucin6.partial: input/output error
2023-08-10 21:13:19 ERROR : Attempt 1/3 failed with 1 errors and: close /mnt/box/test/largefile.qasucin6.partial: input/output error
Transferred: 6.185 GiB / 12 GiB, 52%, 3.481 MiB/s, ETA 28m30s
Transferred: 0 / 1, 0%
Elapsed time: 31m14.5s
Transferring:
* largefile: 3% /6Gi, 4.119Mi/s, 24m5s
I can not see what you tried.
You just post some error
Avngl
August 10, 2023, 1:27pm
12
I tried this
rclone copy /home/user/largefile box-chunker:test/5GplusSize.file -P
this is currently the progress
Transferred: 228.312 MiB / 6 GiB, 4%, 2.268 MiB/s, ETA 43m28s
Transferred: 0 / 1, 0%
Elapsed time: 1m46.6s
Transferring:
* largefile: 3% /6Gi, 2.268Mi/s, 43m28s
Is this intended or should it split the 6GB to a smaller file?
It will split large file. But looks like it will take time:) ETA 43m28s
To make it faster try to increase transfers (unless you know that your uplink connection is slow 30 mbps?):
rclone copy /home/user/largefile box-chunker:test/5GplusSize.file -P --transfers 32
when finished you can see if file is split:
rclone ls box-crypt:test
chunker should show one big file:
rclone ls box-chunker:test
1 Like
Avngl
August 10, 2023, 3:05pm
14
Thank you.
It seems that rclone chunker is working as intended and thank you for the tip on increasing the speed of transfer. My upload is about 500Mbps therefore that helped to quicken the transfer. Thank you for your assistance and super fast responce.
1 Like
system
(system)
Closed
August 13, 2023, 3:05pm
15
This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.