Recommended Box remote setup

[kapitainsky]

in response to:

"Unlimited" alternatives to Google Drive, what are the options?

Hi everyone. Been reading through this great thread. I have also had a read through the setup for rclone and box but i see you guys have a lot of tweaks in there. Is there any post/guide for all those working tweaks for a simpleton such as myself. Clearly you guys are well on top of it but i'd just be lost. I can implement but I can't develop. Just need to get off workspace over the next 59 days. There are 4 of us that are in the same situation and need to move to box.

Box has max file size limitation for most of their plans so it is required to use chunker:

rclone.config:

[box_remote]
type = box
token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"XXX"}

[crypt_remote]
type = crypt
remote = box_remote:data
password = XXXX
password2 = XXXX
filename_encoding = base32768

[chunker_remote]
type = chunker
remote = crypt_remote:
chunk_size = 49G
name_format = *.rcc###
hash_type = sha1all

Create data folder on your box remote - rclone mkdir box_remote:data - it is better to keep data in specific folder rather than in root.

filename_encoding = base32768 option in crypt provides most space efficient file names encoding. Unlike e.g. Google, Box has much lower limits here and without this option it is very easy to get into "path too long" situation during data migration and crypt default base32 encoding.

Adjust chunk_size accordingly to your box account limits, e.g.: 4G, 14G, 49G

hash_type = sha1all option enables SHA1 hashes inside crypt/chunker combo which is very handy for content verification. Normal crypt check won't work as files are chunked. Downside is that extra json metadata file is created for every file. Normally only for chunked ones. If you have many big files it comes effectively at no cost. But if you have many small files it will result in double number of files in box_remote:. I am not aware of any limitations here though.

Always access your Box setup using chunker_remote, e.g.:

rclony sync src_remote:folder chunker_remote:folder

[enz1ey]

Thank you for this helpful info!

Do you have any tips for any rclone mount flags that can/should be used with Box to improve performance? It seems a little slower than Google Drive was for me when using the mount point to access files. Right now I'm just mounting without any flags.

[kapitainsky]

Keep things as simple as possible. Use VFS full cache mode + cache pre-warming for best experience. I do not think you can do much to improve speed of mount on BOX. You could start new thread "recommended mount options for BOX" - maybe others have some useful tips.

[unsaved]

Thank you ever so much for this, I am using box enterprise recently.
if I am using a rclone service where it auto mounts, how to add the command you mentioned.

rclony sync src_remote:data chunker_remote:data

Currently I am using this as my mount configuration, accessing my box mount through crypt and it's working perfectly but it's really slow specially when it comes to plex scanning.

do I have the correct parameters here, your directions are much appreciated.

[Unit]
Description=rclonemount
After=network.target

[Service]
Type=simple
User=%i
Group=%i
ExecStartPre=-/bin/mkdir -p /home/%i/cloud/
ExecStart=/usr/bin/rclone mount crypt: /home/%i/cloud/   --user-agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36'   --config /home/%i/.config/rclone/rclone.conf   --use-mmap   --dir-cache-time 5m0s   --timeout 30s   --umask 002   --vfs-cache-max-size 5G   --vfs-fast-fingerprint   --allow-other   --poll-interval=15s   --vfs-cache-mode full   --vfs-read-chunk-size 1024M  --vfs-write-back 1h --vfs-cache-max-age 9999h  --vfs-read-chunk-size-limit 10G   --tpslimit 10
ExecStop=/bin/fusermount -u /home/%i/cloud
Restart=on-failure
RestartSec=30
StartLimitInterval=60s
StartLimitBurst=3

[Install]
WantedBy=multi-user.target

[kapitainsky]

this thread is about remotes' setup - and I'm happy for any comment and ideas - for mount I do suggest you open new thread.

Mixing every aspect of BOX in one thread does not make it easy to find anything.

I look at BOX purely as a storage option - I have no idea about plex (never used it) or any software operating on it - it is very different subject. mount for Plex might need different tweaking than Adobe rendering or ML software.

My only purpose here is to show how to use crypt and chunker to overcome BOX limitations.

[durval (Durval Menezes)]

Thank you very much @kapitainsky for the great how-to!

I've just started using it after having started with something much more pedestrian, and almost had a heart attack when I started seeing things like this on rclone's -vv output:

2023/07/24 14:05:14 DEBUG : 崝艪㵧ꅎ竑䘃㪜伱熟岧ꍏ選礵㸗槴䥔ᣃ侻釥▌疾ꖗ脘擝㝺ᔿ/ꍐ圿鮝䅵⤞梀脆鈺舂銬䥺哽踮倅櫄褦啗☝桋佭䌄樔䰵軺荔坿艙藇馑㾽眖鋳麑㠃Ɵ: Uploading part 33/108 offset 1Gi/3.363Gi part size 32Mi
2023/07/24 14:05:14 DEBUG : 牁䪌䭮㓉躟洨䌻旿纔螒ꏗ⠪板躎寀暿㧀國訠儠婧䐤ᒵ滚㪩硟/壌轈燛燴垀羬噱暖䮇㪅疬屚㖹旳膳菾嶉ꇂ䃵港ꌒ㿇乩䥒䬍䫗㚂蛵㩾ᴗ絲麒䉓嬩ɿ: Uploading part 35/129 offset 1.062Gi/4.001Gi part size 32Mi
2023/07/24 14:05:15 DEBUG : 崝艪㵧ꅎ竑䘃㪜伱熟岧ꍏ選礵㸗槴䥔ᣃ侻釥▌疾ꖗ脘擝㝺ᔿ/䰍帆㞐逢㫯㬊ᄐ蹏復恶㿛皨㤄杢瀋ᓶ硋䑉䳌旒㱄譺嵹橘蹌蹚臤椳㦱兗繘鑐莯悒鰕譕偮晩䤋䒀谭观榿: Uploading part 23/129 offset 704Mi/4.001Gi part size 32Mi
2023/07/24 14:05:16 DEBUG : 規㝊隇䃕渼骕㑛悆履ᄵ㶋Ꮄ黯䞂袗䜄咰嫩☟鲐褣軒秥ᕇ䫣ᦍ獜䋬㳀毠宱秫ꍊ❇ꎛ㿮㘧褌荩婽伓餏ሲ峣䮡Ⴖ懡ᚪ䲅鴋䟠ɟ: Uploading part 51/129 offset 1.562Gi/4.001Gi part size 32Mi
2023/07/24 14:05:18 DEBUG : 雋茙綉漅䐀⤂㠍㟀鬥䤁少骋鋡箄逆峇䉒籛鷩葎兝夦䛁㥟暥电緥㣤猫欚埪嵱紩禒ɟ: Uploading part 51/59 offset 1.562Gi/1.829Gi part size 32Mi
2023/07/24 14:05:19 DEBUG : 崝艪㵧ꅎ竑䘃㪜伱熟岧ꍏ選礵㸗槴䥔ᣃ侻釥▌疾ꖗ脘擝㝺ᔿ/㲋嗴擤㘼嘭面䵔ⳝ珒蔳鲓諘丒縷全䱔瑙凤簨⣛伔▢㿔㷝烖㡶聎暛呤䩩産怱芡㕾郷髯蒣缮ተ匂螚摪渿: Uploading part 25/129 offset 768Mi/4.001Gi part size 32Mi

So in retrospect it's quite obvious that's expected from base32768 and not a sign that my machine has been cracked by Chinese hackers

[kapitainsky]

⣛伔▢⤞ᣃተ - or aliens:) it is interesting set of characters used indeed - even if at the first look it looks like chinese it is not:) We tested every single character - and however unbelievably scary it looks it works perfectly.

[durval (Durval Menezes)]

@kapitainsky

⣛伔▢⤞ᣃተ - or aliens:) it is interesting set of characters used indeed - even if at the first look it looks like chinese it is not:)

Depending on the exact meaning attached the word "aliens", Chinese most certainly are as neither you or I are chinese

Paraphrasing Von Daniken (the biggest con-man there ever was), "Are the Chinese Astronauts?" -- and it turns out they are! List of Chinese astronauts - Wikipedia

[kapitainsky]

For people interested it is actually quite interesting story how this "alphabet" for base32768 was produced:

What makes a Unicode code point safe? @ Things Of Interest

[Dual-O (Oli)]

FYI.
if you use Business Plan with 5GB file size limit:
chunk_size = 5G doesnt work.
chunk_size = 5000M works!

[Dual-O (Oli)]

what about crypt -> chunker -> box ?

How to use check and cryptcheck with chunker+ crypt remotes

If you want do do a full end to end test then use rclone check --download - that will download all the data and check the checksums which works for any backend whether they support checksums or not. Depending on exactly how you've set up chunker, it may be storing checksums for you so rclone check may work just fine. I don't think you can use rclone cryptcheck if it isn't the top remote. So if you want to use rclone cryptcheck you might need to wrap them the other way round Crypt -> Chunker -> …

[kapitainsky]

@Dual-O

chunk_size = 5G doesnt work.
chunk_size = 5000M works!

It is always PITA to be sure who uses what convention - (base 10 or base 2) - and good luck to trust GB vs GiB - only way is to test or play safe.

So I would not think too much and use 4G in your example. Unless for some special reasons I cared but then I would test very carefully.

[kapitainsky]

@Dual-O

what about crypt -> chunker -> box ?

it will work too.

But if you use crypt you want to utilise its maximum potential:) why to reveal that you are chunking files?

chunker -> crypt -> box

makes everything on box encrypted without leaking any information. Your way makes it clear what you are doing.

At the end it is really up to you what you use. Depends what you want to achieve. There is no one way. But some paths are better than others.

[MrDevanWright]

@kapitainsky

it is better to keep data in specific folder rather than in root.

Why? Is there some performance or stability benefit, or just your best practice opinion?

[kapitainsky]

only best practice and attempt to avoid any issues. root folder often is treated in special way by cloud providers - e.g. they put there some their solution specific objects like shared folders etc. I did not do any research on it with regards to box. No time and no reason. This approach is purely based on play safe tactic:) You can try crypt root folder of course. And share your findings.

I think some structured approach in data storage always helps. I have my crypt data in data_crypt folder. Then I can have different data in data1_crypt folder - different passwords etc. Could I have it all in one folder? Yes. But then I take a risk that I hit some edge cases of any software (rclone here) not working 100% to spec in extreme situations. So as lazy as I am I use separate folders:) Then I do not have to bother people on the forum with questions about missing data:)

[Dual-O (Oli)]

@kapitainsky

hash_type = sha1all

Hey @kapitainsky why do you use sha1all and not sha1. Box supports sha1 checksums, so we can use sha1 to keeps hashes for composite files and falls back to the wrapped remote hash for non-chunked ones, rclone docs advise to choose the same hash type as supported by wrapped remote so that your file listings look coherent.

[kapitainsky]

@Dual-O

Hey @kapitainsky why do you use sha1all and not sha1. Box supports sha1 checksums, so we can use sha1 to keeps hashes for composite files and falls back to the wrapped remote hash for non-chunked ones, rclone docs advise to choose the same hash type as supported by wrapped remote so that your file listings look coherent.

You are right that Box supports sha1 and your logic is correct but does not take into account crypt remote. So the way how you describe it would only work with this setup:

chunker->box

Crypt does not support any hashes. It could be theoretically implemented but is not. My workaround here is to utilize chunker capability to generate and store hashes for all files.

If you do not need it then you can use sha1 option - but then you will have hashes only for chunked files.

The downside of using sha1all is slower upload of small files (for every file another side-car json file has to be created). You have to decide based on your usage patters. If you constantly write tones of small files then maybe it is not worth. I use cloud as my backup storage and after initial upload (and I accept that instead of 1 day it might take 2 days) subsequent uploads are relatively small. Advantage is that I have very reliable way to verify my local files vs files stored in the cloud.

[Bowman92 (Natashia Bowman)]

what command do i need to use chunker? I must have moved all my data from Google to box encrypt

[kapitainsky]

Could you please start new thread if you have questions not related explicitly to box setup? It is better for everybody when different issues are not mixed.

Then provide all details of your setup and description of issue you are facing.

[nachobel]

Kap - do you have recommendation for how to mount the rclone mount on your filesystem? I've been using an rclone_vsf.service to mount google drive, but using the same settings for box.com is unreliable (very very slow to update). If I use rclone move or sync to copy files to the mount (as you recommend - chunker_remote:), the files copy and I can see their obfuscated names in the box.com web interface, but on my file system looking in the directory I've mounted the rclone box.com mount to, I don't see anything. Looking for help!