Having multiple VFS folders for one remote, with striping

I put myself in a kind of a puzzle.
I like to follow a low-budget setup, so trying to use whatever is given to me. This time, 2 16GB USB sticks have been given to me, and since they only work on some devices (maybe physical pins are defective?), I've put them on my raspberry to stay.
So the situation is:

root@palmapi:/home/pi# df -h
File system     Dim. Usati Dispon. Uso% Montato su
/dev/mmcblk0p2  115G   53G     58G  48% /
/dev/sdb2        14G   17M     13G   1% /mnt/2
/dev/sda1        15G   12K     14G   1% /mnt/1

The root file system is a microSD that suffers a lot of stress as I do lots of thing on it, including caching a rclone remote with VFS.
So I thought: What if I could use both the USB sticks and the microSD as VFS for the remote?
The main challenge, apart from the fact that rclone does not support multiple VFS folders, is that both the total size and the size of a single file in the VFS can be over 15G (my example file is 30G).
A RAID-0 striping system would work ; however, that would require partitioning the microSD, which is something I don't want to do, as then I would have a constant amount of space dedicated for cache. (Unless there is a software solution I don't know about)
A mergerfs file-level union system would then solve this problem, but as I said the biggest file is over 30GB and at best it would always go into the microSD, so this does not solve the microSD wear level problem.
So I thought: Why try not to combine/emulate the two systems? And what about using rclone for that?

So I made this:

type = local

type = local

type = local

type = union
upstreams = cache_1:/mnt/1/cache/ cache_2:/mnt/2/cache/ cache_3:/home/pi/cache/
create_policy = lfs
search_policy = epff
cache_time = 3600

type = chunker
remote = cache_u:
hash_type = none
meta_format = none
fail_hard = true

The lfs policy would make sure that the two USB sticks are used first and only when they're full the microSD is used (unless the microSD has less than 13GB, so that's hacky).
The chunker would divide big files in 1GB "stripes".

/usr/bin/rclone mount 'cache:' /home/pi/rclone-cache/ --allow-other

/usr/bin/rclone mount 'my_remote:' /home/pi/my_remote --vfs-cache-mode writes --vfs-cache-max-age 35m --allow-other --cache-dir /home/pi/rclone-cache

But my dream was quickly stopped by the fact that the cache mount itself requires a VFS folder to cache files opened in RW or to change the size of them. So we're back at the starting point now. :slight_smile:

Is this an impossible puzzle? Is there a solution? Thoughts?

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.