Is there an advantage to using the new union remote over using unionfs or mergerfs? Is it just a simpler configuration, or are there features/functional benefits from using this new option in rclone?
Thanks!
I think the main benefit is that it works for platforms where unionfs/mergerfs arenāt supported (either at all or you donāt have the necessary modules or permissions to make mounts).
1 Like
I was excited about this until I saw that only the last remote in the union is writable. The rest are read only so this wonāt work for my use but hopefully one day they make a change.
2 Likes
What would you like to see?
Thanks for the reply. For my situation I would like all the remotes to have read+delete but it would be fine for only the last remote to be write as well.
Maybe though to cater for everyone a system more similar to union where you pick which remotes are read only, delete and which are writable. For example
remote1:folder(rd) remote2:folder(rdw) remote3:folder(rdw)
Just an idea, no idea how hard or easy this request is.
1 Like
Interesting idea. Maybe the thing to do is look at some prior art.
I found this article on unionfs: https://www.linuxjournal.com/article/7714
So if the union backend were to gain the suffixes =RO, =RW that unionfs has would that be what you want?
1 Like
Yes I think that would suit well as long as it was technically feasible. I donāt know how it all works I am just looking to replace unionfs as I am sure some others would too.
I would also like to request a union remote something like:
new dir/file: on highest priority remote
move dir/file: on all remotes
delete dir/file: on all remotes
open r/w file: read from highest possible priority remote, but write to highest possible remote
1 Like
You are really describing all the functionality already in mergerfs.
1 Like
No doubt but in my case I cannot install mergerfs but I can use rclone without restriction. If this functionality never makes it to rclone I am ok with that. The tool is excellent and works for me fantastically. What I am personally describing is a nice to have feature.
What features do you like of mergerfs?
Heās got a whole section of policy configuration that is very well documented and the options are huge in terms of using a disk first, round robin, random, least space, etc.
The two main features for me that I use are the ability to use hard links. So if I stay on the same disk underneath mergerfs, it will allow me to hard link which is very helpful for Sonarr/Radarr/Torrents as I donāt generate a lot of extra IO with double copies since it allows me to hardlink.
Second, I use the always write to the first entry as I have my local disk first so any new write request always goes to the local disk and it just handles the structure much better than unionfs imo as I donāt have to clean up all the hidden leftovers if items move around from local to my GD.
2 Likes
There should be more types like
- JBOD (I would love this)
- RAID 0 / 1
I think that would be great.
Also the ability to have a single RW and a RD flag, so you can say, READ-DELETE.
So, if a delete is done on the union mount, it deletes from the RW, and the RD mounts.
Also would be pretty cool if rclone move /mnt/local_files union:/files
copies the files to all remotes in the union but removes the local files after moving to the last remote
1 Like
Hi @ncw thanks again for adding useful functionality. If I understand your implementation correctly thereās one feature Iād need to be able to ditch unionfs - with unionfs for my RO folders, I can still ādeleteā files - the files arenāt actually deleted but unionfs hides them from the union so they appear deleted to any app looking at that folder.
E.g. if I have a 720p version of a movie in a RO share and I add a 1080p to my RW share, when radarr ādeletesā the 720p file it will be hidden. With rclone union because the files will have diff names the 720p version will still be visible.
If rclone union hid or did something to similar to hide the file, or if all remotes were RW but new files were written to the last one then that would work for me as well ( I think this is how mergerfs works, but I canāt install mergerfs on unRAID/slackware)
1 Like
The new Union remote is a fantastic move for everybody that:
- Has struggled to get their desired performance from rclone cache mounts (relative to vfs); and
- Is in one of these scenarios:
ā Wishes to avoid setting up a mergerfs/unionfs/etc implementation; or
ā Cannot install mergerfs/unionfs/etc on their system (eg. OSX & Windows users).
@Animosity022ās generously shared setup on github is awesome and mergerfs is so flexible - if I were a linux user right now I would just replicate @Animosity022ās config.
But as an OSX user, I am excited by where the Union remote is going. This seems spot on:
I suspect the default setup that most of us Union remote candidates are after is:
- Read: Remote 2 has priority, Remote 1 if not found in Remote 2 (same as now).
- Write/Move: Remote 2 only (same as now).
-
Delete:
ā If filename is only found on Remote 2: Delete from Remote 2 (same as now).
ā If filename is only found on Remote 1: Delete from Remote 1 (change).
ā If filename is found on both remotes: Delete from Remote 1 & Remote 2 (change). -
Optional: the ability to easily commit changes from Remote 2 to Remote 1:
ā An --rc flag or a mount commit frequency option.
ā Optional because a scheduled ārclone move /local_path_to_remote2 Remote1:pathā could achieve this instead.
For many of us:
- āRemote 1ā above would be a crypt mount atop a drive mount.
- āRemote 2ā above would be a local folder (" /local_path_to_remote2"), not a remote.
Yes, implementing this in the Union remote is re-inventing a poor manās mergerfs, but I suspect there are a lot of us for whom this would make rclone the perfect one-stop-shop cloud mounting solution.
3 Likes
Actually the most difficult bit of all of this is designing how it would be configured! So if you all could have a think about what the configuration would look like in the config file (given its current limitations of key, value) that would be really helpful.
So currently the union backend config looks like this
[TestUnion]
type = union
remotes = /tmp/union1 /tmp/union2 /tmp/union3
How would I change that config to specifiy the above?
Maybe something like:
[TestUnion]
type = union
location-0 = gmedia:path
location-1 = /tmp/cache
mount-point = /tmp/union
write-location = location-1
read-priority = location-1 location-0
commit-source = location-1
commit-frequency = 1H
Good start. I think to simplify, rather than read-priority, just default to the priority goes in a certain order e.g. location-0 is highest, then location-1 - all users have to do then is order their locations in priority order
Iād rather lose commit-source and just stick with rclone move as it offers so much more control.
Not bad! Referencing the remotes makes life easierā¦ How about using the implicit position of the position in remotes which would make it backwards compatible and easier for rclone.
The config editor would have to ask for location-0 then location-1 etc. If I made 4 locations in the config editor by default that would probably be enough. You could always add more by editing the config file directly.
The current remotes string could be used also which would default to location-0 location-1 location-2 for backwards compatibility.
What does commit-source do?
Do the location-0 need some modifiers, eg
location-0 = /tmp/path ro location-1 = /tmp/path2 rw
which could control whether files get deleted off there or not.
1 Like