Took me a while to work this out as well - I think this is to replace the offline writes capability of rclone cache i.e. ‘commit’ defines the offline store that is moved to location-0 after 1H
I think this would be icing on the top, and if this was combined with some kind of scheduled write from local location-x to location-y in the cloud then that would be amazing. If not, relying on a scheduled rclone move job to upload files wouldn’t be an issue for me and I’d rather prioritise the core union functionality than replace what already works well.
I don’t think we need the ‘remote’ or ‘local’ bit:
type = union
location-0 = remote: ro sync-destination
location-1 = /Volumes/local-path rw sync-commits 1h
Having each location on one line would be easier to organise for me.
Where config prompts for up to 4 remotes and asks which remote is the optional source for sync-commits, which is the destination (sync-destination) and the schedule. I think only having one sync-destination makes sense but each location could potentially have a sync-commit schedule
Edit: removed the union mount-path as don’t think that should be in the config
If we want location-0location-1 etc, then the config wizard will have to prompt for each one, that is all. The config wizard can only prompt for a fixed number of things, so I suggested limiting it to 4 arbitrarily.
So it would look something like
Enter location-0 which should be a name of a remote followed by flags
Enter location-1 which should be a name of a remote followed by flags
Enter location-2 which should be a name of a remote followed by flags
Enter location-3 which should be a name of a remote followed by flags
At the moment you can put as many as you want in!
It makes a bit more work, but not a lot, and I think the 1 per line makes the config much easier to understand
That is correct, except it is the last remote in the “remotes” config line as there can be more than 2.
I guess that is part of the extension work.
I’m not 100% clear what you’d want to happen - you write to local disk, but in the background rclone uploads it (moves it maybe?) to the cloud remote - is that right?
Agree with those.
rclone needs to write to (at least one) location synchronously, presumably the local disk? I don’t want to build another cache layer! Is the idea that it would write to the cloud locations asynchronously, maybe even deleting the local copy after?
I think if the configuration was split
Then the config for the location-mode could have a nice dropdown (text equivalent) with lots of explanatory text.
Can the commit time be a global? That would be most convenient.
What do we think the different location-modes are
ro - read only - files are never written here only read
rw - files are read and written to here syncrhonously
wb - files are read from here and written back to here asynchronously from the rw locations after commit_time.
Essentially what I’d like to do is do an rclone copy or rclone move from the rw location to any wb locations.
This would give some constraints - the system is read only unless you provide one rw remote. If you have a wb remote then you must have a rw remote.
The current system is all the remotes except the last one are ro and the last one is rw.
should we allow more than one rw remote? This would synchronously write a file to multiple remotes
there is an issue about this
should we have a catchupcopy or move from the rw at regular intervals?
copy the files or move them from the rw remote?
this could be a config option
it could also be copy them, then delete them after a while
Yes I agree
I don’t understand what you mean by sync-commits and sync-destination. Is it covered by what I wrote above about wb remotes?
My preference is write to local disk i.e. highest priority location, so that this file is the one visible in the union mount
if either of the lower priority locations have a file with the same name as the new file, if it’s a rw location delete the dup, if it’s a ro ‘hide it’
For the hidden files it’d be good if there was a record of this kept somewhere so if desired a script can be used to delete the files from the ro remote. e.g. I use this script to cleanup my gdrive files that are ‘hidden’ when ‘deleted’ by unionfs https://enztv.wordpress.com/2017/03/09/unionfs-cleanup/
For my 2 above, I’m not sure of the exact terminology as I’m only a wannabe techie, but it’d be great if there was a ‘read, delete but no write option’ for the location. This would be ideal for me as if I upgrade a file e.g. a movie locally the old cloud file will be deleted and I can transfer the new local file at my leisure. In my scenario I don’t want the union to write to the cloud automatically, but I’m happy for it to delete.
I say no - sounds complicated to me, but I bet there’s someone out there who needs it!
would be nice, but I’m happy with rclone move to control bwlimit, max transfer etc etc
If I download a torrent and then I want to import it to my library, I can import it using a hardlink so I only have one copy of the file that remains in my seeding folder rather than copying which creates a 2nd copy and IO or moving it which would break seeding.
Unionfs doesn’t support hardlinking so sonarr, radarr etc can’t create a hardlink and have to create a 2nd copy - even if I’m trying to add a file to the local part of the union. Mergerfs does support hardlinking.
I see. Assuming we use the local backend for local disk then it doesn’t care about hardlinks (in fact they are completely invisible to it) so it would work just fine. I suspect unionfs works at an inode level where rclone works at a file/path level.
Yes that’s correct. I was also thinking that it might manage situations where the union mount is comprised only of cloud remotes (no local) - but I can see from the rest of conversation that this is not realistic/practical.
Yes, that seems sensible.
I think that would be fine for the majority of use cases.
rw - files are read, created, deleted and updated here synchronously
ro - read only - files are never written here only read
rd - files are read and deleted from here synchronously (including deletion of moved/renamed files)
rowb - files are read from here and written back to here asynchronously from the rw locations after commit_time
rdwb - files are read and deleted from here synchronously and written back to here asynchronously from the rw locations after commit_time
I don’t need this. Most use cases will be 2 remotes:
location-0 = rw local folder (highest read priority)
location-1 = rdwb cloud remote
With this config I probably don’t even need rclone to remember any history.
I would only use move, but if this is configurable then that will help some people I suspect.
Do you mean this should be defined instead at runtime as a parameter of the rclone mount command? Maybe a default one in rclone.conf that can be overridden at run time?
The commit_time/wb/cactchup move etc are all optional. I’m happy to use a scheduled rclone move instead.
Hmmm, thinking about this some more, one side effect of the change to mv (3) above is that the subsequent scheduled rclone move would need to handle files that have the same filename and size in both remotes.
So it would probably require an rclone move plus a deletion cleanup command. That is:
rclone move any file from location-0 to location-1 that is not identical (filename, subfolder, size) in both locations; then
delete any remaining (identical) files left over in location-0.
But also, if we start the union mount with location-0 being an empty local folder, then (3) should not really be able to occur anyway.