After creating rclone union how do you (or can you) move files between remotes? Is there something like rclone switch union-remote:/path/to/file remote1 command? Is it possible at all to choose on which remote a particular file will be stored?
If it's not possible with rclone union are you aware of any other method on linux to logically merge directories of two remotes, so that you can explicitly choose where their files will go?
I imagine you could run rclone move remote1:/path/to/file remote2:/path/to/file, but then you need to repeat the path twice each time, which is prone to human-error (and tedious).
Run the command 'rclone version' and share the full output of the command.
The end goal is to move some of the files off the Google Drive due to recent 5TB restriction. AWS Glacier doesn't have a maximum storage limit, but it is very slow to retrieve the files, so it would be very useful to be explicit which file goes to AWS and which stays on Google Drive.
I was hoping that this could be achieved without separating them. Right now all files are stored in one tree (a big monorepo). It would be good if it was possible to keep them in the same location "logically", but move them somewhere else "physically" if that makes sense.
I suppose that this is doable, but it's a lot of typing (for deeply nested directories) for what seemed like a really common operation. You could write a shell script for that, but I'd rather not reinvent the wheel if someone has solved this "issue" already.
You can control where files are by using path preserving policies -so you can have folder1 on remote1 and folder2 on remote2. But you can achieve the same with combine which gives you one remote with access to multiple remotes.
Thanks, I've discussed this and related topics recently with the mergerfs author. I think the biggest problem for me was lack of demos for this particular purpose. He kindly shared useful collection of articles, which I hope will explain this a little more. I don't doubt that I'm making some wrong assumptions, but most examples focus on RAID-like setup, which is a bit different from "offloading a few selected files to an archive" that makes it hard to crack.
You would like to have your union to always create new content in hot remote - so eplfs (existing path, least free space) policy. As your local drive will always have less space than S3 remote - then you can even mount it and just use from your OS directly.
You would like to have your union to always create new content in hot remote - so eplfs (existing path, least free space)
thank you, that was another pressing question - that policy says least free space, but in this case you want to always use the "hot" remote, which confused me. I suppose that another way to control that is to make the "cold" remote no-create.
This is really cumbersome and. If you cd into that folder you need to format paths to be relative to the remotes. If you miss cache the first command may take a long time. It would be great if there was an rclone command that simply does:
A few gotchas: if you are in a mounted directory and need to convert its files to "rclone absolute paths" (e.g. ./file to 'remote:very/long/path/to/a/file') this can be accomplished with a zle widget (only works with zsh):
rclone-convert-path () {
# split buffer using "shell semantics" (quotes will be recognized)
local words=(${(z)BUFFER})
local newwords=()
# (Q) - Remove one level of quotes from the resulting words.
for word ( ${(Q)words} ) {
if [[ ! -e $word ]] {
newwords+=($word)
continue
}
rclone_drive=$(df "$word" | sed 1d | awk '{print $1}')
rclone_mount_path=$(df "$word" | sed 1d | awk '{print $6}')
file_rel_path=${"$(readlink -f $word)"#$rclone_mount_path}
newwords+=\'$rclone_drive$file_rel_path\'
}
# join array with " " whitespace character
BUFFER=${(j: :)newwords}
CURSOR=$#BUFFER
}
zle -N rclone-convert-path
you can then copy that path to fit rclone copyto remote:very/long/path/to/a/file remote:very/long/path/to/a/file format. This can be done relatively quickly
Thank you for sharing it. Maybe somebody can use it.
When you finish your setup maybe you can create github repo with all details? It can be really useful as I think problem you are solving is not unique.