Union remote - Priority documentation

Help and Support
1

What is the problem you are having with rclone?

Union remote - Trying to understand the policies.

I'm trying to use 2 remotes. One online, one local. There is overlap with files. I'm using this as read only.

What I'd like is to be able to use a union remote to prioritize the local filesystem and if not, then use the online remote to read from.

I see things dealing with used space, free space, objects, timestamps, first to reply.

So, that 'first to reply' sounds close, but if it's on an external disk, that could actually take a bit while external disks spins up.

So I'm not exactly sure if this can be handled. If it can, then I'm just not understanding this.

What is your rclone version (output from rclone version)

1.55.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

macOS 11.2.3

Which cloud storage system are you using? (eg Google Drive)

Union combining Local and Google Drive.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

I'm just trying to make sense of the documentation at this point.

2

I think what you want is epff - I think it is very unlikely your remote will beat your local hard disk - it in order to lose it would have to return a directory listing slower than an http round trip.

3

Thank you!

One more question if I may, is there a way to test or see which remote the union is picking? Just want to test things out or even come up with some logging.

Thank you again.

4

Actually, I have a question interpreting something.

epall mentions,

Search category: Given this order configured, act on the first one found where the relative path exists.

I'm having some issue interpreting this. Would you say it's safe to interpret it as,

Based on the order of the remotes given during configuration, act on the first one where the relative path exists.

If so, assuming on a read operation (based on search category), if I write my Local: remote first on the configuration, that it would first search this one. If that fails, regardless of time, then it goes to the next remote and so on. I of course see the latency issue with this policy with slower remotes.

5

I was going to test things out and may have found a bug,

The TL;DR is, using a Local remote on a Union lists the files, but can't actually read them.

Here's what I've found,

My test configuration,

[Local]
type = local

[Union-epff-local]
type = union
upstreams = Local: GDrive:test/dir
search_policy = epff
cache_time = 1

Local directory,

$ ls
    file_1.txt      file_local.txt

Remote directory,

$ rclone ls GDrive:/test/dir
    7 file_remote.txt
    7 file_1.txt

Notes:

  • GDrive is just a drive remote. type drive, scope drive. In case that changes anything.
  • All plain text files.
  • Content of local files have "Local" in them.
  • Content of remote files have "Remote" in them.

First test, ls shows all files with all 4 combinations

$ rclone ls Union-epff-local:
    6 file_1.txt
    6 file_local.txt
    7 file_remote.txt

Now, this is what looks to be a bug with using the local type remote on a union,

$ rclone -vvvvv cat Union-epff-local:file_local.txt
    2021/05/08 17:32:47 DEBUG : Using config file from "/Users/me/.config/rclone/rclone.conf"
    2021/05/08 17:32:47 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "-vvvvv" "cat" "Union-epff-local:file_local.txt"]
    2021/05/08 17:32:47 DEBUG : Creating backend with remote "Union-epff-local:file_local.txt"
    2021/05/08 17:32:47 DEBUG : Creating backend with remote "GDrive:test/dir"
    2021/05/08 17:32:47 DEBUG : Creating backend with remote "Local:"
    2021/05/08 17:32:47 DEBUG : fs cache: renaming cache item "Local:" to be canonical "Local:/Users/me/PhpstormProjects/rpi/tmp"
    2021/05/08 17:32:47 DEBUG : Creating backend with remote "Local:/file_local.txt"
    2021/05/08 17:32:47 DEBUG : Creating backend with remote "GDrive:test/dir/file_local.txt"
    2021/05/08 17:32:48 DEBUG : union root 'file_local.txt': actionPolicy = *policy.EpAll, createPolicy = *policy.EpMfs, searchPolicy = *policy.EpFF
    2021/05/08 17:32:48 ERROR : : error listing: directory not found
    2021/05/08 17:32:49 DEBUG : 8 go routines active
    2021/05/08 17:32:49 Failed to cat with 2 errors: last error was: directory not found

This does work,

$ rclone -vvvvv cat Local:file_local.txt
    2021/05/08 17:26:09 DEBUG : Using config file from "/Users/me/.config/rclone/rclone.conf"
    2021/05/08 17:26:09 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "-vvvvv" "cat" "Local:file_local.txt"]
    2021/05/08 17:26:09 DEBUG : Creating backend with remote "Local:file_local.txt"
    2021/05/08 17:26:09 DEBUG : fs cache: adding new entry for parent of "Local:file_local.txt", "Local:/Users/me/PhpstormProjects/rpi/tmp"
    2021/05/08 17:26:09 DEBUG : file_1.txt: Excluded
    Local
    2021/05/08 17:26:09 DEBUG : 4 go routines active

This also works,

$ rclone -vvvvv cat Union-epall-remote:file_remote.txt
    2021/05/08 17:39:01 DEBUG : Using config file from "/Users/me/.config/rclone/rclone.conf"
    2021/05/08 17:39:01 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "-vvvvv" "cat" "Union-epall-remote:file_remote.txt"]
    2021/05/08 17:39:01 DEBUG : Creating backend with remote "Union-epall-remote:file_remote.txt"
    2021/05/08 17:39:01 DEBUG : Creating backend with remote "GDrive:test/dir"
    2021/05/08 17:39:01 DEBUG : Creating backend with remote "Local:"
    2021/05/08 17:39:01 DEBUG : fs cache: renaming cache item "Local:" to be canonical "Local:/Users/me/PhpstormProjects/rpi/tmp"
    2021/05/08 17:39:01 DEBUG : Creating backend with remote "Local:/file_remote.txt"
    2021/05/08 17:39:02 DEBUG : Creating backend with remote "GDrive:test/dir/file_remote.txt"
    2021/05/08 17:39:03 DEBUG : fs cache: adding new entry for parent of "GDrive:test/dir/file_remote.txt", "GDrive:test/dir"
    2021/05/08 17:39:03 DEBUG : union root 'file_remote.txt': actionPolicy = *policy.EpAll, createPolicy = *policy.EpMfs, searchPolicy = *policy.EpAll
    2021/05/08 17:39:03 DEBUG : file_1.txt: Excluded
    REMOTE
    2021/05/08 17:39:03 DEBUG : 8 go routines active

Now, I know I can also pass a local path and that works,

$ rclone -vvvvv cat ./file_local.txt
    2021/05/08 17:25:17 DEBUG : Using config file from "/Users/me/.config/rclone/rclone.conf"
    2021/05/08 17:25:17 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "-vvvvv" "cat" "./file_local.txt"]
    2021/05/08 17:25:17 DEBUG : Creating backend with remote "./file_local.txt"
    2021/05/08 17:25:17 DEBUG : fs cache: adding new entry for parent of "./file_local.txt", "/Users/me/PhpstormProjects/rpi/tmp"
    2021/05/08 17:25:17 DEBUG : file_1.txt: Excluded
    Local
    2021/05/08 17:25:17 DEBUG : 4 go routines active

So I tried updating my configuration to use an actual path and that seems to work,

[Union-epff-local]
type = union
upstreams = . GDrive:test/dir
search_policy = epff
cache_time = 1

Then tried reading a local and a remote file which works,

$ rclone -vvvvv cat Union-epff-local:file_local.txt$ 
    2021/05/08 18:09:49 DEBUG : Using config file from "/Users/me/.config/rclone/rclone.conf"
    2021/05/08 18:09:49 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "-vvvvv" "cat" "Union-epff-local:file_local.txt"]
    2021/05/08 18:09:49 DEBUG : Creating backend with remote "Union-epff-local:file_local.txt"
    2021/05/08 18:09:49 DEBUG : Creating backend with remote "GDrive:test/dir"
    2021/05/08 18:09:49 DEBUG : Creating backend with remote "."
    2021/05/08 18:09:49 DEBUG : fs cache: renaming cache item "." to be canonical "/Users/me/PhpstormProjects/rpi/tmp"
    2021/05/08 18:09:49 DEBUG : Creating backend with remote "file_local.txt"
    2021/05/08 18:09:49 DEBUG : fs cache: adding new entry for parent of "file_local.txt", "/Users/me/PhpstormProjects/rpi/tmp"
    2021/05/08 18:09:49 DEBUG : Creating backend with remote "GDrive:test/dir/file_local.txt"
    2021/05/08 18:09:50 DEBUG : union root 'file_local.txt': actionPolicy = *policy.EpAll, createPolicy = *policy.EpMfs, searchPolicy = *policy.EpFF
    2021/05/08 18:09:50 DEBUG : file_1.txt: Excluded
    Local
    2021/05/08 18:09:50 DEBUG : 8 go routines active

$ rclone -vvvvv cat Union-epff-local:file_remote.txt$ 
    2021/05/08 18:09:55 DEBUG : Using config file from "/Users/me/.config/rclone/rclone.conf"
    2021/05/08 18:09:55 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "-vvvvv" "cat" "Union-epff-local:file_remote.txt"]
    2021/05/08 18:09:55 DEBUG : Creating backend with remote "Union-epff-local:file_remote.txt"
    2021/05/08 18:09:55 DEBUG : Creating backend with remote "GDrive:test/dir"
    2021/05/08 18:09:55 DEBUG : Creating backend with remote "."
    2021/05/08 18:09:55 DEBUG : fs cache: renaming cache item "." to be canonical "/Users/me/PhpstormProjects/rpi/tmp"
    2021/05/08 18:09:55 DEBUG : Creating backend with remote "file_remote.txt"
    2021/05/08 18:09:55 DEBUG : fs cache: renaming cache item "file_remote.txt" to be canonical "/Users/me/PhpstormProjects/rpi/tmp/file_remote.txt"
    2021/05/08 18:09:55 DEBUG : Creating backend with remote "GDrive:test/dir/file_remote.txt"
    2021/05/08 18:09:56 DEBUG : fs cache: adding new entry for parent of "GDrive:test/dir/file_remote.txt", "GDrive:test/dir"
    2021/05/08 18:09:56 DEBUG : union root 'file_remote.txt': actionPolicy = *policy.EpAll, createPolicy = *policy.EpMfs, searchPolicy = *policy.EpFF
    2021/05/08 18:09:56 DEBUG : file_1.txt: Excluded
    REMOTE
    2021/05/08 18:09:57 DEBUG : 8 go routines active

I was going to open this as an issue on Github but it asks for a link to the forum post. So I'm just documenting it here.

6

This looks like rclone is getting confused with the Local: backend...

I expect it will work if you put an actual path in

So Local:/tmp

7

Interesting that it works with ls but not cat.

However, it looks like using Local:./ works for what I want here.

For reference, I am in,

/Users/me/PhpstormProjects/rpi/tmp

Thanks!

8

What is happening is that when you are doing

Rclone is trying to find that file at the root of your file system.

Do

Is an OK workaround!

Personally I wouldn't put relative paths in rclone config but each to their own!

9

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.