Rclone union fails when one remote is offline

[brimur]

What is the problem you are having with rclone?

Using a local NAS remote via webdav and a Box.com cloud remote as union remotes with the same data. When the NAS is offline the union remote fails. Doing a directory listing of the Box cloud remote works fine thought. Want to use the Union remote as a load balancer / failover so that the NAS is used when online but the cloud is used when the NAS is offline (save energy costs)

What is your rclone version (output from rclone version)

rclone v1.55.1
- os/type: linux
- os/arch: amd64
- go/version: go1.16.3
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Box.com

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone lsd Union:
2021/11/22 10:36:56 Failed to create file system for "Union:": read metadata failed: Propfind "http://192.168.200.6:5005/Media": dial tcp 192.168.200.6:5005: connect: no route to host

The rclone config contents with secrets removed.

[Union]
type = union
upstreams = nas:/Media Chunk:
search_policy = epff
cache_time = 1

A log from the command with the -vv flag

Paste  log here

[Animosity022]

I'm not sure there is much to do about that as remotes are supposed to be online.

[brimur]

Thanks. My understanding was that it just used the first one to respond but you are saying they all need to be online also so probably not going to work then.

Any idea how I could achieve a fail over in this situation using rclone remotes? As in use the NAS but use the cloud if the NAS is offline?

[Animosity022]

Some reading maybe:

Union "Fail Over"

In production I don't use any vfs cache on my union yet... I think the union should have built-in failover yeah...otherwise what's the point? It should work like this: file is opened, it will follow the union policy, in my case it's random...so it will randomly pick a remote and try to open a file, if that fails for whatever reason, it should call another random remote, and only give up if all remotes fail... It would be sensible to mark failed files in the remotes as "dead" for 24 hours. I'm…

I don't think so offhand as there isn't a way I know to do it.

Maybe @ncw has an idea as he tends to think out of the box as rclone is expecting remotes to exist.

[brimur]

Thanks, my use case is similar to the feature request except I would not use random. Using epff for search/action should just wait for the first remote to respond with the file and obv only online ones will respond.

[brimur]

I wonder if I mount the remotes and then use mergerfs itself with those mounts would that work

[ncw (Nick Craig-Wood)]

The union backend is more like a RAID0 drive array - when any one drive is down, the whole thing is down. You are looking for more like a RAID1 drive array which can survive drive failure.

This would be a nice addon to union, or maybe a different backend, but it doesn't do it yet

[brimur]

Thanks @ncw. Are there any plans to add it? I have it kinda working now by mounting both and using mergerfs but its causing other issues.

[ncw (Nick Craig-Wood)]

I think there is an issue you can subscribe to but I can't find it right now