Union - sync files that exist on one remote but not the other

Help and Support
1

What is the problem you are having with rclone?

I have set up a union remote consisting of two remotes. I am running a copy/sync command from remoteA/dir1 (source - a pCloud remote) to the union of remoteA/dir2 (union1 - the same pCloud remote) and remoteB/dir2 (union2 - an SMB remote).

If a file exists on source but neither union1 or union2, the file gets copied to both union1 and union2. However, if the file exists on either union1 or union2, the file doesn't get copied to the path it doesn't exist in. How can I configure the union so all files are copied to all remotes in the union? I already have every policy set to "all".

Run the command 'rclone version' and share the full output of the command.

  • rclone v1.70.3
  • os/version: debian 12.11
  • os/kernel: 6.1.142-2 (armv5tel)
  • os/type: linux
  • os/arch: arm (ARMv5 compatible, no hardfloat)
  • go/version: go1.24.4
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Union with a combination of smb and pCloud.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

/usr/bin/rclone --skip-links copy pcloud:Documents backups:Darren_Documents -vv

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[backups]
type = union
upstreams = pcloud:Backups beestation:home/Files/backups
create_policy = all
action_policy = all
search_policy = all

[beestation]
type = smb
host = XXX
user = XXX
pass = XXX

[pcloud]
type = pcloud
client_id = XXX
client_secret = XXX
token = XXX
hostname = api.pcloud.com
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the -vv flag

2025/08/04 13:07:57 DEBUG : rclone: Version "v1.70.3" starting with parameters ["/usr/bin/rclone" "--skip-links" "copy" "pcloud:Documents" "backups:Darren_Documents" "-vv"]
2025/08/04 13:07:57 DEBUG : Creating backend with remote "pcloud:Documents"
2025/08/04 13:07:57 DEBUG : Using config file from "/root/.rclone.conf"
2025/08/04 13:07:59 DEBUG : Creating backend with remote "backups:Darren_Documents"
2025/08/04 13:07:59 DEBUG : Creating backend with remote "beestation:home/Files/backups"
2025/08/04 13:07:59 DEBUG : Creating backend with remote "pcloud:Backups"
2025/08/04 13:07:59 DEBUG : smb://darren@192.168.10.2:445/home/Files/backups: Using root directory "home/Files/backups"
2025/08/04 13:07:59 DEBUG : Creating backend with remote "beestation:home/Files/backups/Darren_Documents"
2025/08/04 13:07:59 DEBUG : smb://darren@192.168.10.2:445/home/Files/backups/Darren_Documents: Using root directory "home/Files/backups/Darren_Documents"
2025/08/04 13:08:00 DEBUG : Creating backend with remote "pcloud:Backups/Darren_Documents"
2025/08/04 13:08:01 DEBUG : union root 'Darren_Documents': actionPolicy = *policy.All, createPolicy = *policy.All, searchPolicy = *policy.All
2025/08/04 13:08:02 DEBUG : 5071-0751-4693-tickets.pdf: Size and modification time the same (differ by 0s, within tolerance 1s)
2025/08/04 13:08:02 DEBUG : 5071-0751-4693-tickets.pdf: Unchanged skipping
<snip>
2025/08/04 13:08:32 DEBUG : union root 'Darren_Documents': Waiting for transfers to finish
2025/08/04 13:08:32 INFO  : There was nothing to transfer
2025/08/04 13:08:32 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:               361 / 361, 100%, Listed 844
Elapsed time:        31.4s

2025/08/04 13:08:32 DEBUG : 23 go routines active
2025/08/04 13:08:32 DEBUG : smb://darren@192.168.10.2:445/home/Files/backups: Closing 1 unused connections
2025/08/04 13:08:32 DEBUG : smb://darren@192.168.10.2:445/home/Files/backups/Darren_Documents: Closing 7 unused connections
2

I'd think that's by design as you are expecting two remotes to be in sync, but they aren't and I don't think union would fix that as it would be expensive to query every remote to find if a file exists.

You can sync from remote1 to remote 2 with bisync or something or figure out why they are getting out of sync.

3

You can not. You union is "broken" as with create_policy = all it is illegal union state.

Another question is why you have some file only on one remote? If you side load it (outside union) then do not expect rclone to fix it on the fly.

4

Why can't you use all there?

5

I’m not actually seeing files disappear (except artificially during some testing), I’m just trying to protect against such a scenario. I’ll go back to syncing things via separate commands then. Thanks.

6

You can use all - no problem. But it means that every file copied to such union should exist on all its members.

If a file is missing from some union member it is either result of some problem or side loading.

It is RAID1 - sort of. In ideal world I imagine rclone to be able to self heal such "errors". But it would be another project:)

Main reason for union introduction was to allow combining free space from multiple remotes. It works pretty well. But some other policies (like all) were added for "completeness" only. Without spending time on perfecting them.

7

It's a valid use case, but you don't like it.

I don't try to figure out everyone's use case but it's a valid command and nothing is 'broken' was the point.

8

Let's say we have two union members, A and B. With create_policy = all when you transfer file1 to this union it should result with:

A: file1
B: file1

Now if you side load file2 in the way that union state is as follows:

A: file1,file2
B: file1

It creates invalid union state. It is "broken". How it will behave is not documented at all. Uncharted waters:)

Similarly like simple RAID1 when one file copy is missing or corrupted. Now what happens next depends on implementation. Some systems like ZFS will notice that something is wrong when you try to access file2 and will fix missing copy on the fly. Other implementations will act as nothing happens until you run explicit check/scrub.

Rclone assumes that when you use given policy your union state reflects it fully and does not have any means today to fix it in case if not.

9

I feel we're going off on a tangent here. The point was the flag was valid and the config was valid.

If someone does something to purposely break their setup, you can't really fix that.

rm -rf /*
10

I think the problem is that even if all is valid (config, commands) rclone today does not have any mechanism to rectify problematic union state.

And such state can be result of something as trivial as network outage. Some union members will have file already transferred but some not. Does not have to be malicious action.