Union action_policy = epall updates only one existing copy when atomic .partial uploads are enabled

Suspected Bug
1

What is the problem you are having with rclone ?

I am using an rclone union remote over multiple independent Hetzner Storage Boxes accessed through SFTP.

The union is configured with :

action_policy = epall
create_policy = mfs
search_policy = ff

When the same logical file already physically exists on multiple union upstreams, I expect action_policy = epall to update all existing copies.

However, with current rclone versions and the default atomic .partial upload mechanism, only one physical copy is updated. The other existing copies remain unchanged.

The problem is reproducible with two SFTP upstreams.

For example, before the update :

H:  file.bin = 65536 bytes
H1: file.bin = 65536 bytes

After updating the file through the union without --inplace :

H:  file.bin = 131072 bytes
H1: file.bin = 65536 bytes

The debug log shows that rclone creates a temporary file :

file.bin.<hash>.partial

The temporary path does not exist on either upstream, so the union appears to process it using the create_policy.

Because create_policy = mfs, the temporary file is created on only one upstream. It is then renamed to the final filename on that upstream only.

The other existing physical copy is therefore not updated, even though action_policy = epall is configured.

The same test works correctly when I add --inplace :

H:  file.bin = 131072 bytes
H1: file.bin = 131072 bytes

It also works correctly with an older rclone v1.60.1-DEV executable, before atomic SFTP .partial uploads were enabled by default.

Deleting a duplicated object through the union also correctly removes the object from both upstreams, so epall is active for deletion.

I have also tested the creation of new files:

  • 64 files with rclone v1.74.4 and 8 parallel transfers

  • 128 files with the preserved v1.60.1-DEV executable and 8 parallel transfers

No duplicate was created in either test. Each source file resulted in exactly one physical destination file.

Therefore, I cannot currently prove that rclone created the initial duplicate files in my production storage.

However, I can reproduce that once duplicate physical copies exist, current atomic .partial uploads can update only one copy and leave the other copies stale.

In my production environment, a read-only audit across 12 SFTP upstreams found :

Duplicate logical path groups: 8083
Extra physical copies: 31513
Groups with identical sizes: 6314
Groups with different sizes: 1769
Minimum additional storage used by identical-size copies:
20946004960021 bytes

The union and crypt views only expose one copy because search_policy = ff.

No .zfs paths were found in the decrypted destination during the current audit, so these physical duplicate paths are not explained by .zfs/snapshot entries.

My questions are :

  1. Is this behavior expected when atomic .partial uploads are used with a union remote ?

  2. Should action_policy = epall replace all existing copies of the final object ?

  3. Is --inplace currently required to preserve epall update semantics ?

  4. Should the union backend disable or specially handle PartialUploads when the action policy selects multiple upstreams ?

  5. Could this behavior explain why duplicated physical copies gradually develop different sizes or contents ?

Run the command 'rclone version' and share the full output of the command.

rclone v1.74.4
- os/version: ubuntu 24.04 (64 bit)
- os/kernel: 6.8.0-134-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.26.5
- go/linking: static
- go/tags: none

I also performed a comparison with the exact older executable previously used by my production sync :

rclone v1.60.1-DEV
- os/version: ubuntu 24.04 (64 bit)
- os/kernel: 6.8.0-134-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.2
- go/linking: dynamic
- go/tags: none

SHA-256 of the preserved old executable :

71e94910f24d82f7ef3aefe39bb4df757e7a865712ebea6cf15b5b805ce1cc27

Which cloud storage system are you using ?

Hetzner Storage Boxes accessed through the rclone SFTP backend.

The production architecture is :

12 independent Hetzner Storage Boxes -> rclone union -> rclone crypt

The reproduction uses two independent Hetzner Storage Boxes:

H  = SFTP Storage Box
H1 = SFTP Storage Box

The command you were trying to run

First, I created an identical file directly on both upstreams :

dd if=/dev/urandom \
  of=/tmp/epall-test.bin \
  bs=64K \
  count=1 \
  status=none

rclone copyto \
  /tmp/epall-test.bin \
  H:_rclone_union_probe/epall-test/file.bin

rclone copyto \
  /tmp/epall-test.bin \
  H1:_rclone_union_probe/epall-test/file.bin

Both physical files initially had the same size :

H:  65536 file.bin
H1: 65536 file.bin

I then replaced the local file with a larger file :

dd if=/dev/urandom \
  of=/tmp/epall-test.bin \
  bs=128K \
  count=1 \
  status=none

Command reproducing the issue :

rclone copyto \
  /tmp/epall-test.bin \
  UnionDuplicateProbe:epall-test/file.bin \
  -vv \
  --log-file /root/union-probe-epall-update.log

Result :

H:  131072 file.bin
H1:  65536 file.bin

Control test with --inplace :

rclone copyto \
  /tmp/epall-test.bin \
  UnionDuplicateProbe:epall-test/file.bin \
  --inplace \
  -vv \
  --log-file /root/union-probe-epall-inplace.log

Result :

H:  131072 file.bin
H1: 131072 file.bin

Control test with rclone v1.60.1-DEV :

/root/rclone-v1.60.1-DEV copyto \
  /tmp/epall-old.bin \
  UnionDuplicateProbe:epall-old/file.bin \
  -vv \
  --log-file /root/union-probe-epall-old-update.log

Before this test, both upstream copies were 65536 bytes.

Result after the update :

H:  196608 file.bin
H1: 196608 file.bin

Delete control test :

rclone deletefile \
  UnionDuplicateProbe:delete-test/file.bin \
  -vv \
  --log-file /root/union-probe-epall-delete.log

The file was removed from both H and H1.

The rclone config contents with secrets removed.

[H]
type = sftp
host = XXX
user = XXX
port = 23
pass = XXX
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r

[H1]
type = sftp
host = XXX
user = XXX
port = 23
pass = XXX
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r

[UnionDuplicateProbe]
type = union
cache_time = 120
upstreams = H:_rclone_union_probe H1:_rclone_union_probe
action_policy = epall
create_policy = mfs
search_policy = ff

Production union and crypt configuration :

[HetznerUnion]
type = union
upstreams = H11:PMSNew H10:PMSNew H9:PMSNew H8:PMSNew H7:PMSNew H6:PMSNew H5:PMSNew H4:PMSNew H3:PMSNew H2:PMSNew H1:PMSNew H:PMSNew
create_policy = mfs

[HetznerCloud]
type = crypt
remote = HetznerUnion:
password = XXX
filename_encoding = base32768

The production union uses the default values for the policies that are not explicitly present :

action_policy = epall
search_policy = ff

A log from the command with the -vv flag

Relevant extract from the update without --inplace :

2026/07/20 13:22:31 DEBUG : rclone: Version "v1.74.4" starting with parameters ["rclone" "copyto" "/tmp/rclone-union-probe-batch/file-01.bin" "UnionDuplicateProbe:batch/file-01.bin" "-vv" "--log-file" "/root/union-probe-epall-update.log"]

2026/07/20 13:22:31 DEBUG : Creating backend with remote "UnionDuplicateProbe:batch/"
2026/07/20 13:22:31 DEBUG : Creating backend with remote "H1:_rclone_union_probe"
2026/07/20 13:22:31 DEBUG : Creating backend with remote "H:_rclone_union_probe"

2026/07/20 13:22:32 DEBUG : union root 'batch': actionPolicy = *policy.EpAll, createPolicy = *policy.Mfs, searchPolicy = *policy.FF

2026/07/20 13:22:32 DEBUG : file-01.bin: size = 131072 (Local file system at /tmp/rclone-union-probe-batch)
2026/07/20 13:22:32 DEBUG : file-01.bin: size = 65536 (union root 'batch')
2026/07/20 13:22:32 DEBUG : file-01.bin: Sizes differ

2026/07/20 13:22:32 DEBUG : file-01.bin.627753be.partial: size = 131072 OK

2026/07/20 13:22:32 DEBUG : sftp://USER@HOST:23/_rclone_union_probe/batch: Shell path "/home/_rclone_union_probe/batch/file-01.bin.627753be.partial"

2026/07/20 13:22:33 DEBUG : file-01.bin.627753be.partial: renamed to: file-01.bin

2026/07/20 13:22:33 INFO  : file-01.bin: Copied (replaced existing)

Transferred:          128 KiB / 128 KiB, 100%
Transferred:            1 / 1, 100%
Elapsed time:         0.5s

Relevant extract from the successful test with --inplace :

2026/07/20 13:38:14 DEBUG : rclone: Version "v1.74.4" starting with parameters ["rclone" "copyto" "/tmp/epall-inplace.bin" "UnionDuplicateProbe:epall-inplace/file.bin" "--inplace" "-vv" "--log-file" "/root/union-probe-epall-inplace.log"]

2026/07/20 13:38:15 DEBUG : union root 'epall-inplace': actionPolicy = *policy.EpAll, createPolicy = *policy.Mfs, searchPolicy = *policy.FF

2026/07/20 13:38:15 DEBUG : file.bin: size = 65536 (union root 'epall-inplace')
2026/07/20 13:38:15 DEBUG : file.bin: size = 131072 OK

2026/07/20 13:38:15 INFO  : epall-inplace.bin: Copied (replaced existing) to: file.bin

Physical result after the --inplace test :

H:
131072 2026-07-20 13:38:13 file.bin

H1:
131072 2026-07-20 13:38:13 file.bin

Relevant extract from the successful test with rclone v1.60.1-DEV :

2026/07/20 13:38:43 DEBUG : rclone: Version "v1.60.1-DEV" starting with parameters ["/root/rclone-v1.60.1-DEV" "copyto" "/tmp/epall-old.bin" "UnionDuplicateProbe:epall-old/file.bin" "-vv" "--log-file" "/root/union-probe-epall-old-update.log"]

2026/07/20 13:38:45 DEBUG : union root 'epall-old': actionPolicy = *policy.EpAll, createPolicy = *policy.Mfs, searchPolicy = *policy.FF

2026/07/20 13:38:45 INFO  : epall-old.bin: Copied (replaced existing) to: file.bin

Physical result :

H:
196608 2026-07-20 13:38:43 file.bin

H1:
196608 2026-07-20 13:38:43 file.bin

I can provide the complete anonymized logs if required.
Thanks.