`rclone moveto` errors for files in "chunker over union"

Suspected Bug
1

The problem

rclone moveto errors for files in "chunker over union".

'rclone version'

rclone v1.65.0
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 6.1.58+ (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.4
- go/linking: static
- go/tags: none

rclone backends used

chunker and union

The command

rclone moveto chunker:rclone.deb chunker:something.deb -vv

rclone config file

[chunker]
type = chunker
remote = union:
chunk_size = 1Mi

[union]
type = union
upstreams = /content/chunked/folder_1 /content/chunked/folder_2
action_policy = epff
create_policy = rand
search_policy = epall
cache_time = 604800

Steps to reproduce

curl "https://downloads.rclone.org/v1.65.0/rclone-v1.65.0-linux-amd64.deb" -o /content/rclone.deb
rclone move /content/rclone.deb chunker:
rclone  moveto chunker:rclone.deb chunker:something.deb -vv

Output of rclone moveto chunker:rclone.deb chunker:something.deb -vv

2023/12/24 23:22:18 DEBUG : rclone: Version "v1.65.0" starting with parameters ["rclone" "moveto" "chunker:rclone.deb" "chunker:something.deb" "-vv"]
2023/12/24 23:22:18 DEBUG : Creating backend with remote "chunker:rclone.deb"
2023/12/24 23:22:18 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/12/24 23:22:18 DEBUG : Creating backend with remote "union:rclone.deb"
2023/12/24 23:22:18 DEBUG : Creating backend with remote "/content/chunked/folder_2"
2023/12/24 23:22:18 DEBUG : Creating backend with remote "/content/chunked/folder_2/rclone.deb"
2023/12/24 23:22:18 DEBUG : Creating backend with remote "/content/chunked/folder_1"
2023/12/24 23:22:18 DEBUG : Creating backend with remote "/content/chunked/folder_1/rclone.deb"
2023/12/24 23:22:18 DEBUG : fs cache: adding new entry for parent of "/content/chunked/folder_1/rclone.deb", "/content/chunked/folder_1"
2023/12/24 23:22:18 DEBUG : union root 'rclone.deb': actionPolicy = *policy.EpFF, createPolicy = *policy.Rand, searchPolicy = *policy.EpAll
2023/12/24 23:22:18 DEBUG : Reset feature "ListR"
2023/12/24 23:22:18 DEBUG : Creating backend with remote "chunker:"
2023/12/24 23:22:18 DEBUG : Creating backend with remote "union:"
2023/12/24 23:22:18 DEBUG : union root '': actionPolicy = *policy.EpFF, createPolicy = *policy.Rand, searchPolicy = *policy.EpAll
2023/12/24 23:22:18 DEBUG : Reset feature "ListR"
2023/12/24 23:22:18 ERROR : Attempt 1/3 failed with 1 errors and: "rclone.deb" has missing chunks
2023/12/24 23:22:18 ERROR : Attempt 2/3 failed with 1 errors and: "rclone.deb" has missing chunks
2023/12/24 23:22:18 ERROR : Attempt 3/3 failed with 1 errors and: "rclone.deb" has missing chunks
2023/12/24 23:22:18 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.0s

2023/12/24 23:22:18 DEBUG : 4 go routines active
2023/12/24 23:22:18 Failed to moveto: "rclone.deb" has missing chunks
2

Output of rclone tree /content/chunked

/
├── folder_1
│   ├── rclone.deb
│   ├── rclone.deb.rclone_chunk.002
│   ├── rclone.deb.rclone_chunk.004
│   ├── rclone.deb.rclone_chunk.009
│   ├── rclone.deb.rclone_chunk.013
│   ├── rclone.deb.rclone_chunk.015
│   ├── rclone.deb.rclone_chunk.016
│   ├── rclone.deb.rclone_chunk.017
│   ├── rclone.deb.rclone_chunk.018
│   └── rclone.deb.rclone_chunk.020
└── folder_2
    ├── rclone.deb.rclone_chunk.001
    ├── rclone.deb.rclone_chunk.003
    ├── rclone.deb.rclone_chunk.005
    ├── rclone.deb.rclone_chunk.006
    ├── rclone.deb.rclone_chunk.007
    ├── rclone.deb.rclone_chunk.008
    ├── rclone.deb.rclone_chunk.010
    ├── rclone.deb.rclone_chunk.011
    ├── rclone.deb.rclone_chunk.012
    ├── rclone.deb.rclone_chunk.014
    └── rclone.deb.rclone_chunk.019
3

I can replicate the issue as well - clearly it is a bug. rclone moveto always fails when file has multiple chunks.

BTW I use different union policies

action_policy = epall
create_policy = mfs
search_policy = ff
cache_time = 120

so I think it can be general chunker over union problem.

4

@kapitainsky I tried different configurations of union.

And the result of moveto changes based on SEARCH and ACTION policy.

Also, if there are no file chunks in the remote which contains the file metadata file, rclone moveto just renames the metadata file and exits successfully.

5

yeap - this is also my findings. but it is a bug 100%

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.