`rclone sync` (WebDAV/HiDrive backend) not working as expected

Help and Support
1

What is the problem you are having with rclone?

rclone sync did not work as expected. The description of the command says "[It d]oesn't transfer files that are identical on source and destination, testing by size and modification time or MD5SUM" (emphasis mine). As you can see below, only the file size is identical in this case (which IMHO should never be the only criterion and this behaviour isn't in line with the cited description).

Run the command 'rclone version' and share the full output of the command.

% rclone version
rclone v1.61.1
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.4.0-135-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.4
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

WebDAV/HiDrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

(The options "--interactive", "--progress" are explicitly used here to show that nothing is being transferred:)

# rclone --config /etc/rclone.conf sync --interactive --progress /virtual/shared/pbuilder-common/var/cache/pbuilder/dist-deps/3rdparty/ ionos13:users/[…]/pbuilder/3rdparty/
[…]
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:               268 / 268, 100%
Elapsed time:         0.3s
2023/01/26 11:08:07 NOTICE: 
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:               268 / 268, 100%
Elapsed time:         0.3s

For comparison:

# md5sum /virtual/shared/pbuilder-common/var/cache/pbuilder/dist-deps/3rdparty/focal-amd64/Packages /backup/webdav/ionos13/users/[…]/pbuilder/3rdparty/focal-amd64/Packages
8e27d2dad498288af988be8ea30c1e32  /virtual/shared/pbuilder-common/var/cache/pbuilder/dist-deps/3rdparty/focal-amd64/Packages
ce34a804c50498f415284c56c23c6549  /backup/webdav/ionos13/users/[…]/pbuilder/3rdparty/focal-amd64/Packages
# ll /virtual/shared/pbuilder-common/var/cache/pbuilder/dist-deps/3rdparty/focal-amd64/Packages /backup/webdav/ionos13/users/[…]/pbuilder/3rdparty/focal-amd64/Packages
-rw-r----- 1 sys-maint adm 17189 2023-01-26 09:19:17 /backup/webdav/ionos13/users/[…]/pbuilder/3rdparty/focal-amd64/Packages
-rw-r--r-- 1 sys-maint adm 17189 2023-01-26 11:03:32 /virtual/shared/pbuilder-common/var/cache/pbuilder/dist-deps/3rdparty/focal-amd64/Packages

The rclone config contents with secrets removed.

(Multiple entries; excerpt below:)

[ionos13]
type = webdav
url = https://webdav.hidrive.ionos.com
vendor = other
user = […]
pass = […]

A log from the command with the -vv flag

(Log is huge; the excerpt shows the file mentioned above – note that it does not matter whether you use "--interactive" and "--progress", but without those, the log is much more readable:)

2023/01/26 11:27:03 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "-vv" "--config" "/etc/rclone.conf" "sync" "/virtual/shared/pbuilder-common/var/cache/pbuilder/dist-deps/3rdparty/" "ionos13:users/[…]/pbuilder/3rdparty/"]
2023/01/26 11:27:03 DEBUG : Creating backend with remote "/virtual/shared/pbuilder-common/var/cache/pbuilder/dist-deps/3rdparty/"
2023/01/26 11:27:03 DEBUG : Using config file from "/etc/rclone.conf"
2023/01/26 11:27:03 DEBUG : fs cache: renaming cache item "/virtual/shared/pbuilder-common/var/cache/pbuilder/dist-deps/3rdparty/" to be canonical "/virtual/shared/pbuilder-common/var/cache/pbuilder/dist-deps/3rdparty"
2023/01/26 11:27:03 DEBUG : Creating backend with remote "ionos13:users/[…]/pbuilder/3rdparty/"
2023/01/26 11:27:03 DEBUG : found headers:
2023/01/26 11:27:03 DEBUG : fs cache: renaming cache item "ionos13:users/[…]/pbuilder/3rdparty/" to be canonical "ionos13:users/[…]/pbuilder/3rdparty"
[…]
2023/01/26 11:27:03 DEBUG : focal-amd64/Packages: Sizes identical
2023/01/26 11:27:03 DEBUG : focal-amd64/Packages: Unchanged skipping
[…]
2023/01/26 11:27:03 INFO  : There was nothing to transfer
[…]
2

Note that the webdav backend does not support hashes, so rclone falls back to checking sizes only. The webdav backend doesn't support setting modtimes either so rclone can't use them for syncing.

However you might find that the --update flag is what you want, provided your computer and the server times are reasonably in sync.

-u, --update Skip files that are newer on the destination

3

This worked, thanks! Could I suggest including an explicit clarification about the behaviour of sync in the absence of modification time related information?
(I still think that in such cases, -u should be the default in order to err on the side of caution. As this is also how rsync works by default using mounted WebDAV folders when you don't explicitly specify --size-only – maybe more than one rsync user will misinterpret the behaviour given the current description as cited in my first post?)

4

This could probably do with an entire section in the docs about how rclone considers files to be the same or different.

-u has its own problems so I don't want to make it the default here but making the docs more explicit is always good. You notices the problem from the debug log Sizes identical so that is a start at least!

5

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.