Webdav duplicate objects

Help and Support
1

What is the problem you are having with rclone?

I am using rclone to sync from local disk to four webdav remotes. Three remotes are fine, but one is giving me duplicate objects issues. I have encountered these before on Google Drive and running rclone dedupe fixed it, but I don't know why this would happen with webdav. The logs suggest there are sometimes even 6 or more files with the same name!

2020/06/20 21:02:06 NOTICE: path/to/file1: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file2: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file2: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file2: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file2: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file2: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file2: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file3: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file3: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file3: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file3: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file3: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file3: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file3: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file4: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file4: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file4: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file4: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file4: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file4: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file5: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file5: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file5: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file5: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file5: Duplicate object found in destination - ignoring
2020/06/20 21:02:06 NOTICE: path/to/file5: Duplicate object found in destination - ignoring

The rclone ls command does not show any duplicate files, it shows every file in the log above just once.

What is your rclone version (output from rclone version)

1.52.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Alpine (in a docker container), 64 bit

Which cloud storage system are you using? (eg Google Drive)

webdav (to nginx webdav server)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync /data cryptremote: --transfers 1 --order-by size,asc --exclude-from /excludes.txt

The rclone config contents with secrets removed.

type = webdav
vendor = other
user = **
pass = **
url = **

[cryptremote]
type = crypt
filename_encryption = standard
directory_name_encryption = true
password = **
remote = backend:crypt

A log from the command with the -vv flag
2

There's really no magic here as you'd have to delete the duplicates for rclone to handle it.

You can run rclone dedupe and see how that works out, but I'd be mindful as I've only ever used it with Google Drive.

Ensure you have backups and validate what you are deleting before doing anything would be my suggestion.

3

What is the remote WebDAV server serving? Files off disk or something else?

Did you try rclone lsf of the directories to see if you can see duplicates?

You could try the lsf with -vv --dump bodies and examine the server responses.

4

Just from a regular ext4 disk.

I did notice rclone said the remote time may be incorrect. It was fine on the host, but in the docker container where the webdav server runs I saw this strange behaviour: youtube. com/watch?v=n58dzphTqzs . Basically, every time I run the date command it shows a different date and time between roughly Feb 19, 1970 and Mar 2, 1970. It only happens on this system, not any other systems where I run the same image. I will first try to solve this before taking more of your time in case it is the cause of the issue

5

Dump bodies will reveal the problem very quickly :smiley:

6

Http://transfer.derkad.es/d8125762ea7e

Sorry for the malformed link and no direct attachment, discourse won't let me do that yet

7

I fixed your link and upgraded your account.

8

I can't see any duplicates in there - not in the listing nor in the xml responses...

Does this print anything?

rclone lsf -R remote: | sort | uniq -d

That will print any duplicates if there are any.

9

That didn't print anything, but running it in a different directory did. I guess the duplicates in the original directory have disappeared?

I am syncing hundreds of thousands of files and every time I run the sync command it spams my terminal with duplicate objects errors, filling it up in seconds.

10

:confused:

If you can get me an unredacted log with -vv --dump bodies of it actually producing duplicates then I can check it out. You can email it to nick@craig-wood.com if you want with a link to this forum post.

11

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.