Sync on WebDAV does not respect --ignore-checksum

mgutt · June 21, 2020, 11:51pm

I tried to sync two folders and make it faster by ignoring the checksums, but as it takes 2 minutes to sync 4 files it is obviously creating hashes.

Command:

rclone sync -P --ignore-checksum -vv /mnt/user/Music/ /mnt/disks/owncube/Musik/

Results:

2020/06/22 01:34:13 DEBUG : rclone: Version "v1.52.1" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "sync" "-P" "--ignore-checksum" "-vv" "/mnt/user/Music/" "/mnt/disks/owncube/Musik/"]
2020/06/22 01:34:13 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2020-06-22 01:34:13 DEBUG : Local file system at /mnt/disks/owncube/Musik/: Waiting for checks to finish
2020-06-22 01:34:13 DEBUG : album1.zip: Modification times differ by 25h51m58s: 2020-01-05 23:28:0
7 +0100 CET, 2020-01-07 01:20:05 +0100 CET
2020-06-22 01:34:13 DEBUG : album2.zip: Modification times differ by 44h49m1s: 2019-11-20 13:54:50 +0100 CET, 2019-11-22 1
0:43:51 +0100 CET
2020-06-22 01:34:13 DEBUG : album3.zip: Modification times differ by 44h30m8s: 2019-11-20 13:54:49 +0100 CET, 2019-11-
22 10:24:57 +0100 CET
2020-06-22 01:34:13 DEBUG : album4.zip: Modification times differ by 34h56m37s: 2019-11-20 13:54:47 +0100 CET, 2019-11-
22 00:51:24 +0100 CET
2020-06-22 01:34:48 DEBUG : album1.zip: MD5 = 8df15abbe51f4b120532f5540624f469 OK
2020-06-22 01:34:51 INFO  : album1.zip: Updated modification time in destination
2020-06-22 01:34:51 DEBUG : album1.zip: Unchanged skipping
2020-06-22 01:35:32 DEBUG : album4.zip: MD5 = f2eb96609812acb01ca7a84d37a8388d OK
2020-06-22 01:35:32 INFO  : album4.zip: Updated modification time in destination
2020-06-22 01:35:32 DEBUG : album4.zip: Unchanged skipping
2020-06-22 01:35:40 DEBUG : album2.zip: MD5 = beeeec14ee6f86161141eeae053dd1f8 OK
2020-06-22 01:35:40 INFO  : album2.zip: Updated modification time in destination
2020-06-22 01:35:40 DEBUG : album2.zip: Unchanged skipping
2020-06-22 01:36:04 DEBUG : album3.zip: MD5 = bdfe3ef2030836b328b6b47271ebd3f9 OK
2020-06-22 01:36:04 INFO  : album3.zip: Updated modification time in destination
2020-06-22 01:36:04 DEBUG : album3.zip: Unchanged skipping
2020-06-22 01:36:04 DEBUG : Local file system at /mnt/disks/owncube/Musik/: Waiting for transfers to finish
2020-06-22 01:36:04 DEBUG : Waiting for deletions to finish
2020-06-22 01:36:04 INFO  : There was nothing to transfer
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 4 / 4, 100%
Elapsed time:      1m51.8s
2020/06/22 01:36:04 INFO  : 
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 4 / 4, 100%
Elapsed time:      1m51.8s

2020/06/22 01:36:04 DEBUG : 2 go routines active

Source: Unraid 6.8.3 with rclone docker v1.52.1
Target: Nextcloud 16.0.1

P.S. Although it says "updated moditication time" the time hasn't be changed in the target. Is there a log available to check the posted command on the server and its response?

Animosity022 · June 21, 2020, 11:52pm

How are you trying to copy? By what? Size? Modtime?

Animosity022 · June 22, 2020, 12:07am

    --ignore-checksum                      Skip post copy check of checksums

That skips the destination.

mgutt · June 22, 2020, 12:29am

Size and Modtime as I only disabled the checksum check.

You mean it skips the check, but still creates hashes? Really?!

Animosity022 · June 22, 2020, 12:30am

No, you didn't. You can use --size-only or mod time.

mgutt · June 22, 2020, 7:06am

This means it generates the source checksum, transfers the file, then generates the target checksum and if they do not fit it will be ignored because of the --ignore-checksum flag? So the flag has no influence on already existing files? Ok, but why are checksums generated at all? I mean, I did not enabled them through --checksum or --etag-hash.

Edit: ok it seems webdav has checksums enabled by default as there exist the flag --no-checksum:

Isn't this a conflict with the default behaviour of rclone not to use checksum comparison?

Animosity022 · June 22, 2020, 10:07am

Rclone by default tries to use checksums to validate a copy or sync from a supported remote to another.

You change that behavior by using the size only or mod time options when you copy a file.

If you ignore checksums, it doesn't check the destination:

felix@gemini:~$ rclone copy /etc/hosts GD:  --ignore-checksum    -vv
2020/06/22 06:05:33 DEBUG : rclone: Version "v1.52.1" starting with parameters ["rclone" "copy" "/etc/hosts" "GD:" "--ignore-checksum" "-vv"]
2020/06/22 06:05:33 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2020/06/22 06:05:33 DEBUG : fs cache: renaming cache item "/etc/hosts" to be canonical "/etc"
2020/06/22 06:05:34 DEBUG : hosts: Size and modification time the same (differ by -273.054µs, within tolerance 1ms)
2020/06/22 06:05:34 DEBUG : hosts: Unchanged skipping
2020/06/22 06:05:34 INFO  :
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.0s

2020/06/22 06:05:34 DEBUG : 4 go routines active

and if I remove that.

felix@gemini:~$ rclone copy /etc/hosts GD:  -vv
2020/06/22 06:05:57 DEBUG : rclone: Version "v1.52.1" starting with parameters ["rclone" "copy" "/etc/hosts" "GD:" "-vv"]
2020/06/22 06:05:57 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2020/06/22 06:05:57 DEBUG : fs cache: renaming cache item "/etc/hosts" to be canonical "/etc"
2020/06/22 06:05:57 DEBUG : hosts: Need to transfer - File not found at Destination
2020/06/22 06:05:58 DEBUG : hosts: MD5 = 157c632175c4251012e651322736f0ae OK
2020/06/22 06:05:58 INFO  : hosts: Copied (new)
2020/06/22 06:05:58 INFO  :
Transferred:   	       413 / 413 Bytes, 100%, 379 Bytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         1.0s

2020/06/22 06:05:58 DEBUG : 5 go routines active

You can use --size-only to just check size if the remotes don't support or you don't want to use checksums.

mgutt · June 22, 2020, 10:50am

Now compare your result:

rclone copy /etc/hosts GD:  --ignore-checksum    -vv
...
2020/06/22 06:05:34 DEBUG : hosts: Size and modification time the same (differ by -273.054µs, within tolerance 1ms)
2020/06/22 06:05:34 DEBUG : hosts: Unchanged skipping

with mine:

rclone sync -P --ignore-checksum -vv /mnt/user/Music/ /mnt/disks/owncube/Musik/
...
2020-06-22 01:34:48 DEBUG : album1.zip: MD5 = 8df15abbe51f4b120532f5540624f469 OK
2020-06-22 01:34:51 INFO  : album1.zip: Updated modification time in destination
2020-06-22 01:34:51 DEBUG : album1.zip: Unchanged skipping

Your command did not create an MD5 hash.

If this is the default why do we need the --checksum flag?

Skip based on checksum (if available) & size, not mod-time & size

And if checksum + size is the default, how to enable modtime + size if there only exists --size-only?!

No, because you can see in my logs that source and target contain the same files, but have wrong modtimes. So I only want to update the modtime. If I use --size-only nothing would happen. If I would rely on size and modtime it would re-upload everything.

Animosity022 · June 22, 2020, 11:55am

So you are trying to not use checksums, match on size and update mod time if the size is the same?

mgutt · June 22, 2020, 2:50pm

Correct and the question is: Why isn't --ignore-checksum working?

Animosity022 · June 22, 2020, 2:53pm

Can you try with the latest beta?

Looks to be what you want in that issue.

ncw · June 22, 2020, 4:59pm

Ignore checksum is specifically for skipping the post copy checksum check.

It doesn't stop the check when rclone sees sizes match, but modtimes don't so it does a checksum check to see whether the files are really identical before setting the modtime.

The --refresh-times flag may be what you want which will skip the checksum in this case and just set the modtime of the remote file.

mgutt · June 23, 2020, 7:40am

Yes, I watch that Beta already, but I'm not able to use it as long its not in the official beta channel as I'm using an Unraid server. But it seems it will not solve the issue as Nextcloud/Owncloud does not support updating the modification time.

I would say this is a bug, because the default is not to use checksums. If I want to use them in such a situation I could enable them through --checksum.

ncw · June 23, 2020, 3:54pm

If you set vendor to other then this will disable both the modtimes and the checksums which will probably fix your issue.

mgutt · June 23, 2020, 4:07pm

Ok thank you. But did you read the link? Nextcloud (and Owncloud) does not support file modification updates through PROPPATCH or how does rclone update the timestamp?

ncw · June 23, 2020, 4:11pm

I did

Rclone will re-upload the file if it needs to change the modification time in a sync.

system · August 23, 2020, 12:11pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.