Rclone sync between two WebDav issues

What is the problem you are having with rclone?

I am syncing about 800 GB and it always slows down and becomes unresponsive after about 100 GB. I am syncing between two Nextcloud instances using WebDav. The source Nextcloud has local storage, the target (local for rclone) Nextcloud is using Wasabi S3 storage.

When it stops copying files successfully, I restart rclone. The debug log shows that most files have a different modification date (one is today, when the file was copied, and the other is the date from the original server). This causes rclone to start checking or copying the files again which makes it impossible to finish the sync.

This is after I had locking issues within Nextcloud (target), so I temporarily disabled that. I also reduced the number of checkers and transfers.

What is your rclone version (output from rclone version)

rclone v1.50.2

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux, 64 Bits

Which cloud storage system are you using? (eg Google Drive)

Nextcloud (with Local Storage) and Nextcloud (with External S3 - Wasabi)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy NC: hetzcloud: -P -vvvvv --transfers 8 --checkers 8

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

2020-01-28 11:18:38 INFO : 2005/fil1.JPG: Copied (replaced existing)
2020-01-28 11:18:41 INFO : 2005/file2.bmp: Copied (replaced existing)
2020-01-28 11:18:41 DEBUG : 2008/dir1/file3.AVI: Modification times differ by 102831h28m45s: 2008-05-03 18:28:14 +0000 GMT, 2020-01-26 09:56:59 +0000 GMT
2020-01-28 11:18:45 INFO : 2005/file4.jpg: Copied (replaced existing)
2020-01-28 11:18:45 DEBUG : 2008/dir1/file5.JPG: Modification times differ by 102831h28m13s: 2008-05-03 18:28:48 +0000 GMT, 2020-01-26 09:57:01 +0000 GMT
2020-01-28 11:18:47 INFO : 2005/file6.JPG: Copied (replaced existing)
2020-01-28 11:18:48 INFO : 2005/file7.JPG: Copied (replaced existing)

That is strange. Can you share a debug log?

And that fixes the slow speed? I wonder if some transfers have got blocked maybe.

It looks like your nextcloud isn't configured to store modification times. I suggest you use the --size-only flag to complete the transfer.

Thanks for the response.

Sorry - no debug log. And yes, after restarting it the speed is better. I think you're right that some transfers got blocked or hanged.

You are also right about the NC modification times. I did a sync from NC -> Wasabi directly, and then did a file scan on the external storage. With this, all the files come through correctly, but Nextcloud shows the Modified date as the date of the scan, even though the dates show correctly in Wasabi.

Do you happen to know how to configure Nextcloud to accept these modification times? I see tons of bugs about it on their forums, but no fixes...

I also see a number of posts and use cases for "update mod time without downloading". I saw someone made a binary for this (but in 2017). Has this been considered as a feature?

I don't I'm afraid. The base nextcloud supports it, but it may depend on the backend you are using.

I found a note in the code about this

// SetModTime might be possible
// https://stackoverflow.com/questions/3579608/webdav-can-a-client-modify-the-mtime-of-a-file
// ...support for a PROPSET to lastmodified (mind the missing get) which does the utime() call might be an option.
// For example the ownCloud WebDAV server does it that way.

Can you link me some more info about it?

Can you link me some more info about it?

Sorry - more information about what? The post I'm talking about is this one:

It seems like they did a code update and recompiled a binary to achieve the desired result.

Ah, that link wasn't about WebDAV. The WebDAV backend can't set modification times at the moment, though it might be possible with nextcloud/owncloud (see above).

Thanks for the response - I think this is a bit out of my reach though. If it's not currently possible with RClone then I will focus on updating the metadata directly in the NC database.

1 Like