I am syncing about 800 GB and it always slows down and becomes unresponsive after about 100 GB. I am syncing between two Nextcloud instances using WebDav. The source Nextcloud has local storage, the target (local for rclone) Nextcloud is using Wasabi S3 storage.
When it stops copying files successfully, I restart rclone. The debug log shows that most files have a different modification date (one is today, when the file was copied, and the other is the date from the original server). This causes rclone to start checking or copying the files again which makes it impossible to finish the sync.
This is after I had locking issues within Nextcloud (target), so I temporarily disabled that. I also reduced the number of checkers and transfers.
What is your rclone version (output from rclone version)
rclone v1.50.2
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Linux, 64 Bits
Which cloud storage system are you using? (eg Google Drive)
Nextcloud (with Local Storage) and Nextcloud (with External S3 - Wasabi)
The command you were trying to run (eg rclone copy /tmp remote:tmp)
Sorry - no debug log. And yes, after restarting it the speed is better. I think you're right that some transfers got blocked or hanged.
You are also right about the NC modification times. I did a sync from NC -> Wasabi directly, and then did a file scan on the external storage. With this, all the files come through correctly, but Nextcloud shows the Modified date as the date of the scan, even though the dates show correctly in Wasabi.
Do you happen to know how to configure Nextcloud to accept these modification times? I see tons of bugs about it on their forums, but no fixes...
I also see a number of posts and use cases for "update mod time without downloading". I saw someone made a binary for this (but in 2017). Has this been considered as a feature?
I don't I'm afraid. The base nextcloud supports it, but it may depend on the backend you are using.
I found a note in the code about this
// SetModTime might be possible
// https://stackoverflow.com/questions/3579608/webdav-can-a-client-modify-the-mtime-of-a-file
// ...support for a PROPSET to lastmodified (mind the missing get) which does the utime() call might be an option.
// For example the ownCloud WebDAV server does it that way.
Ah, that link wasn't about WebDAV. The WebDAV backend can't set modification times at the moment, though it might be possible with nextcloud/owncloud (see above).
Thanks for the response - I think this is a bit out of my reach though. If it's not currently possible with RClone then I will focus on updating the metadata directly in the NC database.