[myremote]
type = webdav
url = https://nc-host.it/remote.php/webdav/
vendor = nextcloud
user = qnap-sync
pass = pass
A log from the command with the -vv flag
2022/01/04 17:10:16 DEBUG : Creating backend with remote "/share/Video/record_nvr/channel4/2020-08-04/16D/"
2022/01/04 17:10:16 DEBUG : Using config file from "/share/homes/administrator/.config/rclone/rclone.conf"
2022/01/04 17:10:16 DEBUG : Creating backend with remote "myremote:/Video/QNAP/channel4/2020-08-04/16D/"
2022/01/04 17:10:16 DEBUG : found headers:
2022/01/04 17:10:16 DEBUG : fs cache: renaming cache item "myremote:/Video/QNAP/channel4/2020-08-04/16D/" to be canonical "myremote:Video/QNAP/channel4/2020-08-04/16D"
2022-01-04 17:10:16 DEBUG : .nvr: Excluded
2022-01-04 17:10:23 DEBUG : .nvr: Excluded
2022-01-04 17:10:23 DEBUG : 2020-08-04 16-00-00~16-05-00.avi: Modification times differ by -443486h5m0.724998864s: 2020-08-04 16:05:00.724998864 +0200
CEST, 1970-01-01 01:00:00 +0100 CET
2022-01-04 17:10:23 DEBUG : 2020-08-04 16-05-00~16-10-00.avi: Modification times differ by -443486h10m0.705997719s: 2020-08-04 16:10:00.705997719 +020
0 CEST, 1970-01-01 01:00:00 +0100 CET
2022-01-04 17:10:23 DEBUG : webdav root 'Video/QNAP/channel4/2020-08-04/16D': Waiting for checks to finish
2022-01-04 17:10:23 DEBUG : 2020-08-04 16-10-00~16-15-00.avi: Modification times differ by -443486h15m0.694996575s: 2020-08-04 16:15:00.694996575 +020
0 CEST, 1970-01-01 01:00:00 +0100 CET
2022-01-04 17:10:23 DEBUG : 2020-08-04 16-15-00~16-20-00.avi: Modification times differ by -443486h20m0.669995431s: 2020-08-04 16:20:00.669995431 +020
0 CEST, 1970-01-01 01:00:00 +0100 CET
2022-01-04 17:10:23 DEBUG : 2020-08-04 16-20-00~16-25-00.avi: Modification times differ by -443486h25m0.650994286s: 2020-08-04 16:25:00.650994286 +020
0 CEST, 1970-01-01 01:00:00 +0100 CET
2022-01-04 17:10:23 DEBUG : 2020-08-04 16-25-00~16-30-00.avi: Modification times differ by -443486h30m0.645993142s: 2020-08-04 16:30:00.645993142 +020
0 CEST, 1970-01-01 01:00:00 +0100 CET
I opened this thread some time ago but wasn't able to make further checks until today. I made investigations and it appears in my case rclone isn't able to retain modification times.
I'm uploading to Nextcloud but to an external storage which is also encrypted. I don't know which of the two is impeding rclone from storing the correct time stamp, but as result files stored on this directory all have date 1970-01-01.
$ rclone lsl /share/Video/record_nvr/channel4/2020-08-04/16D/2020-08-04\ 16-00-00~16-05-00.avi
59051506 2020-08-04 16:05:00.724998864 2020-08-04 16-00-00~16-05-00.avi
$ rclone copy /share/Video/record_nvr/channel4/2020-08-04/16D/2020-08-04\ 16-00-00~16-05-00.avi myremote:Video/ --no-check-certificate -vvv
2022/01/04 17:44:29 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "copy" "/share/Video/record_nvr/channel4/2020-08-04/16D/2020-08-04 16-00-00~16-05-00.avi" "myremote:Video/" "--no-check-certificate" "-vvv"]
2022/01/04 17:44:29 DEBUG : Creating backend with remote "/share/Video/record_nvr/channel4/2020-08-04/16D/2020-08-04 16-00-00~16-05-00.avi"
2022/01/04 17:44:29 DEBUG : Using config file from "/share/homes/administrator/.config/rclone/rclone.conf"
2022/01/04 17:44:29 DEBUG : fs cache: adding new entry for parent of "/share/Video/record_nvr/channel4/2020-08-04/16D/2020-08-04 16-00-00~16-05-00.avi", "/share/Video/record_nvr/channel4/2020-08-04/16D"
2022/01/04 17:44:29 DEBUG : Creating backend with remote "myremove:Video/"
2022/01/04 17:44:29 DEBUG : found headers:
2022/01/04 17:44:29 DEBUG : fs cache: renaming cache item "myremote:Video/" to be canonical "myremote:Video"
2022/01/04 17:44:31 DEBUG : 2020-08-04 16-00-00~16-05-00.avi: Need to transfer - File not found at Destination
2022/01/04 17:45:16 DEBUG : 2020-08-04 16-00-00~16-05-00.avi: sha1 = db7b1f39cba5c2bd108c833b83c4598bf6574ac4 OK
2022/01/04 17:45:16 INFO : 2020-08-04 16-00-00~16-05-00.avi: Copied (new)
2022/01/04 17:45:16 INFO :
Transferred: 56.316 MiB / 56.316 MiB, 100%, 1.064 MiB/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 46.9s
2022/01/04 17:45:16 DEBUG : 4 go routines active
$ rclone lsl myremote:Video/2020-08-04\ 16-00-00~16-05-00.avi
59051506 1970-01-01 01:00:00.000000000 2020-08-04 16-00-00~16-05-00.avi
I forgot to add that I tried copying to a non-external storage location on the same Nextcloud server and modification time is preserved, so it must be something either with the external storage or with the encryption app
No experience with Nextcloud, but did a super quick look in the code: Rclone sends timestamp in extra header X-OC-Mtime. Seems Nextcloud should include header X-OC-MTime: accepted in response (source), so perhaps try to verify if this is the case with --dump headers (or maybe --dump responses)?
Thanks for the feedback. As said it works on Nextcloud local storage, it doesn't on external, so it's likely a limitation of Nextcloud itself. I posted here mostly for sharing the experience...
Yeah, I realized that as well after re-reading your previous posts, and the linked NC issue also quite clearly indicate external storage is a problematic area.
It's becoming a nightmare... I've fallen back to --size-only due to the findings above. Unfortunately this doesn't work because as I've enabled encryption, Nextcloud reports the size of the encrypted file, which will never be the same of the original file!!
The only option I have found is --checksum. I ran a test but unfortunately rclone seems willing to copy 90% of the source files, even the already uploaded ones. Debug shows for a sample file:
One question about -c I couldn't figure out from docs: is the checksum stored somewhere in Nextcloud DB or is it calculated every time? If it's the second option then it's not viable, as this would mean fetching the file every time! Some backends, like the one I'm using, charge for outgoing bandwidth, that would mean downloading the entire repo on every run!
This is not entirely true, as after further checking I found for some files / dirs lsl shows the same file size of local source file.
Unfortunately, I cannot find a pattern to identify why some dirs are correct and some others aren't.
-c, --checksum Skip based on checksum (if available) & size, not mod-time & size
I.e. it is called --checksum, not--checksum-only for a reason: It will compare file sizes first, and then if sizes are equal it will proceed to compare checksums (if available). It will never compare modtime. If backend does not support checksums then this effectively is corresponding to --size-only.
Edit: This can be seen from debug log:
NOTICE: Encrypted drive 'mycryptremote:': --checksum is in use but the source and destination have no hashes in common; falling back to --size-only
Regarding:
Rclone is using the checksum that the remote reports, if any. There is a separate command rclone check that can be used to check files (without copy), and it has flag --download to actually do download and calculate checksums.
When accessing a crypt remote in rclone, it reports the decrypted size of files.
This is handled internally by rclone: It fetches the size of the encrypted file from the storage system, and then calculates what the size would be if decrypting that file (without actually downloading or decrypting it). That way rclone can use size to compare a crypt remote with an unencrypted source.
Rclone uses size and modtime by default, and with crypt remote it is not able to use checksums. Therefore, if modtime cannot be trusted, you may want to specify --size-only and rely strictly on size. This has the risk of missing some changes, so you may want to run cryptcheck now and then to pick those up. You can force copy without checking anything first (not checking timestamp, not size, nor checksums!), with option --ignore-times (yes, it does not only ignore times, these options can be a bit confusing).
I wonder how this works, if encryption is done by Nextcloud and not by rclone itself. To my understanding, rclone doesn't know anything about the backend where the filed are stored. I'm not an expert of the Webdav protocol but I doubt it will report if the file is encrypted or not, or if the file is local or remote to the server itself.
Last week, during the various tests I did, I upgraded to the latest minor of Nextcloud, 22.2.3.
After that I ran rclone --checksum on a smaller dir. Ran over the weekend, now with lsl it does show the correct mtime on files! Running the same rclone command found only two files to transfer, which were indeed two errors found in the weekend run.
I did the same on another system (for the same Nextcloud backend), copying a small directory works with -c, without that it will always replace files.
Seems I found a solution . I'll check again tomorrow.
That was just an illusion. Mod times aren't retained, I'm sure I saw it but I wasn't able to replicate the good situation.
I tried ownCloud but, if possible, performs even worse with external FTP backend.
In the end I'm entirely relying on rclone, both for transfer and encryption. I'm mounting the remote storage with rclone and make it available as external (local) in Nextcloud.
