Nextcloud external storage won't retain modification times

What is the problem you are having with rclone?

Modification times are not retained on a Nextcloud remote

What is your rclone version (output from rclone version)

rclone v1.57.0
- os/version: qts 4.4.1 (20191206) (64 bit)
- os/kernel: 4.14.24-qnap (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Nextcloud 22

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy /Video/record_nvr/  myremote:/testmdtime/  -vvv -P --no-check-certificate  --transfers=1 --refresh-times

The rclone config contents with secrets removed.

[myremote]
type = webdav
url = https://nc-host.it/remote.php/webdav/
vendor = nextcloud
user = qnap-sync
pass = pass

A log from the command with the -vv flag

2022/01/04 17:10:16 DEBUG : Creating backend with remote "/share/Video/record_nvr/channel4/2020-08-04/16D/"
2022/01/04 17:10:16 DEBUG : Using config file from "/share/homes/administrator/.config/rclone/rclone.conf"
2022/01/04 17:10:16 DEBUG : Creating backend with remote "myremote:/Video/QNAP/channel4/2020-08-04/16D/"
2022/01/04 17:10:16 DEBUG : found headers:
2022/01/04 17:10:16 DEBUG : fs cache: renaming cache item "myremote:/Video/QNAP/channel4/2020-08-04/16D/" to be canonical "myremote:Video/QNAP/channel4/2020-08-04/16D"
2022-01-04 17:10:16 DEBUG : .nvr: Excluded
2022-01-04 17:10:23 DEBUG : .nvr: Excluded
2022-01-04 17:10:23 DEBUG : 2020-08-04 16-00-00~16-05-00.avi: Modification times differ by -443486h5m0.724998864s: 2020-08-04 16:05:00.724998864 +0200
 CEST, 1970-01-01 01:00:00 +0100 CET
2022-01-04 17:10:23 DEBUG : 2020-08-04 16-05-00~16-10-00.avi: Modification times differ by -443486h10m0.705997719s: 2020-08-04 16:10:00.705997719 +020
0 CEST, 1970-01-01 01:00:00 +0100 CET
2022-01-04 17:10:23 DEBUG : webdav root 'Video/QNAP/channel4/2020-08-04/16D': Waiting for checks to finish
2022-01-04 17:10:23 DEBUG : 2020-08-04 16-10-00~16-15-00.avi: Modification times differ by -443486h15m0.694996575s: 2020-08-04 16:15:00.694996575 +020
0 CEST, 1970-01-01 01:00:00 +0100 CET
2022-01-04 17:10:23 DEBUG : 2020-08-04 16-15-00~16-20-00.avi: Modification times differ by -443486h20m0.669995431s: 2020-08-04 16:20:00.669995431 +020
0 CEST, 1970-01-01 01:00:00 +0100 CET
2022-01-04 17:10:23 DEBUG : 2020-08-04 16-20-00~16-25-00.avi: Modification times differ by -443486h25m0.650994286s: 2020-08-04 16:25:00.650994286 +020
0 CEST, 1970-01-01 01:00:00 +0100 CET
2022-01-04 17:10:23 DEBUG : 2020-08-04 16-25-00~16-30-00.avi: Modification times differ by -443486h30m0.645993142s: 2020-08-04 16:30:00.645993142 +020
0 CEST, 1970-01-01 01:00:00 +0100 CET

I opened this thread some time ago but wasn't able to make further checks until today. I made investigations and it appears in my case rclone isn't able to retain modification times.

I'm uploading to Nextcloud but to an external storage which is also encrypted. I don't know which of the two is impeding rclone from storing the correct time stamp, but as result files stored on this directory all have date 1970-01-01.

[administrator@NVRPSYCHO administrator]$ rclone lsl myremote:/Video/QNAP/channel4/2020-08-04/16D/ --no-check-certificate
 59051506 1970-01-01 01:00:00.000000000 2020-08-04 16-00-00~16-05-00.avi
 59032414 1970-01-01 01:00:00.000000000 2020-08-04 16-05-00~16-10-00.avi
 59174902 1970-01-01 01:00:00.000000000 2020-08-04 16-10-00~16-15-00.avi
 59718910 1970-01-01 01:00:00.000000000 2020-08-04 16-15-00~16-20-00.avi
 60292984 1970-01-01 01:00:00.000000000 2020-08-04 16-20-00~16-25-00.avi
 59405072 1970-01-01 01:00:00.000000000 2020-08-04 16-25-00~16-30-00.avi
 58891756 1970-01-01 01:00:00.000000000 2020-08-04 16-30-00~16-35-00.avi
 32065704 1970-01-01 01:00:00.000000000 2020-08-04 16-35-00~16-37-37.avi

In Nextcloud web those files appears as modified at the upload time, and with an invalid date.

I suppose I have to raise the issue to Nextcloud server. In the meantime, my only option is to use --size-only, isn't it? Thanks

Pick one file to test with.

  1. rclone lsl the file on the source
  2. rclone copy the file to the remote with debug
  3. rclone lsl the file on the remote
felix@gemini:~$ rclone lsl /etc/hosts
      130 2021-12-09 08:44:53.993597456 hosts
felix@gemini:~$ rclone copy /etc/hosts DB:
felix@gemini:~$ rclone lsl DB:hosts -vvv
2022/01/04 11:40:55 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/01/04 11:40:55 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "lsl" "DB:hosts" "-vvv"]
2022/01/04 11:40:55 DEBUG : Creating backend with remote "DB:hosts"
2022/01/04 11:40:55 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/01/04 11:40:55 DEBUG : fs cache: adding new entry for parent of "DB:hosts", "DB:"
      130 2021-12-09 08:44:54.000000000 hosts
2022/01/04 11:40:56 DEBUG : 6 go routines active
2022/01/04 11:40:56 INFO  : Dropbox root '': Commiting uploads - please wait...
$ rclone lsl /share/Video/record_nvr/channel4/2020-08-04/16D/2020-08-04\ 16-00-00~16-05-00.avi
 59051506 2020-08-04 16:05:00.724998864 2020-08-04 16-00-00~16-05-00.avi

$ rclone copy /share/Video/record_nvr/channel4/2020-08-04/16D/2020-08-04\ 16-00-00~16-05-00.avi myremote:Video/ --no-check-certificate -vvv
2022/01/04 17:44:29 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "copy" "/share/Video/record_nvr/channel4/2020-08-04/16D/2020-08-04 16-00-00~16-05-00.avi" "myremote:Video/" "--no-check-certificate" "-vvv"]
2022/01/04 17:44:29 DEBUG : Creating backend with remote "/share/Video/record_nvr/channel4/2020-08-04/16D/2020-08-04 16-00-00~16-05-00.avi"
2022/01/04 17:44:29 DEBUG : Using config file from "/share/homes/administrator/.config/rclone/rclone.conf"
2022/01/04 17:44:29 DEBUG : fs cache: adding new entry for parent of "/share/Video/record_nvr/channel4/2020-08-04/16D/2020-08-04 16-00-00~16-05-00.avi", "/share/Video/record_nvr/channel4/2020-08-04/16D"
2022/01/04 17:44:29 DEBUG : Creating backend with remote "myremove:Video/"
2022/01/04 17:44:29 DEBUG : found headers:
2022/01/04 17:44:29 DEBUG : fs cache: renaming cache item "myremote:Video/" to be canonical "myremote:Video"
2022/01/04 17:44:31 DEBUG : 2020-08-04 16-00-00~16-05-00.avi: Need to transfer - File not found at Destination
2022/01/04 17:45:16 DEBUG : 2020-08-04 16-00-00~16-05-00.avi: sha1 = db7b1f39cba5c2bd108c833b83c4598bf6574ac4 OK
2022/01/04 17:45:16 INFO  : 2020-08-04 16-00-00~16-05-00.avi: Copied (new)
2022/01/04 17:45:16 INFO  :
Transferred:       56.316 MiB / 56.316 MiB, 100%, 1.064 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:        46.9s

2022/01/04 17:45:16 DEBUG : 4 go routines active


$ rclone lsl myremote:Video/2020-08-04\ 16-00-00~16-05-00.avi
 59051506 1970-01-01 01:00:00.000000000 2020-08-04 16-00-00~16-05-00.avi

I forgot to add that I tried copying to a non-external storage location on the same Nextcloud server and modification time is preserved, so it must be something either with the external storage or with the encryption app

Found this NC issue, doesn't have suggestion but seems relevant

I think maybe @albertony can opine on it as from my understanding, it should work that shows the modtime isn't being set.

What if you rclone touch the same file? What's the output?

I know on dropbox you can't without re-uploading it.

felix@gemini:~$ rclone touch DB:hosts -vvv
2022/01/04 12:01:12 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/01/04 12:01:12 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "touch" "DB:hosts" "-vvv"]
2022/01/04 12:01:12 DEBUG : Creating backend with remote "DB:hosts"
2022/01/04 12:01:12 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/01/04 12:01:12 DEBUG : fs cache: adding new entry for parent of "DB:hosts", "DB:"
2022/01/04 12:01:12 DEBUG : Touch time 2022-01-04 12:01:12.569200836 -0500 EST m=+0.294048173
2022/01/04 12:01:12 DEBUG : Dropbox root '': Touching "hosts"
2022/01/04 12:01:12 ERROR : Attempt 1/3 failed with 1 errors and: failed to touch: can't set modified time without deleting existing object
2022/01/04 12:01:12 DEBUG : Touch time 2022-01-04 12:01:12.757720616 -0500 EST m=+0.482567960
2022/01/04 12:01:12 DEBUG : Dropbox root '': Touching "hosts"
2022/01/04 12:01:12 ERROR : Attempt 2/3 failed with 1 errors and: failed to touch: can't set modified time without deleting existing object
2022/01/04 12:01:12 DEBUG : Touch time 2022-01-04 12:01:12.928587903 -0500 EST m=+0.653435243
2022/01/04 12:01:13 DEBUG : Dropbox root '': Touching "hosts"
2022/01/04 12:01:13 ERROR : Attempt 3/3 failed with 1 errors and: failed to touch: can't set modified time without deleting existing object
2022/01/04 12:01:13 DEBUG : 6 go routines active
2022/01/04 12:01:13 Failed to touch: failed to touch: can't set modified time without deleting existing object
2022/01/04 12:01:13 INFO  : Dropbox root '': Commiting uploads - please wait...

I would also expect it to work.

No experience with Nextcloud, but did a super quick look in the code: Rclone sends timestamp in extra header X-OC-Mtime. Seems Nextcloud should include header X-OC-MTime: accepted in response (source), so perhaps try to verify if this is the case with --dump headers (or maybe --dump responses)?

Thanks for the feedback. As said it works on Nextcloud local storage, it doesn't on external, so it's likely a limitation of Nextcloud itself. I posted here mostly for sharing the experience...

Yeah, I realized that as well after re-reading your previous posts, and the linked NC issue also quite clearly indicate external storage is a problematic area.

It's becoming a nightmare... I've fallen back to --size-only due to the findings above. Unfortunately this doesn't work because as I've enabled encryption, Nextcloud reports the size of the encrypted file, which will never be the same of the original file!!

The only option I have found is --checksum. I ran a test but unfortunately rclone seems willing to copy 90% of the source files, even the already uploaded ones. Debug shows for a sample file:

2022/01/05 16:37:40 DEBUG : Emanuela/Casa Cometa Michele/Va.A..mpg: Sizes differ (src 184963072 vs dst 249549976)
...
2022/01/05 16:44:06 DEBUG : Emanuela/Casa Cometa Michele/Va.A..mpg: sha1 = 062c45e2f57ecfe550052ee668afab49008b7dc7 OK
2022/01/05 16:44:06 INFO  : Emanuela/Casa Cometa Michele/Va.A..mpg: Copied (replaced existing)

Ok after further testing I'm officially getting lost! I've ran rclone once again, just for the file above file, I get:

2022/01/05 16:46:07 DEBUG : Va.A..mpg: Size of src and dst objects identical
2022/01/05 16:46:07 DEBUG : Va.A..mpg: Unchanged skipping

The options I used:
-vvv --exclude "System Volume Information" --exclude "desktop.ini" -P --no-check-certificate --checksum --log-file=/dati/rclone-logs/rclone-$(date +%Y.%m.%d_%H:%M:%S).log --bwlimit 1M --transfers=2

  1. Why is the log reporting about the size, if I asked for checksum?
  2. Why now the size of the file is identical, source and in Nextcloud (of course not on the backend filesystem)?
$ rclone lsl myremote:/Video/Disco01/Emanuela/Casa\ Cometa\ Michele/
184963072 1970-01-01 01:00:00.000000000 Va.A..mpg
$ ls -la /mnt/video/Emanuela/Casa\ Cometa\ Michele/Va.A..mpg
-rwxrwxrwx 1 root root 184963072 feb 24  2015 '/mnt/videopitticca/Emanuela/Casa Cometa Michele/Va.A..mpg'

One question about -c I couldn't figure out from docs: is the checksum stored somewhere in Nextcloud DB or is it calculated every time? If it's the second option then it's not viable, as this would mean fetching the file every time! Some backends, like the one I'm using, charge for outgoing bandwidth, that would mean downloading the entire repo on every run!

Thanks for the help

This is not entirely true, as after further checking I found for some files / dirs lsl shows the same file size of local source file.
Unfortunately, I cannot find a pattern to identify why some dirs are correct and some others aren't.

Answer from docs:

-c, --checksum                             Skip based on checksum (if available) & size, not mod-time & size

I.e. it is called --checksum, not --checksum-only for a reason: It will compare file sizes first, and then if sizes are equal it will proceed to compare checksums (if available). It will never compare modtime. If backend does not support checksums then this effectively is corresponding to --size-only.
Edit: This can be seen from debug log:

NOTICE: Encrypted drive 'mycryptremote:': --checksum is in use but the source and destination have no hashes in common; falling back to --size-only

Regarding:

Rclone is using the checksum that the remote reports, if any. There is a separate command rclone check that can be used to check files (without copy), and it has flag --download to actually do download and calculate checksums.

How do I know if NC is reporting it? According to the overview page NC/OC does support it.

I ran -c on a small dir and a second run took just 2s, which should mean it's behaving.
I'm testing on a larger directory, will check tomorrow. :crossed_fingers:

hi,
to see if a remote is storing the hash

rclone lsf remotewithhashsupport: --format=ph
file.txt;d41d8cd98f00b204e9800998ecf8427e

rclone lsf remotewithouthashsupport:  --format=ph
ERROR : file.txt: Failed to read hash: hash type not supported
file.txt;

When accessing a crypt remote in rclone, it reports the decrypted size of files.

This is handled internally by rclone: It fetches the size of the encrypted file from the storage system, and then calculates what the size would be if decrypting that file (without actually downloading or decrypting it). That way rclone can use size to compare a crypt remote with an unencrypted source.

Rclone uses size and modtime by default, and with crypt remote it is not able to use checksums. Therefore, if modtime cannot be trusted, you may want to specify --size-only and rely strictly on size. This has the risk of missing some changes, so you may want to run cryptcheck now and then to pick those up. You can force copy without checking anything first (not checking timestamp, not size, nor checksums!), with option --ignore-times (yes, it does not only ignore times, these options can be a bit confusing).

I wonder how this works, if encryption is done by Nextcloud and not by rclone itself. To my understanding, rclone doesn't know anything about the backend where the filed are stored. I'm not an expert of the Webdav protocol but I doubt it will report if the file is encrypted or not, or if the file is local or remote to the server itself.

Oh, I know nothing about that part. I was talking only about rclone's encryption. That was really off topic then, sorry about that.

I agree.

Last week, during the various tests I did, I upgraded to the latest minor of Nextcloud, 22.2.3.

After that I ran rclone --checksum on a smaller dir. Ran over the weekend, now with lsl it does show the correct mtime on files! Running the same rclone command found only two files to transfer, which were indeed two errors found in the weekend run.

I did the same on another system (for the same Nextcloud backend), copying a small directory works with -c, without that it will always replace files.

Seems I found a solution :crossed_fingers:. I'll check again tomorrow.

That was just an illusion. Mod times aren't retained, I'm sure I saw it but I wasn't able to replicate the good situation.

I tried ownCloud but, if possible, performs even worse with external FTP backend.

In the end I'm entirely relying on rclone, both for transfer and encryption. I'm mounting the remote storage with rclone and make it available as external (local) in Nextcloud.

Thanks everyone for the continuous help!