File modification date is not preserved when runnng rclone sync from an Android script

[ioorclone]

What is the problem you are having with rclone?

Files uploaded to remote with rclone sync command do not have their modified time stamps preserved. They are getting the current time. I have double checked and I am not using the --no-modtime parameter anywhere.

I am trying to create an automatic phone backup by modifying the sync script from the project (https://github.com/Magisk-Modules-Repo/com.piyushgarg.rclone/blob/master/binary/syncd.sh). The script was originally using rclone copy and I changed it to rclone sync. Everything is working extremely well except that the mod date is not preserved when the file is uploaded. Unfortunately this is critical for an automatic sync project and I need to keep this modified date.

Below I have bolded parts of the logs below which look relevant.

What is your rclone version (output from rclone version)

rclone v1.53.0-DEV
os/arch: android/arm
go version: go1.13.4

Which OS you are using and how many bits (eg Windows 7, 64 bit)

LineageOS 17 (Android 10)

Which cloud storage system are you using? (eg Google Drive)

pCloud

The command you were trying to run (eg rclone copy /tmp remote:tmp)

nice -n 19 ionice -c 2 -n 7 ${HOME}/rclone sync "/storage/emulated/${PROFILE}/${SYNCDIR}" "$CLOUDROOTMOUNTPOINT/${remote}/${SYNCDIR}" --retries-sleep=10m --retries 6 --transfers 1 --multi-thread-streams 1 --log-file=/storage/emulated/0/.rclone/file.log

Note: The $variables are part of the script, they are all working and the files upload and deleted from the remote successfully, this is essentially jsut a simple rclone sync command. The only problem is the mod time is not preserved.

My mount options

rclone mount remote: /mnt/cloud/remote --vfs-cache-mode off --cache-dir /data/rclone/cache --cache-chunk-path /data/rclone/cache-backend --cache-db-path /data/rclone/cache-backend --cache-temp-upload-path /data/rclone/cache --vfs-cache-max-size 1G --cache-chunk-size 1M --cache-chunk-total-size 1G --cache-workers 1 --cache-info-age 1h0m0s --dir-cache-time 30m0s --attr-timeout 30s --cache-chunk-no-memory --use-mmap --buffer-sizw 0 --max-read-ahead 128k --no-checksum --uid 0 --gid 1015 --allow-others --dir-perm 0775 --file-perms 0644 --umask 002 -fast-list --daemon

A log from the command with the -vv flag

Script run #1:

2020/11/13 22:43:05 DEBUG : 3 go routines active
2020/11/13 22:43:10 DEBUG : rclone: Version "v1.53.0-DEV" starting with parameters ["/data/adb/modules/com.piyushgarg.rclone/rclone" "sync" "/storage/emulated/0/tests" "/mnt/cloud/remote/tests" "--retries-sleep=10m" "--retries" "6" "--transfers" "1" "--multi-thread-streams" "1" "--log-file" "/storage/emulated/0/.rclone/file.log" "-vv"]
2020/11/13 22:43:10 DEBUG : Creating backend with remote "/storage/emulated/0/tests"
2020/11/13 22:43:10 DEBUG : Using config file from "/data/adb/modules/com.piyushgarg.rclone/.config/rclone/rclone.conf"
2020/11/13 22:43:10 DEBUG : Creating backend with remote "/mnt/cloud/remote/tests"
2020/11/13 22:43:10 DEBUG : Local file system at /mnt/cloud/remote/tests: Waiting for checks to finish
2020/11/13 22:43:10 DEBUG : Local file system at /mnt/cloud/remote/tests: Waiting for transfers to finish
2020/11/13 22:43:10 DEBUG : **preAllocate: got error on fallocate, trying combination 1/2: operation not supported**
2020/11/13 22:43:10 DEBUG : **preAllocate: got error on fallocate, trying combination 2/2: operation not supported**
2020/11/13 22:43:11 DEBUG : testTime.1: MD5 = 62d4610f0d8b1a0fa57c1d3d820b39ef OK
2020/11/13 22:43:11 INFO  : testTime.1: Copied (new)
2020/11/13 22:43:11 DEBUG : Waiting for deletions to finish
2020/11/13 22:43:11 INFO  : 
Transferred:   	       540 / 540 Bytes, 100%, 922 Bytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         0.6s

Result: test file is successfully transferred to remote. The modification date however is not preserved.

Note: there are two debug errors that I get every time I upload a file, could these be causing the date problem?

2020/11/13 22:43:11 DEBUG : 4 go routines active
2020/11/13 22:43:16 DEBUG : rclone: Version "v1.53.0-DEV" starting with parameters ["/data/adb/modules/com.piyushgarg.rclone/rclone" "sync" "/storage/emulated/0/tests" "/mnt/cloud/remote/tests" "--retries-sleep=10m" "--retries" "6" "--transfers" "1" "--multi-thread-streams" "1" "--log-file" "/storage/emulated/0/.rclone/file.log" "-vv"]
2020/11/13 22:43:16 DEBUG : Creating backend with remote "/storage/emulated/0/tests"
2020/11/13 22:43:16 DEBUG : Using config file from "/data/adb/modules/com.piyushgarg.rclone/.config/rclone/rclone.conf"
2020/11/13 22:43:16 DEBUG : Creating backend with remote "/mnt/cloud/remote/tests"
2020/11/13 22:43:16 DEBUG : Local file system at /mnt/cloud/remote/tests: Waiting for checks to finish
2020/11/13 22:43:16 DEBUG : testTime.1: **Modification times differ by 236h45m10.627350357s:** 2020-11-04 01:57:59.372649643 +0000 UTC, 2020-11-13 22:43:10 +0000 UTC
2020/11/13 22:43:17 DEBUG : testTime.1: MD5 = 62d4610f0d8b1a0fa57c1d3d820b39ef OK
2020/11/13 22:43:17 INFO  : testTime.1: **Updated modification time in destination**
2020/11/13 22:43:17 DEBUG : testTime.1: Unchanged skipping
2020/11/13 22:43:17 DEBUG : Local file system at /mnt/cloud/remote/tests: Waiting for transfers to finish
2020/11/13 22:43:17 DEBUG : Waiting for deletions to finish
2020/11/13 22:43:17 INFO  : There was nothing to transfer
2020/11/13 22:43:17 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         1.4s

Result: It looks like rclone can see the date problem, it reports that it has updated the file:

INFO  : testTime.1: Updated modification time in destination

however it does not successfully change the file date.

Every subsequent script run will report the same date log lines everytime but it never fixes the date.

I have also tried using the --refresh-times parameter but it also did not work.

rclone lsl remote:/path/tests/ shows today's date and time:
540 2020-11-13 17:43:10.000000000 testTime.1

not the modified time of the file:
/storage/emulated/0/tests/
Nov 3, 2020 83:57:33 p.m. testTime.1

Accorging to this page, my remote (pcloud) does support modification times: https://rclone.org/pcloud/#modified-time-and-hashes

I have been working on this automatic backup project all week and this is the last critical piece of the puzzle. I can't find anything to get past this mod date problem. I know rclone is capable of preserving a file date, I am not sure why it is not working here.

[ncw (Nick Craig-Wood)]

@ioorclone

Note: there are two debug errors that I get every time I upload a file, could these be causing the date problem?

No, these are normal I think - the android file system doesn't support pre-allocate.

pCloud does support modification dates however it doesn't support setting modification dates - the file needs to be re-uploaded - so I'm a bit suprised by this

@ioorclone

2020/11/13 22:43:17 INFO  : testTime.1: **Updated modification time in destination**

Looking at your command line

rclone sync /storage/emulated/0/tests /mnt/cloud/remote/tests --retries-sleep=10m --retries 6 --transfers 1 --multi-thread-streams 1 --log-file /storage/emulated/0/.rclone/file.log -vv

You don't appear to be syncing to pcloud you are syncing to a mount...

Because pcloud can't set modification times execpt by uploading files the rclone mount doesn't actually set the modification times, it just says it does.

If you sync direct to pcloud (not via the rclone mount) then it will work properly I think.

[ioorclone]

@ncw

You don't appear to be syncing to pcloud you are syncing to a mount...

Because pcloud can't set modification times except by uploading files the rclone mount doesn't actually set the modification times, it just says it does.

If you sync direct to pcloud (not via the rclone mount) then it will work properly I think.

Thank you for this confirmation! Is this a limitation of pcloud only or can other providers set modification times in this way? Aside from this limitation, the default behavior of the script I am working with is syncing to a mount and I am trying to understand why it would be setup this way.

My end goal is to create a fully-automated, real-time OSS dropbox replacement for Android by mirroring /storage/emulated/0 (with exceptions) to a remote. That script appears to utilize inotify as a trigger for running rclone sync command so it might be ideal for this purpose.

A few questions about my command (below). I have poured over the documentation but a few aspects of caching are unclear- mainly the differences between vfs cahce and cache.

1. Is the main benefit of caching or vfs caching to allow for seeking operations for streaming media files? In my application, having a local database of cached file attributes may reduce transactions however it adds a whole layer of overhead by mirroring the data on disk which may be a disadvantage for a storage constrained device. Is there any other advantage to syncing with a cache in this scenario? From a best practice point of view, would you suggest I stick with syncing to the mount or modify to sync directly to the remote?

2. Conceptually, what is the main difference between vfs caching and cache? I thought the V in VFS implied that it was using the device memory as a virtual file system (like a ram disk) to hold files and attributes and speed up operations. When I set --vfs-cache-mode writes however I can see it actually writes the files to disk so I think my understanding is wrong. How is --vfs-cache-mode writes or --vfs-cache-mode full then different from caching in this regard?

3. Am I only using a cache if I wrap a cache back end around a remote in the config file?

For example:

rclone mount remote: /mnt/cloud/remote --vfs-cache-mode off --cache-dir /data/rclone/cache --cache-chunk-path /data/rclone/cache-backend --cache-db-path /data/rclone/cache-backend --cache-temp-upload-path /data/rclone/cache --vfs-cache-max-size 1G --cache-chunk-size 1M --cache-chunk-total-size 1G --cache-workers 1 --cache-info-age 1h0m0s --dir-cache-time 30m0s --attr-timeout 30s --cache-chunk-no-memory --use-mmap --buffer-size 0 --max-read-ahead 128k --no-checksum --uid 0 --gid 1015 --dir-perm 0775 --file-perms 0644 --umask 002 --daemon

Does this command use a vfs cache, a cache or both? What if I change --vfs-cache-mode to writes?

4. Flags that contain the "vfs-" prefix seems clear however are there some flags that do not contain this "vfs" prefix that are actually vfs flags or can be both vfs and cache flags?

For example, are these vfs or cache flags or both?
--cache-dir
--cache-chunk-path
--cache-db-path
--cache-temp-upload-path
--cache-chunk-size
--cache-chunk-total-size
--dir-cache-time
--attr-timeout

5. If I use a cache flag with a rclone sync operation but I am not wrapping a remote back end with a cache back end, will these flags be ignored or what will the effect be?

6. Is it possible to get the benefit of the cache directory database file (--cache-db-path) without caching the files locally? Can cache-db-path benefit a vfs cache or does a vfs cache have any similar functionality?

7. It would appear that files are not being removed form my vfs cache even across reboots. (--vfs-cache-max-age duration) defaults to 1h. Does this mean 1h after last access? What determines if a file will stay in the cache even after this time has expired? If I have a sync script running every minute which will look at the files and see if they need to be uploaded, will they stay in the cache in the case?

8. When aliasing a remote with conflicting flags, which scenario would take precedence? I.e. If I set writes = true is defined in rclone.conf and I mount an alias of this remote with --vfs-cache-mode off what would happen?

Sorry if some of my questions are nonsensical. I have been trying to understand caching and vfs caching for some time now with limited success. Any advice you could provide would really be appreciated. Thank you very much for rclone, it is an AWESOME project.

[ncw (Nick Craig-Wood)]

@ioorclone

Thank you for this confirmation! Is this a limitation of pcloud only or can other providers set modification times in this way?

Other providers can set the modtime like this, but others can't...

@ioorclone

Aside from this limitation, the default behavior of the script I am working with is syncing to a mount and I am trying to understand why it would be setup this way.

One possibility is that syncing through the mount allows the VFS cache to get updated instantly rather than waiting for the --dir-cache-time to expire.

If you add --size-only to the rclone sync command then it will ignore the modification times.

@ioorclone

Is the main benefit of caching or vfs caching to allow for seeking operations for streaming media files?

You can read media files with seeking without the VFS cache. However if you want to write and seek then you need the VFS cache. Note that --vfs-cache-mode writes only caches stuff you right, stuff you read will be read direct.

@ioorclone

From a best practice point of view, would you suggest I stick with syncing to the mount or modify to sync directly to the remote?

In terms of data integrity, the best practice is always to sync direct to the remote. Syncing via the mount should work OK but kicks up problems like the modtime problem you are seeing which is to do with features getting lost in the conversion between the layers.

@ioorclone

Conceptually, what is the main difference between vfs caching and cache? I thought the V in VFS implied that it was using the device memory as a virtual file system (like a ram disk) to hold files and attributes and speed up operations. When I set --vfs-cache-mode writes however I can see it actually writes the files to disk so I think my understanding is wrong. How is --vfs-cache-mode writes or --vfs-cache-mode full then different from caching in this regard?

The VFS does stand for virtual file system, but what is virtual about it is that rclone builds it from a backend. Attributes are held in memory, but files aren't except if you use --vfs-cache-mode. In general writes holds files you have written via the mount, but full holds files you read or write (these are loaded on demand).

@ioorclone

Am I only using a cache if I wrap a cache back end around a remote in the config file?

The cache backend is a different thing to the VFS cache. It isn't well maintained so I don't recommend using it.

You get the VFS cache if you use rclone mount, but not otherwise. However I have plans to add the VFS cache to any backend at some point.

@ioorclone

For example, are these vfs or cache flags or both?
--cache-dir
--cache-chunk-path
--cache-db-path
--cache-temp-upload-path
--cache-chunk-size
--cache-chunk-total-size
--dir-cache-time
--attr-timeout

Flags starting --cache belong to the cache backend, except for --cache-dir which sets the cache directory for all of rclone.

--dir-cache-time is a VFS flag and --attr-timeout is an rclone mount flag.

@ioorclone

If I use a cache flag with a rclone sync operation but I am not wrapping a remote back end with a cache back end, will these flags be ignored or what will the effect be?

They will be ignored

@ioorclone

Is it possible to get the benefit of the cache directory database file (--cache-db-path) without caching the files locally? Can cache-db-path benefit a vfs cache or does a vfs cache have any similar functionality?

I think so, but I don't recommend using the cache backend for new setups. The VFS can't do this yet, but it will be able to soon!

@ioorclone

It would appear that files are not being removed form my vfs cache even across reboots. (--vfs-cache-max-age duration) defaults to 1h. Does this mean 1h after last access?

Yes

@ioorclone

What determines if a file will stay in the cache even after this time has expired? If I have a sync script running every minute which will look at the files and see if they need to be uploaded, will they stay in the cache in the case?

If an rclone mount is running it will check the VFS cache every minute by default and expire old files.

@ioorclone

When aliasing a remote with conflicting flags, which scenario would take precedence? I.e. If I set writes = true is defined in rclone.conf and I mount an alias of this remote with --vfs-cache-mode off what would happen?

You can't set --vfs-cache-mode in the config file yet.

Hope that helps!