Does sync invalidate mount cache on the same machine?

What is the problem you are having with rclone?

I am mounting a Google Drive folder using rclone mount
I am also syncing to the same folder using rclone sync
I have very long attribute and directory cache times for the mount, but whenever I list the directories in the mount they seem to update as sync is working - I was expecting to not see the changes from sync until the cache expires (so rclone is actually doing a better job than I was expecting)

Is this expected behaviour? Is the fact that I'm using the same rclone configuration entry from the same machine causing the sync to invalidate the rclone mount cache (I haven't been able to find where that cache is, I guess it's in memory)

What is your rclone version (output from rclone version)

rclone v1.49.5

  • os/arch: linux/amd64
  • go version: go1.12.10

Which OS you are using and how many bits (eg Windows 7, 64 bit)

CoreOS

uname -a:
Linux core1 4.19.68-coreos #1 SMP Wed Sep 4 02:59:18 -00 2019 x86_64 Intel(R) Core(TM) i5-6600K CPU @ 3.50GHz GenuineIntel GNU/Linux

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

/opt/bin/rclone sync -l --fast-list --bwlimit=4M /media/ media:

/opt/bin/rclone mount media: /mnt/google-drive/Media
--config=/root/.config/rclone/rclone.conf
--drive-acknowledge-abuse
--drive-chunk-size=256M
--buffer-size=64M
--use-mmap
--attr-timeout=72h
--dir-cache-time=72h
--gid=1000
--uid=1000
--modify-window=1s
--log-level=INFO
--allow-other

The VFScache can pick up changes that are made through it - but as you say, if the changes are made outside the mount/VFS then it can't just do this automatically as it happens.

Normally you are right that such changes would not be picked up until cache expires.
However, Google drive (and a few other backends) have a really really useful feature called changenotify (or "polling" in rclone terms).

This works the way that every so often (1min by default) the VFS-layer (effectively the mount) will ask the Google server "hey, did any of your files change in the last minute?" - and then the google server replies that "yea, I have two new files since last time actually - here are the list-info for them".
Then the VFS goes "thanks! I'll update my cache using this info so I am up-to-date".
And then they lived happily ever after... :wink:

So long story short - yes, this is not only intentional but also hugely helpful. With polling you can actual precache (pre-list) the entire drive when you start it, and polling will keep that information up-to-date no matter how long it runs (I run a year expiration timer on my setup).

If you want you can change the polling timer
--poll-interval 1m
as you please, or disable it with a value of 0 (although why would you want that? ...)

Do note that each polling request takes 1 API call. You have 1000 to use every 100sec.
Thus a polling interval of 10s would use 1% of the limit just for polling. This is about as low as I would go. For most uses the default 1m is perfectly fine and uses a trivial amount of the API limit.

If you are interested in precaching as I mentioned in passing - let me know. I can help you, but I didn't want to go too far off-topic right off the bat :slight_smile:

Awesome, this is great, thank you for the thorough explanation

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.