Running vfs/refresh with a dir parameter will throw a "file does not exist" error if the directory isn't yet in the mount's VFS directory cache even though it exists in the remote. Instead of erroring I would expect rclone to walk the remote looking for the directory. If it finds it it should load it into the VFS and only throw that error if it doesn't exist on the remote.
If I refresh the new directory's parent directory then the new directory will appear in the mount, but if the parent directory is large then this is is a very intensive approach that I believe should be unnecessary.
I know --dir-cache-time and --poll-interval could help to work around this issue but I would still expect vfs/refresh to discover the new directory.
What is your rclone version (output from rclone version)
rclone v1.57.0-beta.5629.e45c23ab7
os/version: Microsoft Windows 10 Pro 2009 (64 bit)
os/kernel: 10.0.19042.1165 (x86_64)
os/type: windows
os/arch: amd64
go/version: go1.16.7
go/linking: dynamic
go/tags: cmount
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Windows 10 (64 bit)
Which cloud storage system are you using? (eg Google Drive)
Google Drive
The command you were trying to run (eg rclone copy /tmp remote:tmp)
If you create something outside the mount and the directory cache hasn't expired and polling hasn't occurred, rclone wouldn't know it existed and it produces the error you described.
There's no magic here to 'know' it's there as that's how polling and dir cache works as you stated.
Hi Animosity022, while I have you here I just want to thank you for all the help you've provided in the forums. I've learnt a lot from you. Thank you!
You're right that I wouldn't expect a new directory/file created outside the mount to automatically appear without use of --dir-cache-time or --poll-interval, but I was under the impression that using the rc command vfs/refresh would allow me to update the directory cache manually. Thats definitely how it appears to work.
vfs/refresh without a dir= rebuilds the directory cache from the root directory, and this is working for me as expected. When vfs/refresh is passed a dir=, rclone will update at that directory only (as well as all its children if recursive=true is used). If the directory that was passed already exists in the directory cache then this command also works fine. The issue I'm seeing is that for vfs/refresh to update a specific directory then the directory must already exist in the directory cache. Instead of this behaviour, as rclone is attempting to update the directory provided, I would have thought that rclone should walk that path on the remote (Google Drive in my case) and if that path is valid on the remote it would update the directory cache by caching any new folders it needs in order to get there.
Maybe it wasn't implemented this way due to some technical limitation or perhaps I have some kind of misunderstanding of how vfs/refresh is supposed to work.
Also, for context, the reason I'm not using the high --dir-cache-time and low --poll-interval approach is because in my actual implementation I have a drive > crypt > union > mount setup in order to combine Google Drive with local disks. It appears that --poll-interval isn't supported through the union as the directory cache isn't being updated automatically and running rclone rc vfs/poll-interval returns the error "poll-interval is not supported by this remote".
Are you saying that the behavior I'm seeing is implemented that way by design? Or is it a limitation for which there is no current plan to address? To me, this seems well within the scope of what vfs/refresh should be able to do.
I don't think walking would actually be necessary. I'm not familiar with every remote but surely simply they would all support being queried for if the dir= exists.
Having to query a parent directory that already exists could be a very intensive approach, especially when using recursive=true to get the desired directories children. For example, in my setup:
Radarr (running on my NAS) puts a new movie in my union:/movies/ directory.
Radarr tells rclone (running on my media server) to do a vfs/refresh dir=/movies/new_movie/ recursive=true
Rclone on the media server throws a "file does not exist" error
Instead, what you're saying, is rclone would need to do vfs/refresh dir=/movie/ recursive=true and thus refresh the entire movies directory recursively. This will update thousands of directories and take many minutes instead of the couple of seconds it would take if rclone could simply attempt to discover the directory it was provided if it didn't yet have it in the directory cache.
Not ensuring consistency would lead to funkiness in the cache as it doesn't know what changed. It's no different once your dir-cache time runs out as it has repoll everything anyway.
I find Windows/NAS to be very lacking in that setup and you hit problems like these where Linux is a much better fit for the use case.