"File Delete" operations in Windows Explorer not cached by vfs cache?

Help and Support
1

What is the problem you are having with rclone?

I am working with a mounted remote of type crypt that is located on a subfolder on a Google Drive (see command line below).

So far the vfs cache seems to work great for file read and write operations, but whenever I delete files interactively in Windows Explorer, that operation does not appear to be cached.

As an example I recently deleted a directory tree containing a few thousand files and directories on my google-workspace-vault: remote in Windows Explorer and that delete operation proceeded at the rate of about 1-2 files per second (according to the Windows Progress Indicator).

Am I missing a parameter that would also cache/buffer interactive file delete operations or is this simply not implemented? I realize that there might be more performant ways to accomplish this task of mass file deletion (perhaps via direct rclone commands?), but I'm enjoying the seamless integration into Explorer and would prefer to work through the Windows UI.

Just to be clear I don't care how long a cached delete operation might take in the background, it would just be nice to have the "interactive" part of the deletion finish quicker than it does in my current configuration. So can I do something to improve that?

What is your rclone version (output from rclone version)

rclone 1.57.0 for Windows/AMD64 (running on Windows 10 Pro 21H1, OS Build 19043.1348)

FWIW my WinFSP version is 1.9.21096

Which cloud storage system are you using? (eg Google Drive)

I'm accessing a mounted remote of type crypt that points to a subfolder on Google Drive (which comes with a Google Workspace account).

The command you were trying to run (eg rclone copy /tmp remote:tmp)

My mount command line is

rclone mount google-workspace-vault: V: --volname "Google Workspace RCrypt Vault" --drive-acknowledge-abuse --network-mode --vfs-cache-mode full --vfs-cache-max-size 64G --daemon-timeout 599s --bwlimit 1200k -v

The issue/observation itself happens during direct interaction (file deletion in Windows Explorer).

The rclone config contents with secrets removed.

[google-workspace]
type = drive
client_id = ***
client_secret = ***
scope = drive
token = ***
team_drive = 

[google-workspace-vault]
type = crypt
remote = google-workspace:rcrypt_vault
password = ***

A log from the command with the -vv flag

Due to the length of the logfile I'm just providing a link here: Dropbox - rclone_log_delete_files.txt - Simplify your life

Please note that the file delete operation that I logged was done with only a small quantity of files to avoid making the log even more bloated.
1 Like
2

I don't think the cache is doing what you think.

https://rclone.org/commands/rclone_mount/#vfs-cache-mode-full

As it's documented a bit there.

Google is slow as doing what you are describing and if you Windows Explorer on top of it, I wouldn't even guess to what could be going on.

3

@ncw
Does VFS support object tombstones in the metadata cache as triggers of lazy removal in background?
If not, do you think it deserves a feature request?

4

At the moment deletes are synchronous in the VFS cache. This means that they are quite slow!

Other operations are synchronous too, like rename.

It is certainly a feature that we could implement given enough user demand, and I don't think it would be too hard. Bear in mind that once we make an operation async, we have no means of reporting failure to the user (other than via rclone logs) though.

So I think it deserves a feature request so we can collect user interest in the feature.

5

@ivandeex if you don't mind helping me out a bit more, since I'm new to the forums and unfamiliar with the procedures and best practices here: Now that ncw recommended creating a feature request, do I create this here in the forums as a new topic of category "feature" or should I instead create a github issue and tag it as "enhancement"? Or something else altogether?

Thanks so much in advance for your advice.

6
7

Yep, just create a feature request on Github as the link was shared and reference this post.

If you plan to delete large amounts of things though, doing via rclone CLI command rather than a mount would be much easier/faster as it does a better job at grouping things up as you can't do that stuff on the mount.

I tend to use the mount when I have to script a bit more complex things like renames or something along those lines as that does work well on a mount.

The mount is very good at doing things like streaming media with or without the cache. I try to use the right tool for the job even though a hammer tends to fix anything :slight_smile:

9

Thank you. Enhancement request on github has been created: Enhancement of VFS cache for mounted remotes: Cache file and directory Deletions for lazy background processing · Issue #5852 · rclone/rclone · GitHub

10

Thank you for that insight. It makes perfect sense to me that a dedicated rclone delete command would be much, much faster (though I'm wondering if in case of a crypt remote it may have to still do individual deletes, since it may not be able to let the backend handle the deletion...).

However I would ultimately like to keep this remote as a perma-mount, similar to perma-mounting a local server share to my PC. Thus I'm assuming that I would need to dismount this remote every time I'm executing direct rclone commands to avoid any issues with the vfs-cache (once I've verified that the vfs-cache has executed all write requests), correct?

There is some warning about using two mount processes with the same vfs-cache for overlapping remotes, but no explicit statement about running a second rclone process (as a direct command) on an already mounted remote, yet I am concerned that the 2nd process would not be aware of the vfs-cache, which might lead to trouble.

Thus it would seem much more convenient for my workflow to do as much as possible via the mounted remote and only use direct rclone commands as the last resort. But perhaps I am misjudging here, since I'm very new to rclone?

11

You definitely do not want to overlap any of the -cache-dirs on a mount.

You do not need to unmount anything.

You can just run rclone delete remote: with everything going.

Google Drive is polling remote so it'll pick up any changes in a minute based on the default polling interval.

12

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.