The cache is not maintained and the system hangs if the path is accessed during that time

What is the problem you are having with rclone?

Creating a new folder in a particular path invalidates the directory cache and starts rebuilding the cache, and accessing the path in the meantime freeze the system.
Detailed explanation with log below

What is your rclone version (output from rclone version)

Symptoms appear from 1.53.1 ~ 1.53.3 and from the beta version

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Tested on 64-bit Ubuntu 18.04 and 20.10

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount gdrive: /mnt/plexdrive \
--umask 000 \
--allow-other \
--cache-dir=/mnt/rc_cache \
--dir-cache-time 1000h \
--poll-interval 15s \
--log-level INFO \
--log-file /home/j/logs/rclone/rclone-media.log \
--vfs-read-chunk-size=32M \
--vfs-cache-mode full \
--vfs-write-back 5s \
--vfs-cache-max-size 3072G \
--vfs-cache-max-age 336h \
--bwlimit-file 16M \
--buffer-size 16M \
--vfs-read-ahead 32M \

The rclone config contents with secrets removed.

[gdrive]
type = drive
client_id = 
client_secret = 
scope = drive
token = {"access_token":"","token_type":"Bearer","refresh_token":"","expiry":"2020-12-14T07:58:14.267223337+09:00"}

A log from the command with the -vv flag

When changeNotify occurs, it becomes an invalidation directory.
In my case I have about 7000 folders and the system freezes for about 1 minute.
If there are fewer folders, the freezing time is shorter

2020/12/15 02:07:19 DEBUG : : changeNotify: relativePath="VIDEO/영화/test5", type=2
2020/12/15 02:08:09 DEBUG : VIDEO/영화: invalidating directory cache
2020/12/15 02:30:27 DEBUG : Google drive root '': Checking for changes on remote
2020/12/15 02:30:27 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2020/12/15 02:30:27 DEBUG : : changeNotify: relativePath="VIDEO/영화/test8", type=2
2020/12/15 02:30:27 DEBUG : VIDEO/영화: invalidating directory cache
2020/12/15 02:30:27 DEBUG : : >changeNotify: 
2020/12/15 02:30:34 DEBUG : VIDEO/영화/: Attr: 

Can you try this without the cache backend?

Are you saying mount without cache like this?

rclone mount gdrive: /mnt/pd \
--umask=000 \
--allow-other

The same symptoms appear
Regardless of which path I am in, when a new folder is created or the first list is loaded after mounting, if there are many folders in that path, it will hang for a longer time.

I think.
When a new folder is created, rclone re-imports all the lists, and the more the list, the more time it stops.
I am using plex using rclone vfs full mode mount
Assuming you have folders A, B
If a new folder is added under the A folder, there is a problem in using all the images under the A folder.
The videos under the B folder are fine.

I'll also attach the log I tested with the mountrc.log (2.9 MB)

OK, that makes sense - I just wanted to check.

What is happening is that when something changes in a big folder, rclone re-reads the entire folder which in your case takes ages.

We are working on a more precise change notify system which should fix this problem eventually I think.

Thank you for confirming it.
Is the only way to fix this symptom is not to have many folders in the folder?
Is there any way to keep the cache without fetching the list from scratch every time there is a change?

That is one way...

You can disable change notifications using --poll-interval 0. Then the directory structure will only get refreshed when --dir-cache-time expires.

Do you think this could be a regression with the same root cause fixed in this commit?

I know it's not a vfs/refresh call but I believe they use the same codepath?

I think this issue is more straight forward - the directory is very large so takes about ~1 minute to reload. In this time the directory can't be used as a lock is held on it while it is reloading - thus causing the system freeze.

I guess it would be possible to "double buffer" the directory so show the old contents until the new contents had arrived rather than taking a lock.

That would add quite a bit of extra complexity but might be worth it...

Ah, got it.

For my use-case, it takes around 2-3 minutes minimum to load a directory with 10k folders.

I have it handled right now by simply disabling the polling but would obviously prefer the proper solution if and when you have the time.

Another interesting thing is that it seems to invalidate the cache recursively instead of the just that directory which is probably a drawback of the current changeNotify implementation?

We probably only want to do the double buffered read if the directory read was caused by a ChangeNotify message...

If you invalidate a directory, it invalidates everything in it I think

As opposed to the dir-cache time invalidation? Makes sense, yes.

I do know of this code section:

Considering the user's scenario where we receive a changeNotify for VIDEO/영화/test8, then just invalidating the VIDEO/영화 directory non-recursively and then VIDEO/영화/test8 recursively should probably solve the issue but I am not sure if that breaks any other use-case. Can't think of any but I may be missing something.

1 Like