"rclone rc vfs/refresh recursive=true" seems to clear the directory cache

Suspected Bug
1

What is the problem you are having with rclone?

rclone rc vfs/refresh recursive=true seems to clear the directory cache.

What is your rclone version (output from rclone version)

rclone v1.50.2
- os/arch: linux/amd64
- go version: go1.13.4

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Devuan GNU/Linux 3 (beowulf) (Linux version 4.19.67, x86_64)

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

/usr/bin/rclone mount rclone-crypt: /mnt/remote --buffer-size=256M --cache-dir=/mnt/local/.cache/rclone --config=/etc/rclone/rclone.conf --fast-list --links --rc --rc-no-auth --timeout=1h --allow-other --attr-timeout=8760h --default-permissions --dir-cache-time=8760h --poll-interval=0 --umask=002 --vfs-cache-mode=writes -vv

/usr/bin/rclone rc vfs/refresh recursive=true --config=/etc/rclone/rclone.conf -vv

ls -l /mnt/remote

/usr/bin/rclone rc vfs/refresh recursive=false --config=/etc/rclone/rclone.conf -vv

ls -l /mnt/remote

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

Log for rclone mount (formatted for readability)

# /usr/bin/rclone mount rclone-crypt: /mnt/remote --buffer-size=256M --cache-dir=/mnt/local/.cache/rclone --config=/etc/rclone/rclone.conf --fast-list --links --rc --rc-no-auth --timeout=1h --allow-other --attr-timeout=8760h --default-permissions --dir-cache-time=8760h --poll-interval=0 --umask=002 --vfs-cache-mode=writes -vv
2020/01/03 16:33:19 DEBUG : rclone: Version "v1.50.2" starting with parameters ["/usr/bin/rclone" "mount" "rclone-crypt:" "/mnt/remote" "--buffer-size=256M" "--cache-dir=/mnt/local/.cache/rclone" "--config=/etc/rclone/rclone.conf" "--fast-list" "--links" "--rc" "--rc-no-auth" "--timeout=1h" "--allow-other" "--attr-timeout=8760h" "--default-permissions" "--dir-cache-time=8760h" "--poll-interval=0" "--umask=002" "--vfs-cache-mode=writes" "-vv"]
2020/01/03 16:33:19 NOTICE: Serving remote control on http://127.0.0.1:5572/
2020/01/03 16:33:19 DEBUG : Using config file from "/etc/rclone/rclone.conf"
2020/01/03 16:33:19 DEBUG : Encrypted drive 'rclone-crypt:': Mounting on "/mnt/remote"
2020/01/03 16:33:20 DEBUG : vfs cache root is "/mnt/local/.cache/rclone/vfs/rclone-crypt"
2020/01/03 16:33:20 DEBUG : Adding path "vfs/forget" to remote control registry
2020/01/03 16:33:20 DEBUG : Adding path "vfs/refresh" to remote control registry
2020/01/03 16:33:20 DEBUG : Adding path "vfs/poll-interval" to remote control registry
2020/01/03 16:33:20 DEBUG : : Root: 
2020/01/03 16:33:20 DEBUG : : >Root: node=/, err=<nil>

2020/01/03 16:35:00 DEBUG : rc: "vfs/refresh": with parameters map[recursive:true]
2020/01/03 16:35:00 DEBUG : : Reading directory tree
2020/01/03 16:35:01 DEBUG : : Reading directory tree done in 309.447146ms
2020/01/03 16:35:01 DEBUG : rc: "vfs/refresh": reply map[result:map[:OK]]: <nil>

2020/01/03 16:35:51 DEBUG : /: Attr: 
2020/01/03 16:35:51 DEBUG : /: >Attr: attr=valid=8760h0m0s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2020/01/03 16:35:51 DEBUG : /: ReadDirAll: 
2020/01/03 16:35:51 DEBUG : /: >ReadDirAll: item=0, err=<nil>

2020/01/03 16:36:21 DEBUG : rc: "vfs/refresh": with parameters map[recursive:false]
2020/01/03 16:36:21 DEBUG : rc: "vfs/refresh": reply map[result:map[:OK]]: <nil>

2020/01/03 16:36:52 DEBUG : /: Attr: 
2020/01/03 16:36:52 DEBUG : /: >Attr: attr=valid=8760h0m0s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2020/01/03 16:36:52 DEBUG : /: ReadDirAll: 
2020/01/03 16:36:52 DEBUG : /: >ReadDirAll: item=4, err=<nil>
2020/01/03 16:36:52 DEBUG : /: Attr: 
2020/01/03 16:36:52 DEBUG : /: >Attr: attr=valid=8760h0m0s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2020/01/03 16:36:52 DEBUG : /: Lookup: name="dir1"
2020/01/03 16:36:52 DEBUG : /: >Lookup: node=dir1/, err=<nil>
2020/01/03 16:36:52 DEBUG : dir1/: Attr: 
2020/01/03 16:36:52 DEBUG : dir1/: >Attr: attr=valid=8760h0m0s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2020/01/03 16:36:52 DEBUG : /: Lookup: name="dir2"
2020/01/03 16:36:52 DEBUG : /: >Lookup: node=dir2/, err=<nil>
2020/01/03 16:36:52 DEBUG : dir2/: Attr: 
2020/01/03 16:36:52 DEBUG : dir2/: >Attr: attr=valid=8760h0m0s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2020/01/03 16:36:52 DEBUG : /: Lookup: name="file1"
2020/01/03 16:36:52 DEBUG : /: >Lookup: node=file1, err=<nil>
2020/01/03 16:36:52 DEBUG : file1: Attr: 
2020/01/03 16:36:52 DEBUG : file1: >Attr: a=valid=8760h0m0s ino=0 size=0 mode=-rw-rw-r--, err=<nil>
2020/01/03 16:36:52 DEBUG : /: Lookup: name="dir3"
2020/01/03 16:36:52 DEBUG : /: >Lookup: node=dir3/, err=<nil>
2020/01/03 16:36:52 DEBUG : dir3/: Attr: 
2020/01/03 16:36:52 DEBUG : dir3/: >Attr: attr=valid=8760h0m0s ino=0 size=0 mode=drwxrwxr-x, err=<nil>

^C2020/01/03 16:37:50 DEBUG : cache cleaner exiting
2020/01/03 16:37:50 DEBUG : rclone: Version "v1.50.2" finishing with parameters ["/usr/bin/rclone" "mount" "rclone-crypt:" "/mnt/remote" "--buffer-size=256M" "--cache-dir=/mnt/local/.cache/rclone" "--config=/etc/rclone/rclone.conf" "--fast-list" "--links" "--rc" "--rc-no-auth" "--timeout=1h" "--allow-other" "--attr-timeout=8760h" "--default-permissions" "--dir-cache-time=8760h" "--poll-interval=0" "--umask=002" "--vfs-cache-mode=writes" "-vv"]
#

Log for rclone rc + ls (formatted for readability)

# /usr/bin/rclone rc vfs/refresh recursive=true --config=/etc/rclone/rclone.conf -vv
2020/01/03 16:35:00 DEBUG : rclone: Version "v1.50.2" starting with parameters ["/usr/bin/rclone" "rc" "vfs/refresh" "recursive=true" "--config=/etc/rclone/rclone.conf" "-vv"]
{
    "result": {
        "": "OK"
    }
}
2020/01/03 16:35:01 DEBUG : 5 go routines active
2020/01/03 16:35:01 DEBUG : rclone: Version "v1.50.2" finishing with parameters ["/usr/bin/rclone" "rc" "vfs/refresh" "recursive=true" "--config=/etc/rclone/rclone.conf" "-vv"]

# ls -l /mnt/remote
total 0

# /usr/bin/rclone rc vfs/refresh recursive=false --config=/etc/rclone/rclone.conf -vv
2020/01/03 16:36:21 DEBUG : rclone: Version "v1.50.2" starting with parameters ["/usr/bin/rclone" "rc" "vfs/refresh" "recursive=false" "--config=/etc/rclone/rclone.conf" "-vv"]
{
    "result": {
        "": "OK"
    }
}
2020/01/03 16:36:21 DEBUG : 5 go routines active
2020/01/03 16:36:21 DEBUG : rclone: Version "v1.50.2" finishing with parameters ["/usr/bin/rclone" "rc" "vfs/refresh" "recursive=false" "--config=/etc/rclone/rclone.conf" "-vv"]

# ls -l /mnt/remote
total 0
drwxrwxr-x 1 root root 0 Dec 30 14:15 dir1
drwxrwxr-x 1 root root 0 Dec 30 14:15 dir2
-rw-rw-r-- 1 root root 0 Jan  2 23:03 file1
drwxrwxr-x 1 root root 0 Dec 30 14:15 dir3
#

Config

[rclone-drive]
type = drive
scope = drive,drive.appfolder
service_account_file = <service_account_file>
root_folder_id = appDataFolder
impersonate = <email>

[rclone-crypt]
type = crypt
remote = rclone-drive:
filename_encryption = standard
password = <password>
password2 = <password2>
2

I tried reproducing this, but I haven't been able to yet.

The worrying bit is the empty listing after the refresh - that should definitely never happen!

Can you see if you get the same thing just mounting the rclone-drive: remote?

This is suspiciously quick...

What happens if you try to do

rclone ls rclone-drive:

vs

rclone ls --disable ListR rclone-drive:

You could try adding --disable ListR to your mount command.

I wonder if this is something to do with ListR not working with appDataFolder

3

I do. Exactly the same behaviour.

rclone ls rclone-drive: produces no output.
rclone ls --disable ListR rclone-drive: produces lots of output. Yay!

The original mount command with --disable ListR added makes rclone rc vfs/refresh recursive=true work as expected, except now I get rate limited. Oh well.

4

OK so it looks like --fast-list or ListR as it is known internally is broken for appdata.

Can you please make a new issue on github about this with a link to this forum page and I'll see if I can fix it!

Thanks

5

Done! I changed the title a bit to hopefully make it more descriptive: https://github.com/rclone/rclone/issues/3851

1 Like
6

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.