VFS Refresh Directory

What is the problem you are having with rclone?

Unable to force a vfs refresh on a newly created subdirectory (eg: TV/Showname/Season Number) (unable to vfsrefresh the Season Number folder - if that was newly created)

Mount command:
/volume1/rclone/rclone mount this_one: /mnt/gdrive.de/ --rc --rc-web-gui --rc-addr=10.0.1.3:5572 --rc-user rclone --rc-pass rclone --allow-other --config=/volume1/rclone.conf --attr-timeout 5000h --dir-cache-time 5000h --drive-pacer-burst 200 --drive-pacer-min-sleep 10ms --no-checksum --poll-interval 0 --read-only &

On version 1.57 issuing the command, after the creation of The Tourist folder and Season 01 folder:

/volume1/rclone/rclone rc vfs/refresh 'dir=TV/The Tourist/Season 01' recursive=true --rc-user=rclone --rc-pass=rclone --rc-addr=10.0.1.3:5572 --config=/volume1/rclone.conf --drive-pacer-burst 200 --drive-pacer-min-sleep 10ms --timeout 30m

Provides a result of:

{
        "result": {
                "TV/The Tourist/Season 01": "file does not exist"
        }
}

On Version 1.57.0-DEV
Mounted with:
/rclone mount --allow-other --rc --fast-list plex: /mnt/gdrive --cache-db-purge

{
        "result": {
                "TV/The Tourist/Season 01": "OK"
        }
}

Run the command 'rclone version' and share the full output of the command.

Both:

Server 1:

#rclone version
rclone v1.57.0-DEV
- os/version: centos 8.5.2111 (64 bit)
- os/kernel: 4.18.0-305.7.1.el8_4.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.16.12
- go/linking: dynamic
- go/tags: none

Server 2:

ash-4.3# /volume1/rclone/rclone version
rclone v1.57.0
- os/version: unknown
- os/kernel: 3.10.105 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

gdrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

as above in description

Confirming removing --dir-cache-timeout from the mount command, does work by doing the refresh command.

However by logic shouldn't when issuing a rc vfs/refresh check the cache against the source and update?

It appears that this is not the case.

pushing a particular subdirectory vfs/refresh does not check for changes at the source - but only the current cache.

Happy to run some tests as required.

further confirmation of doing a vfs/forget prior to doing the vfs/refresh is successful

/volume1/rclone/rclone rc vfs/forget 'dir=TV/The Tourist/Season 01' --rc-user=rclone --rc-pass=rclone --rc-addr=10.0.1.3:5572 --config=/volume1/rclone.conf
/usr/bin/sleep 5
/volume1/rclone/rclone rc vfs/refresh 'dir=TV/The Tourist/Season 01' --rc-user=rclone --rc-pass=rclone --rc-addr=10.0.1.3:5572 --config=/volume1/rclone.conf --drive-pacer-burst 200 --drive-pacer-min-sleep 10ms --timeout 30m
/usr/bin/sleep 10
+ /volume1/rclone/rclone rc vfs/forget 'dir=TV/The Tourist/Season 01' --rc-user=rclone --rc-pass=rclone --rc-addr=10.0.1.3:5572 --config=/volume1/rclone.conf
{
        "forgotten": [
                "TV/The Tourist/Season 01"
        ]
}
+ /usr/bin/sleep 5
+ /volume1/rclone/rclone rc vfs/refresh 'dir=TV/The Tourist/Season 01' --rc-user=rclone --rc-pass=rclone --rc-addr=10.0.1.3:5572 --config=/volume1/rclone.conf --drive-pacer-burst 200 --drive-pacer-min-sleep 10ms --timeout 30m
{
        "result": {
                "TV/The Tourist/Season 01": "OK"
        }
}

Note: that I have had to do the vfs/forget to maintain the --dir-cache-timeout in my mount command. you do not need to do the vfs/forget if you do not have the --dir-cache-timeout

To me, this makes sense, because you created a new folder outside of the static, read-only mount which you then run refresh against. How would Rclone know in this case that the folder exists? For your use case, it's probably better you use a writeable mount with the default poll interval, so that changes are reflected before you run refresh.

Come to think of it, the mount doesn't have to be writeable for this to work. Simply change --poll-interval from 0 to something that suits your needs (e.g. 15s), and then set your script to wait slightly longer (e.g. 20s) to trigger the refresh. That should do the trick.

EDIT: If you end up making your mount writeable, make sure you also remove --no-checksum.

Google Drive is a polling backend.

Setting the polling to 0 cripples a lot of things like using a high dir cache.

If you remove it or leave it at the default, you don't need to do all the things you are doing and it'll pick up changes by itself on the poll interval.

There's no reason to refresh the mount each time you add something.

So, you're basically confirming what I said :smiley:

Would probably be less confusing to the OP if you started with "Like @VBB said above..." or something like that :stuck_out_tongue:

As always, thanks for chiming in.

1 Like

Just to reiterate, I fully agree with this. My mount is read-only and entirely static, so for my use case, it makes sense.

Thanks guys completely understand.
My assumption was that:

  • we mount gdrive with no polling and read only.
  • we create a folder separately in gdrive.
  • we do a vfs refresh to update "just that folder" (to save updating the whole cache)
  • that folder is now available in cache.

with --dir-cache-timeout set with the mount to a long period (like 5000h) you can not vfs/refresh just that directory. however you can refresh it, if you do a vfs/forget first, then a vfs/refresh. seemed to me that a logic set had broken there but if that's the way it is by design then no probs.

You can set it up however you like as people make things hard on purpose all the time.

It's a polling remote so if you do nothing, it finds new files when it polls which defaults to one minute.

If you disable polling and want to add things outside of the mount, you have to expire your cache if you have a long dir-cache-time.

Your server. Your choice.

That's how I understood your post from last night, yeah. Although it still doesn't make sense to me that a mount that doesn't poll, with a folder created externally, would be able to refresh that particular folder. How does Rclone know of the new folder, even with vfs/forget?

The way I do it, I add to and create on an entirely different mount, and then later run a refresh on the static mount as a whole. Doing so allows refresh to recognize any and all external changes, even if I don't unmount and remount first (as it should).

Maybe I'm just confusing myself here...

And that's what a refresh does, no? Which is why it works for me. See my post above as to why I'm confused about what @drpau experienced.

[quote="VBB, post:11, topic:28412"]
The way I do it, I add to and create on an entirely different mount, and then later run a refresh on the static mount as a whole.......
[/quote] yeah that's right, and it works perfectly fine. For simplicity I also drop into the same gdrive through a different mount too.

I was trying to (and now can with the vfs/forget) just update the single folder and folder contents rather than the whole cache.

because this happens on a per episode/movie, I can get almost instant availability in plex for anything that gets finished uploading. While your vfs refresh takes "about 10 mins" (or there abouts) I can get a refresh done in about 5/10 seconds.

no right or wrong way, just a different way :slight_smile:

Yes, if you have no poll interval. You'd expire the directory and the new item should show. More likely a user process error I'd imagine.

Oh, I fully agree :slight_smile: Was more questioning what was happening and why...

give it a go.

go make manually a folder in your tv directory under "the simpsons" (for example) make "Season 99"
(eg mkdir /mnt/gdrive/TV/The Simpsons/Season 99)

of course if you do an ls /mnt/gdrive/TV/The Simpsons/Season 99 it wont be there (But do it to confirm anyway).

then do:

rclone rc vfs/refresh 'dir=TV/The Simpsons/Season 99'

(it will fail - even though assuming it would do a refresh of that directory against the source).

next do a

rclone rc vfs/forget 'dir=TV/The Simpsons/Season 99'
(it will work even though there is no directory there)

then do the refresh again

rclone rc vfs/refresh 'dir=TV/The Simpsons/Season 99'

and it will be there.

1 Like

I think that's where the --magic flag takes over :wink: @asdffdsa

Your word choices there a bit loose. There can be many working solutions to a problem as mine try to follow the "KISS" concept.

I can make my remote not poll and do many steps to make it work but I'd rather do 0 steps to make it work if it takes 5 seconds longer as more steps == more chances for things to break in the process.

I have used Google Drive for a few years now and my setup for my use case requires very little human interaction ever as it's simplistic but robust.

I think that's what @drpau is going for, unlike myself. I'm a very social being :partying_face:

1 Like

Time to make the donuts :slight_smile:

1 Like