Slow for new files to appear on GCS mounted on windows when written via gsutil

What is the problem you are having with rclone?

I have a google bucket mounted on windows 2016 with this command:

rclone.exe mount --vfs-cache-mode full --vfs-cache-max-age 1m --gcs-bucket-policy-only ":google cloud storage:my_bucket" H: --log-file c:\Windows\Temp\rclone.log'

If I drop files into that bucket directly (ie, via gsutil or the google api) they take several minutes to appear in windows (they show up immediately if I list the bucket with either gsutil or the web console for GCS). I've tried with and without the vfs-cache options. Are there any options to make it more responsive? I'd be happy with 10-20 seconds. There are not normally more than a couple of files in the bucket so the list operation shouldn't take long.

What is your rclone version (output from rclone version)

rclone is at version 1.56.0 , installed via chocolatey

It's not a polling remote as you deleted the part of the template that would have helped out here.

You can reduce dir-cache time as the default is 5 minutes.

hello and welcome to the forum,

that can be tweaked using this, as the default is 5 minutes
but would use API calls and you might be paying for them...

That's really helpful, thanks. I looked up the price and it appears bucket list is a class A operation priced at 5c/10000 requests. So it's going to be less than $1 / month even if I poll every 30 seconds. I'll try setting vfs-directory-cache to one minute to see if my application is happy with that.

might try wasabi, a s3 clone known for hot storage.
they do not charge for api calls and in my testing, very low latency....

not exactly sure about api calls in general and on rclone mount in specific.
listing a bucket can use multiple api requests.

Polling every minute is working well for this, so I'll leave it there - though I'll be checking the bill at the end of the month just to check! We're pretty much tied into Google so changing storage provider would be tricky at this stage, but thanks for the tip on Wasabi - I'll have a read up on them.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.