Idea: Subscribing to Google Drive change notifications for directory updates

[Asheesh_Laroia]

Hi all! Big rclone fan here.

I'm setting up a Google Drive rclone mount, and I'm tuning my --poll-interval and --dir-cache-time. It seems if rclone can rely on the Google Drive changes.watch API, it can receive automatic notifications of directory changes. I wanted to share that idea and ask if that's an approach the rclone team would consider adding support for.

I'm interested in having an rclone mount that's as up-to-date as possible w/r/t changes in Google Drive. If I understand correctly, fast directory listings are possible due to the directory cache, so any new directory contents need to wait for the directory cache to expire. In my experience with rclone mount so far, there's a trade-off between freshness (which I want) and fast directory listings (which I also want).

I see from the documentation and source code that --poll-interval is used to periodically retrieve the Google Drive page token. I'm under the impression that this expires the directory cache, based on my rclone mount debug logs.

Google offers an API which will notify us when the page token changes called "changes.watch". The docs say, "Use the changes.watch method to subscribe for updates to the change log."

I wanted to ask if the rclone team would consider adding support for this. For me, that would be the best of both worlds for me! I believe it could be used to support a mode where directory contents are cached permanently, and rclone subscribes to the "changes.watch" API to get a new Google Drive page token and expire directory cache entries based on these notifications. I imagine it might look like "--dir-cache-time auto --poll-interval auto" on the command line.

I did search the forum for changes.watch and didn't find previous discussion of this, but if this is a FAQ, apologies for the noise.

P.S. I would have added links, but I think being a new forum user, I can't add links yet.

[ncw (Nick Craig-Wood)]

@Asheesh_Laroia

I'm interested in having an rclone mount that's as up-to-date as possible w/r/t changes in Google Drive. If I understand correctly, fast directory listings are possible due to the directory cache, so any new directory contents need to wait for the directory cache to expire.

That is incorrect... The changes from drive are used to invalidate the directory immediately.

@Asheesh_Laroia

I wanted to ask if the rclone team would consider adding support for this. For me, that would be the best of both worlds for me! I believe it could be used to support a mode where directory contents are cached permanently, and rclone subscribes to the "changes.watch" API to get a new Google Drive page token and expire directory cache entries based on these notifications. I imagine it might look like "--dir-cache-time auto --poll-interval auto" on the command line.

There are quite a lot of people who do this already. They set --dir-cache-time very large and leave --poll-interval small (1min say).

--poll-interval is the interval rclone uses to poll the drive changes API.

[Asheesh_Laroia]

That is incorrect... The changes from drive are used to invalidate the directory immediately.

Thanks! I was confused about this.

--poll-interval is the interval rclone uses to poll the drive changes API.

That makes sense, and probably works well enough.

I was originally wondering about having Google notify rclone with a push (rather than rclone polling) when the GDrive data changes.

This would involve rclone creating a HTTP(S) server that GDrive can ping when the data does change.

If I set my poll interval to 10 seconds, that's pretty close to immediate, so maybe this isn't a big deal.

Thanks for working on rclone and answering this question!

[Asheesh_Laroia]

poll interval to 10 seconds

OK, this triggers a rate limit. But 60 seconds is probably fine.