Use the google drive "Changes API" to maintain a fresh backup of my entire google drive

Help and Support
1

I would like to maintain a "fresh" backup of my google drive into a local SSD.
This backup will be done by router hardware so with limited resources.

My understanding is that if I run rclone copy/sync daily, rclone will traverse the entire google drive which will take a long time to just to apply a few changes.

I am interested to understand whether rclone can use instead the "Changes API" to simply download the few changes on a daily basis. Through several forum searches, I understand than rclone mount with VFS cache could help here, but I am still confused:
1- Would the VFS Cache hold all the physical files? Rclone documentatino suggests that the cache is smart to only download the files which are accessed, or even only the chunks of files which are needed
2- How is the VFS Cache initialized? Will it download the entire google drive on first run? What if the process restarts - will rclone delete the cache and redownload everything again?
3- Apart from rclone mount, is there a lighter periodic process I could use? My idea was simply: 1) download entire drive once, 2) run rclone to refresh the files on a daily basis

1 Like
2

hello and welcome to the forum,

  1. rclone downloads just the chunks of the file required by whatever app is accessing the mount.
  2. the vfs file cache is not initialized. start, it is empty.
    on process restart, rclone will not delete the files in the vfs file cache.
    will not re-download everything again.
  3. rclone sync.

those change events would be in the log file.

there might be additional way using the rclone rcd, tho not sure how that would work.

3

Not currently.

The VFS cache does support the changes interface.

You'd have to use it like this

  • set up an rclone mount (or an rclone serve webdav or rclone serve sftp) mounting the google drive
  • traverse the mount to fill up the metadata for each file
  • leave it running so the VFS cache fills up with changes
  • rclone sync from the mount to make your backup

You'd use the VFS cache without file caching, so --vfs-cache-mode off and you'd need to tweak the directory caching times so rclone kept them for a while.

4

Thanks for the prompt reply

  • traverse the mount to fill up the metadata for each file

I guess a command like ls -R or find will be able to accomplish that?

  • leave it running so the VFS cache fills up with changes

The VFS cache will update with the Changes api, which will be efficient

  • rclone sync from the mount to make your backup

Correct me if I'm wrong but here rclone sync will traverse the entire mount filesystem and compare it to the destination filesystem. Traversing the mount filesystem should be ok (although I start to worry about memory) but traversing the destination SSD will be CPU intensive and draw lot of current. Would it make sense to add --max-age here to only sync files modified from the latest run?

5

that can work, but there are some common scenarios that will not work.
for example, for a source file older than --max-age
--- if that file is moved, rclone will not re-sync it.
--- if that file is renamed , rclone will not re-sync it.

the OS will just check the file allocation table, should not stress a system.

6

You can use vfs/refresh for this which is probably the most efficient way.

Traversing a local file system, especially on on SSD is very quick and doesn't use a lot of CPU or IO as only the directory entries have to be read into RAM and likely they will be there already.

It takes about 10s to traverse the 1.5M files in my home directory on SSD.

7

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.