Snapshot overlay

Feature
1

I was wondering if there was any thought about adding a "snapshot filesystem" style overlay to a current remote? I would envision it would be similar to a cache layer but instead of caching, obviously it would be a snapshot layer.

In a snapshot filesystem, all file changes are tracked and one can go back to any version of a file or filesystem. This would be very nice to have even to be able to overlay on the local filesystem.

Notes:

  • the filesystem could grow quite fast, especially with changes to large binary files.
  • how to map unchanged files across time (no hard links as it would have to be possible to do in WIndows etc
  • access to the current version if the snapshot overlay is not active (?)

I wouldn't know how to implement this type of thing (git? probably not, sqlite databases for file tracking? not sure). But I think it would be a really useful overlay on top of any remote.

2

hi,

you get get some of those features with a command such as

rclone sync ./source remote:full --backup-dir=remote:incrementals/`date +%Y%m%d.%I%M%S`
1 Like
3

That is interesting, I hadn't thought of putting those command line options together. It seems like much of the infrastructure is there already.

I was thinking more of an actual layer so that the "complexity" could be abstracted out for the user.

4

well, not sure what the "complexity" is.

each time the command is run, there will be a new subfolder with a date.time stamp
the 'user' can simply browse the folders and see the files therein using rclone browser or the website of whatever backend you are using.

also, you can do a rclone mount remote: /path/to/local/folder
and then use your favorite GUI file manager to view a folder such as /path/to/local/folder/incrementals/20210732.1500 and download whatever files you need....

5

I had proposed a chunk-based deuplicating backend which, if ever done (doubtful) could easily also do this. The idea is that it uses content-defined chunking (a la restic) and then has a blob directory and a file directory. Each file the metadata and a list of blocks. I like the idea a lot but it is also a ton of work!

Something like what you said has also been discussed recently (I was dissenting because I think --backup-dir gets that job done too).

6

tl;dr - KISS

imho, i trust rclone because it is simple, no databases, not dedupe backend, no blobs, no blocks, no chunks, no content defined anything.
rclone already has a chunk backend, that has never left beta, cannot use it cause i cannot trust it....

nice and simple,
please rclone, just copy this file from here to there with checksum verification.
and if a file can be crypted along the way, bonus points.

and a tons of bugs, alphas, betas and perhaps perchance, maybe someday become stable.

restic is already a thing that makes use of rclone so why turn rclone into restic light edition.

in my case, i use/trust veeam backup, which has a database, and VSS delta snapshots.
and use rclone just to copy those files to cloud with checksum verification.

7

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.