Creating a file-versioning and a bin system

Help and Support
1

I'll be using rclone sync to sync certain folders to a remote. I understand I can use --backup-dir to store changes into a particular folder but I need more granularity.

Example

Original source remote:

Main
├── bar.txt (containing "old")
└── foo.txt

Modified source remote:

Main
└── bar.txt (Containing "new")

Destination after syncing:

├── Deleted
│   └── Main
│       └── foo.txt
├── History
│   └── Main
│       └── bar.txt
│           └── bar.txt-001 (containing "old")
└── bar.txt (containing "new")

I'm almost certain I'll have to write some code for this - I wish to know the most reliable way to perform this operation since (I believe) it will require multiple commands.

It's preferable to me if there's a tool for achieving this sort of behavior

2

Is there a way to "intercept" rclone sync's behaviour? My plan currently is to read the output of "--dry-run" and do each operation by myself.
That sounds really error-prone though, all I want to do it influence the output directory of files at each step.

3

Suppose you could folk rclone and rewrite to your tastes. However, you could write your own script if you need more control using the basic linux commands.

Also question your usecase: ask why? What's wrong with rclone puting deleted objects in their file structure within the backup folder?

When using sync, copy or move any files which would have been overwritten or deleted are
moved in their original hierarchy into this directory.

And you can play around with --suffix to achive a history.

If --suffix is set, then the moved files will have the suffix added to them. If there is
a file with the same path (after the suffix has been added) in DIR, then it will be
overwritten.

source for my quotes.

To me, sounds like best solution which has already been written, well tested and does a great job. Consider using rclone as intended.

rclone sync local/ remote: --backup-dir remote2: --suffix=date +"%Y%m%d_%T"

1 Like
4

--backup-dir does pretty much what you want, but combines the Deleted and History folders.

I guess you could to two syncs, the first an rclone copy with --backup-dir set to History and then do an rclone sync with --backup-dir set to Deleted - that should do more or less what you want.

Try running a --dry-run with these flags

  --match string            Report all matching files to this file
  --missing-on-dst string   Report all files missing from the destination to this file
  --missing-on-src string   Report all files missing from the source to this file

That will show you what rclone wants to do.

1 Like
5

The real challenge is how are you going to handle errors. As it all looks nice until some operation(s) fail half way because of whatever (network glitch). Some files in your main repo can be already updated, some not. Deleted files - all depends how you configure it. But as long as your algorithm does not produce atomic snapshots it is designed as a solution waiting for disaster - if one day you decide to restore your data as it was at some time in the past.

How much I appreciate rclone I am strongly against using it as a backup software substitute. Of course unless your main goal is to enjoy tinkering and playing with software.

Otherwise I would say - there is no point really given how many options are out there which are battle hardened and used my thousands of users.

1 Like
6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.