Pre/post file sync hooks?

Feature
1

I would like to be able to pass rclone a command string to run before or after each file sync.

More specifically, I want to lock the file before uploading and unlock afterward. Example for uploading VM images:

rclone --pre="virtsh suspend $filename" --post="virtsh resume $filename" sync /var/lib/libvirt/images my-remote:/virt-images/

The $filename would have to be substituted within rclone.

This seems like it might have a few other uses too (like custom encryptions, custom upload monitoring). I have two questions:

  1. Is this a reasonable request or is there a better way to approach my use-case? Is wrapping rclone in a custom script better?

  2. Where within the project should I look to add this functionality myself?

2

Personally, I would wrap it as I want to do some checking to validate return codes and such before I continue on certain things.

3

virtsh suspend $filename && rclone sync /var/lib/libvirt/images my-remote:/virt-images/ && virtsh resume $filename

This may be better.

virtsh suspend $filename && rclone sync /var/lib/libvirt/images my-remote:/virt-images/ ; virtsh resume $filename

4

virtsh suspend $filename && rclone sync /var/lib/libvirt/images my-remote:/virt-images/ ; virtsh resume $filename

this is the temporary solution I've been using, but you need to do that for each file. I have a directory full of files that I want to do like that. Although I could script it in bash, it is an inefficient solution to call rclone many times with one file each.

5

It sounds like you need a snaphot mechanism to take a snapshot at a point in time and then backup the snapshot rather than trying to have rclone run before/after commands for each file. It could be done that way but it still sounds pretty inefficient even if rclone was able to do it.

btrfs could easily do that for you if you wanted to entertain that route.

6

I think you’re 100% correct that I should be using snapshots instead.

The reality of my situation is that I have no backups at all in place for a couple dozen aging systems, so pushing images periodically is meant as a quick-and-dirty improvement as I’m migrating things over.

I need to read some more to understand btrfs and it’s benefits though. I’m still a bit confused after reading the wiki and a quickstart on it. Looks like something I should consider trying out on my next deployment.