I have a directory with a snapshot of data.
I want to backup that data to a remote but:
I don't want the backup client to be able to delete, overwrite or expire the data on the remote.
I want to keep the last 7 version (or rather days really) of the backup.
I don't want to re-upload complete files if just a few bytes are different.
It seems like 1. and 2. is more about settings on the remote. Unfortunately it seems to defining CRUD permissions are not that straight forward on AWS S3 as one would hope.
Just keeping the last 7 versions on the remote would also allow for tampering. (Null the files and call the backup 7 times)
It seems like this should be possible to do with, versioning, object locking and lifecycle rules. But it does not feel very straight forward.
Has anyone set this up successfully?
As for 3. it seems like that is out of scope for rclone.
Or is there a path I am not seeing yet?
I could imagine chunking and content addressing could do the trick.
I found a couple of interesting post in the realm of this discussion.
But it would be great to get some further pointers.
It is like perfect description of S3 object locking with versioning bucket backup setup. Unfortunately it is something out of scope for rclone. Try some other open source tools designed to do this e.g. kopia:
As mentioned above both 1 and 2 are possible with object locking and versioning in the S3 bucket. There's a working example of this for Google Cloud Storage here:
You would just need to figure out the exact roles that your account would need on the S3 bucket instead of GCS to ensure that they are immutable but they should be fairly similar. Once the bucket is configured just run rclone in a similar way to the script in the immutable-backups git repository linked above, or run the full script.