Requesting support for S3-compatible object-lock properties

With Backblaze having now joined Amazon in supporting object-lock (and internal demand here) it would be beneficial if rclone could support object-locking for any S3-compatible target.

Resources:


Looks interesting...

Want to have a go at sketching out how this might work with rclone? Which additional parameters rclone would need to have?

Hi, thanks for getting back to me.

In my imagining this would need work in the following areas as a first pass - I'm attempting to track calls with Amazon's API reference for consistency:

  1. Bucket Creation (object lock must be set when the bucket is created for objects within it to be lockable):
    --s3-bucket-object-lock-enabled

  2. File upload operations:
    --s3-object-lock-mode [GOVERNANCE|COMPLIANCE]
    --s3-object-lock-retain-until-date [date]
    --s3-object-lock-legal-hold [ON|OFF] (default OFF)

  3. Delete operations:
    --s3-bypass-governance-retention

  4. Enumeration:
    --s3-get-object-lock-configuration
    --s3-get-object-retention
    --s3-get-object-legal-hold

  5. Error handling:
    What do we return when a delete operation was run against a locked object?

API reference: https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations.html

Also I should mention that as far as I know this only applies to S3 implementations with Amazon and Backblaze, so it makes sense to only allow those flags to work with AWS and Other storage types maybe?

Perhaps this could be implied if any of the other object lock modes are specified

I think those seem good. I think --s3-object-lock-mode is the master setting. You'll need to supply that to get it to work at all. I think that --s3-object-lock-retain-until-date might be better as --s3-object-lock-retention-period so users can specify 1d 1y etc from now. You can also put a date in here too 2021-01-01.

I think this is optional isn't it?

Would people upload objects with a legal hold? I guess they might. However I was thinking an additional backend command to add/remove a legal hold might be appropriate rclone backend legal-hold add s3:bucket/path

OK

Rclone doesn't have a great way of surfacing these at the moment.

Maybe an additional rclone backend list-object-lock command or similar might be appropriate.

S3 will return an error, I suggest we just return that.

That is easy enough to arrange in the configurator.

1 Like

Thanks, this seems really promising. The only point I want to emphasize from the above is that object locking will work if and only if object-lock-enabled is set at the time of bucket creation, and only for objects within that bucket. For that reason I believe it's necessary to add that flag to 'rclone mkdir' - and if I'm wrong about this please accept my apology.

I think this is very useful. Would you mind writing it up as a GitHub issue? If you could link it here and vice versa that would be helpful. Thanks

Hi, sorry for the delay. I've created the GitHub issue per your request:

1 Like

Thanks for doing that.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.