Azure Blob snapshot support

elexm · December 6, 2022, 9:18am

Hey!

I just looked at the code for the Azure Blob storage backend and the used SDK. It seems the SDK allows the usage of Blob snapshots with a single command, but the backend has no reference to it.

It doesn't seem I'm able to pass any custom commands to the underlying SDK from the rclone CLI, so I cannot 'finish' a backup by creating a snapshot. Is this possible at all or do I actually need a 'feature request' to use snapshots?

ncw · December 6, 2022, 12:33pm

Firstly note that I've just re-written the Azure blob SDK to use the new SDK! So don't send PRs on azureblob until I've merged that.

You are right, rclone doesn't support blob snapshots at the moment, though according to the docs, supporting versions might be better.

Other backends have support for versions, eg here is the s3 versions support.

Is that the kind of thing you are looking for? If so would your employer like to sponsor the feature?

It would be easy to make an rclone backend command to take a snapshot, but rclone wouldn't be able to read it currently.

You can add backend specific commands - azureblob doesn't have any right now, but if you want to see s3's for example, type rclone backend help s3

elexm · December 6, 2022, 1:10pm

Ah, thanks for your reply.

Versions won't work, afaik, as I am trying to clone an entire interdependent directory structure at a specific point in time. So I would have just frozen it, cloned it and then, after the cloning was successful, created a snapshot:
That way the remote structure would always be coherent.

Sadly, it isn't for my employer. With custom commands I meant for example Graph API calls or something in the context of the rclone process, for example before or after the clone.

ncw · December 7, 2022, 8:20am

It looks easy to create a snapshot of a blob with the new SDK (which I've just merged).

An rclone backend command could be given a directory, iterate it to find all blobs and call the snapshot command on every blob.

Here is an example of an s3 backend command which restores blobs from glacier - the logic would be very similar

github.com

rclone/rclone/blob/d7cb17848dac7ab9b004f3a64f580fcde1d69534/backend/s3/s3.go#L4208


      
          
          
// Command the backend to run a named command
          //
          // The command run is name
          // args may be used to read arguments from
          // opts may be used to read optional arguments from
          //
          // The result should be capable of being JSON encoded
          // If it is a string or a []string it will be shown to the user
          // otherwise it will be JSON encoded and shown to the user like that
          func (f *Fs) Command(ctx context.Context, name string, arg []string, opt map[string]string) (out interface{}, err error) {
          	switch name {
          	case "restore":
          		req := s3.RestoreObjectInput{
          			//Bucket:         &f.rootBucket,
          			//Key:            &encodedDirectory,
          			RestoreRequest: &s3.RestoreRequest{},
          		}
          		if lifetime := opt["lifetime"]; lifetime != "" {
          			ilifetime, err := strconv.ParseInt(lifetime, 10, 64)
          			if err != nil {

And the loop

github.com

rclone/rclone/blob/d7cb17848dac7ab9b004f3a64f580fcde1d69534/backend/s3/s3.go#L4239


      
          			req.RestoreRequest.Description = &description
          		}
          		type status struct {
          			Status string
          			Remote string
          		}
          		var (
          			outMu sync.Mutex
          			out   = []status{}
          		)
          		err = operations.ListFn(ctx, f, func(obj fs.Object) {
          			// Remember this is run --checkers times concurrently
          			o, ok := obj.(*Object)
          			st := status{Status: "OK", Remote: obj.Remote()}
          			defer func() {
          				outMu.Lock()
          				out = append(out, st)
          				outMu.Unlock()
          			}()
          			if operations.SkipDestructive(ctx, obj, "restore") {
          				return

So you could have a go at that to create the snapshot?

elexm · December 7, 2022, 10:06am

Ah, I switched to trying the recent SMB backend with Azure Storage. This enables policy-based snapshotting without any changes in rclone.

system · February 5, 2023, 10:06am

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.