Moving objects Swift cluster, what is on container enable versioning

Feature
1

Hi everyone, I'm Luan,
I use rclone to transfer objects between 2 Openstack Swift cluster. I meet problem below, the source container have been enabling versioning with flag X-History-Location has set. And I try to move dynamic large object, when move action was successful, the current flow of rclone is try to delete segments and manifest file. But base on Openstack swift documentation, those segments need not delete because those segments will be refer for old version.
I wonder, can we add to current flow with check source container is enabled versioning? If the container was enabled versioning, those segments will reserve otherwise delete the segments as current flow.

My environment:
rclone v1.49.5-DEV

  • os/arch: linux/amd64
  • go version: go1.12.1

Thanks in advance,
Luan

2

That doesn't sound too hard...

Do you want to make a patch?

1 Like
3

Hi @ncw,
I had created a pull request. Could you check and merge it?

Thank in in advance,
Luan

4

Hi @ncw,
During move objects between 2 swift cluster, i was accidentally find out another problem when a DLO have segments what was located at the segment conainer with custom name, the deletion will missed remove segment.
Example,
i store a object have name test_object with dlo format, the manifest store in container_test and the segments store in container_test_test.

the object info:

swift stat container_test test_object
               Account: AUTH_801f3cbc6b134e739432d8c048f3cdb4
             Container: container_test
                Object: test_object
          Content Type: application/octet-stream
        Content Length: 104857600
         Last Modified: Mon, 04 Nov 2019 06:57:49 GMT
                  ETag: "38efd411ec12bdd01ecfc6c4cc13caf7"
              Manifest: container_test_test/test_object/1557744387.033861/104857600/10485760/
            Meta Mtime: 1557744387.033861
         Accept-Ranges: bytes
            Connection: keep-alive
                Server: nginx
            X-Trans-Id: tx12c01345f47749b987e66-005dbfcc01
X-Openstack-Request-Id: tx12c01345f47749b987e66-005dbfcc01
           X-Timestamp: 1572850668.31850

the segments with info

swift list container_test_test --prefix test_object/1557744387.033861/104857600/10485760/
test_object/1557744387.033861/104857600/10485760/00000000
test_object/1557744387.033861/104857600/10485760/00000001
test_object/1557744387.033861/104857600/10485760/00000002
test_object/1557744387.033861/104857600/10485760/00000003
test_object/1557744387.033861/104857600/10485760/00000004
test_object/1557744387.033861/104857600/10485760/00000005
test_object/1557744387.033861/104857600/10485760/00000006
test_object/1557744387.033861/104857600/10485760/00000007
test_object/1557744387.033861/104857600/10485760/00000008
test_object/1557744387.033861/104857600/10485760/00000009

The current flow is not delete segments, because rclone try to delete segments in container segment with format "current_container_name" + "_segments". i try to add code to find actual container what is contained segment.
Please help me review my solution.

Thank in advance.
Luan.

5

Ok will do - thanks!

6

Hi,
I had pushed another commit what refactor code base on your advices.

Thanks,
Luan

1 Like
7

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.