Swift Stack API Support

What is the problem you are having with rclone?

I'm trying to copy data from a Swift Stack bucket to MinIO. Yes, I have a ticket open with them but because I'm using rclone and not their "mc" command I have to work both tickets.

I'm seeing errors on object copies primarily on Swift Stack buckets that are shards like "harbor+segments".

Does rclone suppoert the Swift Stack API?

Does rclone have any issues copying object that are within shards?

The data within is Docket data.

XMinioInvalidObjectNameObject name contains unsupported characters.registry-cdc/segments/2f6/46f636b65722f72656769737472792f76322f7265706f7369746f726965732f6261652f616c70696e652f5f75706c6f6164732f64306239343039662d356430342d346563612d393265382d6465383964386331626435652f64617461f01f9857428c2775282ef2d1d0abeda1d92f9cabd38f6a8a36e33fbc633a8049da39a3ee5e6b4b0d3255bfef95601890afd80709/0000000000000001vc3-registry/vc3-registry/registry-cdc/segments/2f6/46f636b65722f72656769737472792f76322f7265706f7369746f726965732f6261652f616c70696e652f5f75706c6f6164732f64306239343039662d356430342d346563612d393265382d6465383964386331626435652f64617461f01f9857428c2775282ef2d1d0abeda1d92f9cabd38f6a8a36e33fbc633a8049da39a3ee5e6b4b0d3255bfef95601890afd80709/0000000000000001us-west-116F1D0946DDB3B2Bd08e02bb-81bf-48ee-b114-4b890f98b902
2022-05-23 11:51:14 ERROR : registry-cdc/segments/2f6/46f636b65722f72656769737472792f76322f7265706f7369746f726965732f6261652f616c70696e652f5f75706c6f6164732f61306136386663342d616261632d346638652d626632302d6365653437656633633639342f646174614991f6c858f31e54b8c97f8ba8d7601b482d8f2ec4ab652e509ec814368576cfda39a3ee5e6b4b0d3255bfef95601890afd80709/0000000000000001: Failed to copy: s3 upload: 400 Bad Request: <?xml version="1.0" encoding="UTF-8"?>

Run the command 'rclone version' and share the full output of the command.


Which cloud storage system are you using? (eg Google Drive)

Swift Stack and MinIO

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Paste command here

rclone copy swift-AUTH_svc_test: --include "*" minio-test:test-registry --progress

The rclone config contents with secrets removed.

Paste config here

A log from the command with the -vv flag

Paste  log here

Yes it supports the Openstack Swift interface with the swift backend.

It should be able to copy multipart files properly using the main file. You want to exclude the segments from the copy as you'll end up copying the data twice.

The error message is from MinIO not swift though

Are there control characters in the file names?

I think first thing you should do is exclude the segments from the sync, and just sync the files that make up the segments.

So keep in mind and something I wanted to mention. I ran the command with the "*" because as you may know Swift Stack has storage accounts and then buckets within.

My goal to copy 'many' buckets to 1 bucket on MinIo.

That being said if I run and "exclude" - rclone copy swift-AUTH_svc_test: --exclude "+segments" minio-test:test-registry --progress. Do you still think this command will assume that after the ":" I want to copy all buckets within the storage account over to MinIo?

The --dry-run feature never really tells me much when using that.

You are a beast, you must be the only one answering questions on this forum : )

To show a very crude example I'm copying multiple buckets from Swift Stack by way of a Storage Account into one bucket within MinIO.

Swift Storage Account/
-bucket 1
-bucket 2
-bucket 3

-TO -

minio bucket/

If I run the command above I receive an error:

2022/05/26 07:49:03 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
2022/05/26 07:49:03 Failed to create file system for "swift-AUTH_svc_test:": didn't find section in config file

SO I still need to have somethin to tell rclone that I want all of the buckets under the Siwft Storage Account copied, like I previously did with the --include "*" command.

--exclude "**/segments/**" would be better

It should do yes, provided you are using a recent-ish version of rclone.

That looks like there is a typo in the swift-AUTH_svc_test: name that is what didn't find section in config file means

You are correct, I didn't catch that, ugh, the typo. I will try the command in a short few and see if that helps.

So it seems to have worked using that command. I still get some other random error messages like the following - "Failed to copy: multipart upload failed to upload part: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method."

Is there any extra syntax I can add to ignore that error or something?

Thats probably a network failure which would have been retried.

You can run the sync multiple times until it runs clean.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.