sync.CopyDir from S3 to MinIO fails for "NoSuchBucket"

Help and Support
1

Running programmatically with: GitHub - rclone/rclone: "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files v1.60.0

I'm trying to create 2xFS, one S3 and one MinIO and run sync.CopyDir. The function fails for "NoSuchBucket". I dug a little and realized that Rclone tries to run the copy on server side, but it can't be done because S3 is AWS's and MinIO is running locally on my machine.

I'm configuring both as s3 backends, with different provider name ("AWS" and "MinIO"). Obviously MinIO also requires endpoint which is my localhost.

I can successfully sync.CopyDir S3 to S3 and also MinIO to MinIO, but can't do the cross sync.CopyDir.

How can I properly work with that?

2

Hi Elad,

We need some more information to help you, such as

  • The full version info - that is rclone version
  • The redacted config of the two remotes
  • The RC API command you are executing with parameters
  • A debug log showing the issue you see

Without that information it is a bit like being asked to help somebody over the street while blindfolded.

3
  1. Using Golang, not the CLI, v1.60.0
  2. Defining 2 remotes, both are S3 but one of them is configured with a local MinIO endpoint.
  3. Executing sync.CopyDir(ctx, s3fs, miniofs).
  4. The operation succeeds when running between 2 s3 remotes or 2 minio remotes.
  5. The operation fails with NoSuchBucket when trying to copy from s3 to minio. The reason is it tries to run server side copy but it's impossible.
  6. I need a way to programmatically define minio as a separate remote, without a config file.

Does this help?

4

That can be done through the RC API with connection strings.

No sorry, you are pretty much on your own when not using RC API.

The RC API also has a Golang interface in the librclone folder:
https://github.com/rclone/rclone/tree/master/librclone

Why not use that?

5

Guess I'm pretty much on my own then :joy:
librclone looks like a good start.. are there any usage examples available?

6

librclone is an in-memory access to the rc and it is guaranteed stable, unlike rclone internals.

I don't think I made an example for librclone :blush:

The docs are here - it should be reasonably straight forward hopefully!

7

Thanks! I'll take a look :pray:

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.