MinIO SSE-C synchronization

Help and Support
1

What is the problem you are having with rclone?

I have two different instances of MinIO servers running in two remote servers that have no communication with each other.
In one of the servers I have users storing data in their buckets with customer provided encryption keys. I do not have access to those keys and do not wish to decrypt their data.
All I want to do is synchronize the data from one MinIO instance to the other by exporting all the data from one instance into some binary format that I can then import to the other instance.

What is your rclone version (output from rclone version)

rclone v1.52.0

  • os/arch: linux/amd64
  • go version: go1.14.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux, 64 bit

Which cloud storage system are you using? (eg Google Drive)

MinIO self-hosted

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync minio:aris backup/

The rclone config contents with secrets removed.

[minio]
type = s3
provider = Minio
env_auth = false
access_key_id = 
secret_access_key = 
region = us-east-1
endpoint = https://127.0.0.1:9000
acl = private

A log from the command with the -vv flag

2020/06/01 18:24:46 DEBUG : rclone: Version "v1.52.0" starting with parameters ["rclone" "-vv" "sync" "minio:aris" "backup/"]
2020/06/01 18:24:46 DEBUG : Using config file from "/home/aris/.config/rclone/rclone.conf"
2020/06/01 18:24:46 DEBUG : fs cache: renaming cache item "backup/" to be canonical "/home/aris/dev/eclipse-workspace/minio/backup"
2020/06/01 18:24:46 DEBUG : Local file system at /home/aris/dev/eclipse-workspace/minio/backup: Waiting for checks to finish
2020/06/01 18:24:46 DEBUG : file.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2020/06/01 18:24:46 DEBUG : file.txt: Unchanged skipping
2020/06/01 18:24:46 DEBUG : Local file system at /home/aris/dev/eclipse-workspace/minio/backup: Waiting for transfers to finish
2020/06/01 18:24:46 ERROR : enc_file.txt: Failed to copy: failed to open source object: InvalidArgument: Requests specifying Server Side Encryption with Customer provided keys must provide a valid encryption algorithm.
	status code: 400, request id: 16147829117C713C, host id: 
2020/06/01 18:24:46 ERROR : Local file system at /home/aris/dev/eclipse-workspace/minio/backup: not deleting files as there were IO errors
2020/06/01 18:24:46 ERROR : Local file system at /home/aris/dev/eclipse-workspace/minio/backup: not deleting directories as there were IO errors
2020/06/01 18:24:46 INFO  : There was nothing to transfer
2020/06/01 18:24:46 ERROR : Attempt 1/3 failed with 1 errors and: failed to open source object: InvalidArgument: Requests specifying Server Side Encryption with Customer provided keys must provide a valid encryption algorithm.
	status code: 400, request id: 16147829117C713C, host id: 
2020/06/01 18:24:46 DEBUG : Local file system at /home/aris/dev/eclipse-workspace/minio/backup: Waiting for checks to finish
2020/06/01 18:24:46 DEBUG : file.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2020/06/01 18:24:46 DEBUG : file.txt: Unchanged skipping
2020/06/01 18:24:46 DEBUG : Local file system at /home/aris/dev/eclipse-workspace/minio/backup: Waiting for transfers to finish
2020/06/01 18:24:46 ERROR : enc_file.txt: Failed to copy: failed to open source object: InvalidArgument: Requests specifying Server Side Encryption with Customer provided keys must provide a valid encryption algorithm.
	status code: 400, request id: 1614782911B16445, host id: 
2020/06/01 18:24:46 ERROR : Local file system at /home/aris/dev/eclipse-workspace/minio/backup: not deleting files as there were IO errors
2020/06/01 18:24:46 ERROR : Local file system at /home/aris/dev/eclipse-workspace/minio/backup: not deleting directories as there were IO errors
2020/06/01 18:24:46 INFO  : There was nothing to transfer
2020/06/01 18:24:46 ERROR : Attempt 2/3 failed with 1 errors and: failed to open source object: InvalidArgument: Requests specifying Server Side Encryption with Customer provided keys must provide a valid encryption algorithm.
	status code: 400, request id: 1614782911B16445, host id: 
2020/06/01 18:24:46 DEBUG : Local file system at /home/aris/dev/eclipse-workspace/minio/backup: Waiting for checks to finish
2020/06/01 18:24:46 ERROR : enc_file.txt: Failed to copy: failed to open source object: InvalidArgument: Requests specifying Server Side Encryption with Customer provided keys must provide a valid encryption algorithm.
	status code: 400, request id: 1614782911E106A2, host id: 
2020/06/01 18:24:46 DEBUG : file.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2020/06/01 18:24:46 DEBUG : file.txt: Unchanged skipping
2020/06/01 18:24:46 DEBUG : Local file system at /home/aris/dev/eclipse-workspace/minio/backup: Waiting for transfers to finish
2020/06/01 18:24:46 ERROR : Local file system at /home/aris/dev/eclipse-workspace/minio/backup: not deleting files as there were IO errors
2020/06/01 18:24:46 ERROR : Local file system at /home/aris/dev/eclipse-workspace/minio/backup: not deleting directories as there were IO errors
2020/06/01 18:24:46 INFO  : There was nothing to transfer
2020/06/01 18:24:46 ERROR : Attempt 3/3 failed with 1 errors and: failed to open source object: InvalidArgument: Requests specifying Server Side Encryption with Customer provided keys must provide a valid encryption algorithm.
	status code: 400, request id: 1614782911E106A2, host id: 
2020/06/01 18:24:46 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Checks:                 3 / 3, 100%
Elapsed time:         0.0s

2020/06/01 18:24:46 DEBUG : 9 go routines active
2020/06/01 18:24:46 Failed to sync: failed to open source object: InvalidArgument: Requests specifying Server Side Encryption with Customer provided keys must provide a valid encryption algorithm.
	status code: 400, request id: 1614782911E106A2, host id:
2

This seems to be the problem.

So I think you probably need to set sse_customer_algorithm

3

I understand that, but I am not interested in decrypting the data, also, I may not know the customer_algorithm (although it seems like there is only one choice currently).

Even so, assuming that I provide the algorithm, it still fails with

Command

rclone -vv --s3-sse-customer-algorithm AES256 sync minio:aris backup/

Output

2020/06/01 23:23:36 DEBUG : rclone: Version "v1.52.0" starting with parameters ["rclone" "-vv" "--s3-sse-customer-algorithm" "AES256" "sync" "minio:aris" "backup/"]
2020/06/01 23:23:36 DEBUG : Using config file from "/home/aris/.config/rclone/rclone.conf"
2020/06/01 23:23:36 DEBUG : fs cache: renaming cache item "backup/" to be canonical "/home/aris/dev/eclipse-workspace/minio/backup"
2020/06/01 23:23:36 DEBUG : Local file system at /home/aris/dev/eclipse-workspace/minio/backup: Waiting for checks to finish
2020/06/01 23:23:36 DEBUG : file.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2020/06/01 23:23:36 DEBUG : file.txt: Unchanged skipping
2020/06/01 23:23:36 DEBUG : Local file system at /home/aris/dev/eclipse-workspace/minio/backup: Waiting for transfers to finish
2020/06/01 23:23:36 ERROR : enc_file.txt: Failed to copy: failed to open source object: BadRequest: Bad Request
	status code: 400, request id: 16148877BDFFA64B, host id: 
2020/06/01 23:23:36 ERROR : Local file system at /home/aris/dev/eclipse-workspace/minio/backup: not deleting files as there were IO errors
2020/06/01 23:23:36 ERROR : Local file system at /home/aris/dev/eclipse-workspace/minio/backup: not deleting directories as there were IO errors
2020/06/01 23:23:36 INFO  : There was nothing to transfer
2020/06/01 23:23:36 ERROR : Attempt 1/3 failed with 1 errors and: failed to open source object: BadRequest: Bad Request
	status code: 400, request id: 16148877BDFFA64B, host id: 
2020/06/01 23:23:36 DEBUG : Local file system at /home/aris/dev/eclipse-workspace/minio/backup: Waiting for checks to finish
2020/06/01 23:23:36 DEBUG : file.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2020/06/01 23:23:36 DEBUG : file.txt: Unchanged skipping
2020/06/01 23:23:36 DEBUG : Local file system at /home/aris/dev/eclipse-workspace/minio/backup: Waiting for transfers to finish
2020/06/01 23:23:36 ERROR : enc_file.txt: Failed to copy: failed to open source object: BadRequest: Bad Request
	status code: 400, request id: 16148877BE32D884, host id: 
2020/06/01 23:23:36 ERROR : Local file system at /home/aris/dev/eclipse-workspace/minio/backup: not deleting files as there were IO errors
2020/06/01 23:23:36 ERROR : Local file system at /home/aris/dev/eclipse-workspace/minio/backup: not deleting directories as there were IO errors
2020/06/01 23:23:36 INFO  : There was nothing to transfer
2020/06/01 23:23:36 ERROR : Attempt 2/3 failed with 1 errors and: failed to open source object: BadRequest: Bad Request
	status code: 400, request id: 16148877BE32D884, host id: 
2020/06/01 23:23:36 DEBUG : Local file system at /home/aris/dev/eclipse-workspace/minio/backup: Waiting for checks to finish
2020/06/01 23:23:36 ERROR : enc_file.txt: Failed to copy: failed to open source object: BadRequest: Bad Request
	status code: 400, request id: 16148877BE5BA1E7, host id: 
2020/06/01 23:23:36 DEBUG : file.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2020/06/01 23:23:36 DEBUG : file.txt: Unchanged skipping
2020/06/01 23:23:36 DEBUG : Local file system at /home/aris/dev/eclipse-workspace/minio/backup: Waiting for transfers to finish
2020/06/01 23:23:36 ERROR : Local file system at /home/aris/dev/eclipse-workspace/minio/backup: not deleting files as there were IO errors
2020/06/01 23:23:36 ERROR : Local file system at /home/aris/dev/eclipse-workspace/minio/backup: not deleting directories as there were IO errors
2020/06/01 23:23:36 INFO  : There was nothing to transfer
2020/06/01 23:23:36 ERROR : Attempt 3/3 failed with 1 errors and: failed to open source object: BadRequest: Bad Request
	status code: 400, request id: 16148877BE5BA1E7, host id: 
2020/06/01 23:23:36 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Checks:                 3 / 3, 100%
Elapsed time:         0.0s

2020/06/01 23:23:36 DEBUG : 9 go routines active
2020/06/01 23:23:36 Failed to sync: failed to open source object: BadRequest: Bad Request
	status code: 400, request id: 16148877BE5BA1E7, host id:
4

Is it possible to read the data without decrypting it? I haven't heard of that before. If it is possible can you find some docs and I'll see if I can make rclone do it!

5

At filesystem level the data is readable - though scrambled by the encryption. If I use mc to query minio without the proper encryption key it will return something similar to what rclone states

mc cat minio/aris/enc_file.txt

returns

mc: <ERROR> Unable to read from `minio/aris/enc_file.txt`. Requests specifying Server Side Encryption with Customer provided keys must provide a valid encryption algorithm.

But I understand the mc cat checks that there is encryption and therefore "protects" the user from providing non-sense data.
I was/am hoping that rclone uses some underlying S3 API that accesses the raw binary data instead of generating some human-readable ASCII text that mc cat tries to.
To my (limited) knowledge there is no such functionality in minio client.

And here comes the important question (though not related to rclone if this is a limitation of S3/minio), how does an administrator backup his minio server if he does not have access to the customer keys?

6

Seems to be answered here:

If you don't have the keys, I'd assume you just backup the actual data files / location as noted in that issue (it's a long issue).

7

Given that the issue appears to not be related to rclone I am closing this issue.

For anyone looking back at this, copying the actual data files is NOT a proper backup solution, if you then try to reimport the synchronized files, minio no longer understands that the encrypted files are encrypted and treats them as raw making them unusable!

8

It says on the S3 docs about SSE-C

If you lose the encryption key, any GET request for an object without its encryption key fails, and you lose the object.

Which kind of implies it isn't possible to read the object without its encryption key

I'm reasonably sure that this error

InvalidArgument: Requests specifying Server Side Encryption with Customer provided keys must provide a valid encryption algorithm.

Comes from minio itself as I found that message it its source code in the cmd/api-errors.go file.

So alas I don't think it is possible to read the encrypted objects.

It might be worth asking on the minio forum or as an issue if it is possible to read the encrypted objects for backup purposes...

So I think you'll need to get the key from the customer if you want to back up using the S3 API.

9

I would be happy with a backup that is not necessarily S3 API compliant.
I understand the design decision behind failing GET requests for the object without the encryption key, but I still think that the server administrator should have a means (not necessarily S3 API call) to backup his installation without requiring his users to handover their encryption keys.
I have submitted a ticket on minio github here

Thank you for your help!

1 Like
10

I will follow along the issue! Maybe they could add some API to support this...

11

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.