Redux: Some transfers to S3 fail unless `–s3-server-side-encryption aws:kms` is supplied

Help and Support
1

My original thread has been closed and no longer allows replies, but unfortunately the solution there is not a full/workable solution.

Very brief recap (see linked thread above for details):

  • File transfers to S3 buckets encrypted with KMS fail if --s3-server-side-encryption aws:kms is not provided; however, I'm working with a web GUI around rclone where the user will have no idea in advance if the bucket is encrypted with KMS or not.
  • If that flag is used on a bucket that is NOT encrypted with KMS, it will also fail.
  • If we add --s3-upload-cutoff=0 to the command (and leave out --s3-server-side-encryption aws:kms) the transfer will succeed, but will produce output like the following (Since the transfer output is shown to the user, this is going to alarm the user and they will wonder if everything is OK, even though the transfer ended up succeeding.)
2025/06/02 13:52:52 ERROR : test.txt: Failed to copy: multipart upload corrupted: Etag differ: expecting 7689e7cb9a872871f466f8d6c999906f-1 but got 3d3e681bbdf3e8a4e85b3e58e30ea498-1
2025/06/02 13:52:52 ERROR : Attempt 1/3 failed with 1 errors and: multipart upload corrupted: Etag differ: expecting 7689e7cb9a872871f466f8d6c999906f-1 but got 3d3e681bbdf3e8a4e85b3e58e30ea498-1
2025/06/02 13:52:52 ERROR : Attempt 2/3 succeeded
  • I added some boto3 code that checks the bucket encryption type and can add --s3-server-side-encryption aws:kms if needed; however, some users do not have permission to call the API that checks the encryption type. Some of the buckets have KMS encryption and some do not so it is impossible to guess what type of encryption is going to be used.
  • The AWS CLI does not have this problem. Files can be copied to KMS-encrypted buckets with no extra flags required. The same command (with a different bucket name) can be used to copy a file to either a KMS- or AWS-encrypted bucket (or a non-encrypted bucket).
  • The only other hack that occurs to me is to try/except the transfer and if I get the error that indicates this problem, then insert the --s3-server-side-encryption aws:kms flag and try again. Because of the way our app is architected, this would be a major change. As I mentioned, we show the user the output of the transfer in real time, so we'd be showing them all the errors associated with the failed attempt. Also, it takes time for these transfers to fail.

So - is there a real solution to this?

If not I'd like to make a feature request, which is to make rclone indifferent to the type of S3 bucket encryption used, perhaps in whatever way the AWS CLI does so.

If there is interest in implementing this feature, I can open an issue requesting it.

FWIW, the AWS docs say (emphasis added):

Amazon S3 automatically encrypts all new objects that are copied to an S3 bucket. When copying an object, if you don’t specify encryption information in your copy request, the encryption setting of the target object is set to the default encryption configuration of the destination bucket. By default, all buckets have a base level of encryption configuration that uses server-side encryption with Amazon S3 managed keys (SSE-S3). If the destination bucket has a different default encryption configuration, Amazon S3 uses the corresponding encryption key to encrypt the target object copy.

So I'm wondering if rclone can get away without specifying the encryption type.

2

On top of all that, it turns out that objects uploaded by rclone to a KMS encrypted bucket end up with the wrong KMS encryption key ARN, if I do not specify the correct ARN in my copy command.

The bucket is set to use an ARN that ends with 1246, but unless I pass the actual arn in the copy command (and in my case users will not know this), the uploaded object will end up with the wrong ARN.

Adding --ignore-checksum fixes all of this. But we want rclone to compare checksums, without having to do a separate check, that is a key feature of our app. But rclone compares the local file's MD5 sum to the Etag, which won't match if the bucket is KMS-encrypted.

So it would be great if more of this could be handled automatically, and we could upload files to KMS or non-KMS buckets with essentially the same command lines, end up with the correct encryption ARNs, and not sacrifice checksum checking.

Thanks again.