So I ended up hacking something together that mostly fixes this in our UI around rclone.
Basically I found that I just needed to add the flag --s3-server-side-encryption aws:kms to my rclone copy[to] command, but ONLY when the bucket has KMS encryption, never otherwise.
Since this is for a web app that is a UI that wraps rclone, I don't necessarily know beforehand what type of encryption (if any) a bucket has.
Unfortunately I could not find a way to check this within rclone, so I added a bit of boto3 code (our app is written in python). This is a little inelegant since hitherto rclone was the only way the app interacts with local/cloud storage.
But it did the trick.
The code calls the S3 API GetBucketEncryption.
The only problem is if the user does not have permission to call that API.
I finally decided to punt in that case and not add the --s3-server-side-encryption aws:kms flag. Most buckets have the default (SSE) encryption, or they are old and have none at all, so chances are that will be ok.
If it turns out to be an issue, I'll have to revisit my fix.
An interesting side note - our app also allows users to create "folders" in a remote by using (under the hood) rclone touch to create an empty file in the "folder". I found that --s3-server-side-encryption aws:kms is not needed when using rclone touch for this, in buckets that use KMS encryption.
So I guess the issue only arises when the file is small (but greater than 0 bytes).
Noting that the AWS CLI does not have this issue, I wonder if a feature request for rclone to not have to know about bucket encryption types would be welcome?
Thanks for the help. If anyone have suggestions for making my fix better, I am all ears.
Dan