When uploading a .txt file to Kingsoft object storage, it returns "SignatureDoesNotMatch" error.
additional information:
I was told by Kingsoft crew that the reason of this is KS3 use “Content-Type: text/html; charset=UTF-8” in which is the capitalized “UTF-8” when calculating signature, but rclone set a lowercased “uft-8” when excuting.
So their solution to avoid this error is setting the content-type to “Content-Type: text/html; charset=UTF-8” on the client side.
What is your rclone version (output from rclone version)
rclone v1.51.0
os/arch: linux/amd64
go version: go1.13.7
Which OS you are using and how many bits (eg Windows 7, 64 bit)
centos 7.5, 64 bit
Which cloud storage system are you using? (eg Google Drive)
Kingsoft object storage(KS3)
The command you were trying to run (eg rclone copy /tmp remote:tmp)
A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)
2020/02/21 16:27:37 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "-vv" "copyto" "/root/a.txt" "ks3-remote:bucket-name/a.txt"]
2020/02/21 16:27:37 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2020/02/21 16:27:37 DEBUG : Using v2 auth
2020/02/21 16:27:38 DEBUG : a.txt: Need to transfer - File not found at Destination
2020/02/21 16:27:38 ERROR : a.txt: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="utf-8" standalone="yes"?>
SignatureDoesNotMatch
The request signature we calculated does not match the signature you provided. Check your KS3 secret access key and signing method. For more information, see REST Authentication and SOAP Authentication for details.
/bucket-name/a.txt
requestID-xxxxxxxx
SignatureDoesNotMatch
The request signature we calculated does not match the signature you provided. Check your KS3 secret access key and signing method. For more information, see REST Authentication and SOAP Authentication for details.
/bucket-name/a.txt
requestID-xxxxxxxx
2020/02/21 16:27:38 DEBUG : a.txt: Need to transfer - File not found at Destination
2020/02/21 16:27:38 ERROR : a.txt: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="utf-8" standalone="yes"?>
SignatureDoesNotMatch
The request signature we calculated does not match the signature you provided. Check your KS3 secret access key and signing method. For more information, see REST Authentication and SOAP Authentication for details.
/bucket-name/a.txt
requestID-xxxxxxxx
SignatureDoesNotMatch
The request signature we calculated does not match the signature you provided. Check your KS3 secret access key and signing method. For more information, see REST Authentication and SOAP Authentication for details.
/bucket-name/a.txt
requestID-xxxxxxxx
2020/02/21 16:27:39 DEBUG : a.txt: Need to transfer - File not found at Destination
2020/02/21 16:27:39 ERROR : a.txt: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="utf-8" standalone="yes"?>
SignatureDoesNotMatch
The request signature we calculated does not match the signature you provided. Check your KS3 secret access key and signing method. For more information, see REST Authentication and SOAP Authentication for details.
/bucket-name/a.txt
requestID-xxxxxxxx
SignatureDoesNotMatch
The request signature we calculated does not match the signature you provided. Check your KS3 secret access key and signing method. For more information, see REST Authentication and SOAP Authentication for details.
/bucket-name/a.txt
requestID-xxxxxxxx
SignatureDoesNotMatch
The request signature we calculated does not match the signature you provided. Check your KS3 secret access key and signing method. For more information, see REST Authentication and SOAP Authentication for details.
/bucket-name/a.txt
requestID-xxxxxxxx
Thanks for your kindly responding!
Can you give me a more specific demo, I run the following cmd, but it doesn't work.
rclone copyto --http-headers "charset=UTF-8" /root/a.txt remote:bucket-name/a.txt
[root@vm10-0-0-18 ~]# rclone copyto -vv --http-headers '"Content-Type","text/html; charset=UTF-8"' --dump headers /root/a.txt ks3-remote:bucket-name/a.txt
2020/02/21 18:00:58 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "copyto" "-vv" "--http-headers" ""Content-Type","text/html; charset=UTF-8"" "--dump" "headers" "/root/a.txt" "ks3-remote:bucket-name/a.txt"]
2020/02/21 18:00:58 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2020/02/21 18:00:58 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2020/02/21 18:00:58 DEBUG : Using v2 auth
2020/02/21 18:00:58 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/02/21 18:00:58 DEBUG : HTTP REQUEST (req 0xc0005b6a00)
2020/02/21 18:00:58 DEBUG : HEAD /bucket-name/a.txt HTTP/1.1
Host: ks3-endpoint
User-Agent: rclone/v1.51.0
Authorization: XXXX
Date: Fri, 21 Feb 2020 10:00:58 UTC
SignatureDoesNotMatch
The request signature we calculated does not match the signature you provided. Check your KS3 secret access key and signing method. For more information, see REST Authentication and SOAP Authentication for details.
/bucket-name/a.txt
ececb7b4c32f4276bf12c4a0eb8cd8b2
SignatureDoesNotMatch
The request signature we calculated does not match the signature you provided. Check your KS3 secret access key and signing method. For more information, see REST Authentication and SOAP Authentication for details.
/bucket-name/a.txt
ececb7b4c32f4276bf12c4a0eb8cd8b2
Looks like rclone is either not setting the header or its not displaying them. On second thought, setting the header this way may not be right since every request won't be a html request.
I would say that is a bug in the Kingsoft implementation - rclone works fine with s3/minio/wasabi/ceph/alibaba all of which are independent implementations of the s3 protocol...
Rclone doesn't have a way of directly setting the Content-Type at the moment.
You could fiddle with the OS settings - ultimately mime types for files will come from there.
You could also try this small patch to see if that does work
diff --git a/backend/s3/s3.go b/backend/s3/s3.go
index 94462b4bc..71abc32f0 100644
--- a/backend/s3/s3.go
+++ b/backend/s3/s3.go
@@ -2301,6 +2301,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
// Guess the content type
mimeType := fs.MimeType(ctx, src)
+ mimeType = "Content-Type: text/html; charset=UTF-8"
req := s3.PutObjectInput{
Bucket: &bucket,
ACL: &o.fs.opt.ACL,
I think that is extremely relevant! There is even a reference to rclone in the ticket - great find!
This came up because the java server jetty uppercases the "utf-8" in the incoming request's Content-type charset before the servlet starts. Normally this isn't a problem, but it breaks S3-like signature verification, where client and server sign properties of a request (e.g. the Content-type header).
So I would guess that Kingsoft is using Jetty and has this problem... It looks like it is possible to work-around in the Jetty using code.
It would also be possible, I realise now, to work around this in rclone by upper casing the charset= in fs.MimeType - that said, I don't know what else it would break!