MD5 error reported when uploading m4a and files larger than 5G

Suspected Bug
1

What is the problem you are having with rclone?

Rclone does not realize the generality of all cloud storage products that are compatible with AWS S3 API, such as Kingsoft cloud KS3. Therefore, it shows that when the big data is uploaded and backed up in blocks, Etag (caused by MD5 inconsistency) is verified, but the Etag uploaded in blocks of Amazon S3 is not the standard MD5, and then the check is skipped.

What is your rclone version (output from rclone version)

Latest version

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Which cloud storage system are you using? (eg Google Drive)

Kingsoft Cloud KS3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

.rclone copyto/root/ExchangeServer2016-x64-cu11.iso ks3-sh:fzy-shtest-181024/ExchangeServer.iso

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

2

For large files rclone adds extra metadata which is the MD5 sum when it uploads them.

It is unfortunately impossible to test the Etag for a multipart upload since it is necessary to know the chunk size of the upload and that isn't recorded anywhere. The chunks could even be different sizes.

What do you think rclone should be doing?

3

In order to be compatible with more cloud storage, we can also optimize the logic of judgment. I recommend that we can make the MD5 verification of block upload configurable and enabled by default.

4

What sort of thing were you thinking of?

5

The specific solution is can rclone set a parameter when uploading? This parameter can ignore Etag verification.
My problem is that when I upload using KS3, I will encounter an error. But I also want to continue to use rclone, which is why I suggest that you support KS3 just as well as Alibaba cloud. I can provide our code here as long as you want to help us.

6

We can do a work-around for KS3 support, sure.

What is the error?

7

Problems I encountered when using rclone

1. In case of uploading text type file, return signature verification mismatch

Process Description
Use the Access_key_id / Secret_access_key with the root account permission of KS3 to use rclone, upload git-log.txt file, and the specific content of error reporting is as follows:
liupengcheng$ rclone copy /Users/liupengcheng/rclone-v1.51.0-osx-amd64/git-log.txt ks3:shanghaidemo
ERROR : git-log.txt: Failed to copy: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="utf-8" standalone="yes"?>

SignatureDoesNotMatch
The request signature we calculated does not match the signature you provided. Check your KS3 secret access key and signing method. For more information, see REST Authentication and SOAP Authentication for details.

Troubleshooting results
Although KS3 computing signature algorithm is compatible with AWS S3, the content type encoding of the file defaults to UTF-8, while rclone is a lowercase UTF-8, which is forced to be changed to uppercase and then uploaded successfully

2.
Process Description:
When I use block upload to upload files larger than 5G, the code is as follows :
[root@vm10-16-87-6 rclone]# ./rclone copyto /root/ExchangeServer2016-x64-cu11.iso ks3-sh:fzy-shtest-181024/ExchangeServer.iso
ERROR : ExchangeServer.iso: corrupted on transfer: MD5 hash differ "6beb50a2743335e6b76f552d2bf8a533" vs "9efe4a62304a6eac0a3101eeb38ee4b9"

The following figure also shows the code area with errors:

问题截图
问题截图1362×477 215 KB

Troubleshooting results
Rclone will verify the MD5 value of the locally calculated file and the Etag (MD5 value) returned from the cloud when uploading the shards. If the MD5 is inconsistent, an error will be reported and the upload fails, which is caused by the different Etag calculation methods of KS3 and Amazon S3.

At last, thank you sooo much for your willingness to help KS3, we will try our best to assist you, here can offer you our Access_key_id / Secret_access_key via email and the bucket used to download and upload files for your testing. If you need any other help, please feel free to contact us. I hope that you can have a nice sunny weekend，greetings and thanks from China.

8

That is very helpful - thank you.

It would be useful if I could have some login details.

I can make a "Kingsoft Cloud" S3 provider which implements the tweaks required.

I don't understand what exactly is happening here. Rclone doesn't check the Etag for files bigger than --s3-upload-custoff - have you set that value?

Can you put the above post in a new issue on github with a link to the forum thread please?

9

Firstly, ‘ Rclone doesn't check the Etag for files bigger than --s3-upload-custoff - have you set that value?‘ for this question , we didn't set a value. We used the default value, also here is our product documentation of multipart upload and Etag, you can be more familiar with the way we upload in blocks. Complete Multipart Upload

Secondly, I have already posted the post in a new issue on github ,here is the link https://github.com/rclone/rclone/issues/4048

Thirdly, we have created a new sub user for you to test, and a bucket named rclone-test. You can upload and download files to KS3. Because access_key_id / secret_access_key is private, please provide an email address. I will send it to you by email

10

Thanks

Great, an issue is the right place for this :slight_smile:

If you could send it to nick@craig-wood.com that would be perfect - thank you.

11

Good morning :smiley: I have already sent you the access_key_id / secret_access_key. Please check the email. And in order to make it easier for you to find it quickly , my email address is WANGZHIJU@kingsoft.com.

1 Like