I'm getting Md5Mismatch

carlos.martinez · June 12, 2024, 12:24am

What is the problem you are having with rclone?

RESPONSE 400: 400 The MD5 value specified in the request did not match with the MD5 value calculated by the server.
ERROR CODE: Md5Mismatch

Run the command 'rclone version' and share the full output of the command.

rclone v1.66.0

os/version: Microsoft Windows 11 Pro for Workstations 23H2 (64 bit)
os/kernel: 10.0.22631.3593 (x86_64)
os/type: windows
os/arch: amd64
go/version: go1.22.1
go/linking: static
go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Amazon S3
Azure Blob Storage

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copy cxonetest:/ao-prod-sea-storage/SEA_4607836/2024-05-21 azureblob:testcxone


rclone copy cxonetest:/ao-prod-sea-storage/SEA_4607836/2024-05-21 azureblob:testcxone --ignore-checksum

The rclone config contents with secrets removed.

Paste config here

A log from the command with the `-vv` flag

2024/06/11 20:22:49 DEBUG : rclone: Version "v1.66.0" starting with parameters ["C:\\rclone-v1.66.0-windows-amd64\\rclone-v1.66.0-windows-amd64\\rclone.exe" "copy" "cxonetest:/ao-prod-sea-storage/SEA_4607836/2024-05-21" "azureblob:testcxone" "--ignore-checksum" "-vv"]
2024/06/11 20:22:49 DEBUG : Creating backend with remote "cxonetest:/ao-prod-sea-storage/SEA_4607836/2024-05-21"
2024/06/11 20:22:49 DEBUG : Using config file from "C:\\Users\\cmdic\\AppData\\Roaming\\rclone\\rclone.conf"
2024/06/11 20:22:49 DEBUG : Resolving service "s3" region "us-west-2"
2024/06/11 20:22:49 DEBUG : fs cache: renaming cache item "cxonetest:/ao-prod-sea-storage/SEA_4607836/2024-05-21" to be canonical "cxonetest:ao-prod-sea-storage/SEA_4607836/2024-05-21"
2024/06/11 20:22:49 DEBUG : Creating backend with remote "azureblob:testcxone"
2024/06/11 20:22:51 DEBUG : 2024-03-22-04-04-08_463365507182_VOICE_e48a6036-c494-4c92-823d-c13ac50b57cb.mp4: Need to transfer - File not found at Destination
2024/06/11 20:22:51 DEBUG : Azure container testcxone: Waiting for checks to finish
2024/06/11 20:22:51 DEBUG : Azure container testcxone: Waiting for transfers to finish
2024/06/11 20:22:52 ERROR : 2024-03-22-04-04-08_463365507182_VOICE_e48a6036-c494-4c92-823d-c13ac50b57cb.mp4: Failed to copy: PUT https://voicerecordingblobs.blob.core.windows.net/testcxone/2024-03-22-04-04-08_463365507182_VOICE_e48a6036-c494-4c92-823d-c13ac50b57cb.mp4
--------------------------------------------------------------------------------
RESPONSE 400: 400 The MD5 value specified in the request did not match with the MD5 value calculated by the server.
ERROR CODE: Md5Mismatch
--------------------------------------------------------------------------------
<?xml version="1.0" encoding="utf-8"?><Error><Code>Md5Mismatch</Code><Message>The MD5 value specified in the request did not match with the MD5 value calculated by the server.
RequestId:c7bb7441-b01e-0002-5b5e-bc9343000000
Time:2024-06-12T00:22:49.7020590Z</Message><UserSpecifiedMd5>/iyP3ZdtOlmCvp0oC/LqXA==</UserSpecifiedMd5><ServerCalculatedMd5>E+50zGiLMXXDw3aSbMDpXg==</ServerCalculatedMd5></Error>
--------------------------------------------------------------------------------
2024/06/11 20:22:52 ERROR : Attempt 1/3 failed with 1 errors and: PUT https://voicerecordingblobs.blob.core.windows.net/testcxone/2024-03-22-04-04-08_463365507182_VOICE_e48a6036-c494-4c92-823d-c13ac50b57cb.mp4
--------------------------------------------------------------------------------
RESPONSE 400: 400 The MD5 value specified in the request did not match with the MD5 value calculated by the server.
ERROR CODE: Md5Mismatch
--------------------------------------------------------------------------------
<?xml version="1.0" encoding="utf-8"?><Error><Code>Md5Mismatch</Code><Message>The MD5 value specified in the request did not match with the MD5 value calculated by the server.
RequestId:c7bb7441-b01e-0002-5b5e-bc9343000000
Time:2024-06-12T00:22:49.7020590Z</Message><UserSpecifiedMd5>/iyP3ZdtOlmCvp0oC/LqXA==</UserSpecifiedMd5><ServerCalculatedMd5>E+50zGiLMXXDw3aSbMDpXg==</ServerCalculatedMd5></Error>
--------------------------------------------------------------------------------
2024/06/11 20:22:52 DEBUG : 2024-03-22-04-04-08_463365507182_VOICE_e48a6036-c494-4c92-823d-c13ac50b57cb.mp4: Need to transfer - File not found at Destination
2024/06/11 20:22:52 DEBUG : Azure container testcxone: Waiting for checks to finish
2024/06/11 20:22:52 DEBUG : Azure container testcxone: Waiting for transfers to finish
2024/06/11 20:22:52 ERROR : 2024-03-22-04-04-08_463365507182_VOICE_e48a6036-c494-4c92-823d-c13ac50b57cb.mp4: Failed to copy: PUT https://voicerecordingblobs.blob.core.windows.net/testcxone/2024-03-22-04-04-08_463365507182_VOICE_e48a6036-c494-4c92-823d-c13ac50b57cb.mp4
--------------------------------------------------------------------------------
RESPONSE 400: 400 The MD5 value specified in the request did not match with the MD5 value calculated by the server.
ERROR CODE: Md5Mismatch
--------------------------------------------------------------------------------
<?xml version="1.0" encoding="utf-8"?><Error><Code>Md5Mismatch</Code><Message>The MD5 value specified in the request did not match with the MD5 value calculated by the server.
RequestId:c7bb7651-b01e-0002-405e-bc9343000000
Time:2024-06-12T00:22:50.2540584Z</Message><UserSpecifiedMd5>/iyP3ZdtOlmCvp0oC/LqXA==</UserSpecifiedMd5><ServerCalculatedMd5>E+50zGiLMXXDw3aSbMDpXg==</ServerCalculatedMd5></Error>
--------------------------------------------------------------------------------
2024/06/11 20:22:52 ERROR : Attempt 2/3 failed with 1 errors and: PUT https://voicerecordingblobs.blob.core.windows.net/testcxone/2024-03-22-04-04-08_463365507182_VOICE_e48a6036-c494-4c92-823d-c13ac50b57cb.mp4
--------------------------------------------------------------------------------
RESPONSE 400: 400 The MD5 value specified in the request did not match with the MD5 value calculated by the server.
ERROR CODE: Md5Mismatch
--------------------------------------------------------------------------------
<?xml version="1.0" encoding="utf-8"?><Error><Code>Md5Mismatch</Code><Message>The MD5 value specified in the request did not match with the MD5 value calculated by the server.
RequestId:c7bb7651-b01e-0002-405e-bc9343000000
Time:2024-06-12T00:22:50.2540584Z</Message><UserSpecifiedMd5>/iyP3ZdtOlmCvp0oC/LqXA==</UserSpecifiedMd5><ServerCalculatedMd5>E+50zGiLMXXDw3aSbMDpXg==</ServerCalculatedMd5></Error>
--------------------------------------------------------------------------------
2024/06/11 20:22:52 DEBUG : 2024-03-22-04-04-08_463365507182_VOICE_e48a6036-c494-4c92-823d-c13ac50b57cb.mp4: Need to transfer - File not found at Destination
2024/06/11 20:22:52 DEBUG : Azure container testcxone: Waiting for checks to finish
2024/06/11 20:22:52 DEBUG : Azure container testcxone: Waiting for transfers to finish
2024/06/11 20:22:53 ERROR : 2024-03-22-04-04-08_463365507182_VOICE_e48a6036-c494-4c92-823d-c13ac50b57cb.mp4: Failed to copy: PUT https://voicerecordingblobs.blob.core.windows.net/testcxone/2024-03-22-04-04-08_463365507182_VOICE_e48a6036-c494-4c92-823d-c13ac50b57cb.mp4
--------------------------------------------------------------------------------
RESPONSE 400: 400 The MD5 value specified in the request did not match with the MD5 value calculated by the server.
ERROR CODE: Md5Mismatch
--------------------------------------------------------------------------------
<?xml version="1.0" encoding="utf-8"?><Error><Code>Md5Mismatch</Code><Message>The MD5 value specified in the request did not match with the MD5 value calculated by the server.
RequestId:c7bb7748-b01e-0002-205e-bc9343000000
Time:2024-06-12T00:22:50.6820152Z</Message><UserSpecifiedMd5>/iyP3ZdtOlmCvp0oC/LqXA==</UserSpecifiedMd5><ServerCalculatedMd5>E+50zGiLMXXDw3aSbMDpXg==</ServerCalculatedMd5></Error>
--------------------------------------------------------------------------------
2024/06/11 20:22:53 ERROR : Attempt 3/3 failed with 1 errors and: PUT https://voicerecordingblobs.blob.core.windows.net/testcxone/2024-03-22-04-04-08_463365507182_VOICE_e48a6036-c494-4c92-823d-c13ac50b57cb.mp4
--------------------------------------------------------------------------------
RESPONSE 400: 400 The MD5 value specified in the request did not match with the MD5 value calculated by the server.
ERROR CODE: Md5Mismatch
--------------------------------------------------------------------------------
<?xml version="1.0" encoding="utf-8"?><Error><Code>Md5Mismatch</Code><Message>The MD5 value specified in the request did not match with the MD5 value calculated by the server.
RequestId:c7bb7748-b01e-0002-205e-bc9343000000
Time:2024-06-12T00:22:50.6820152Z</Message><UserSpecifiedMd5>/iyP3ZdtOlmCvp0oC/LqXA==</UserSpecifiedMd5><ServerCalculatedMd5>E+50zGiLMXXDw3aSbMDpXg==</ServerCalculatedMd5></Error>
--------------------------------------------------------------------------------
2024/06/11 20:22:53 INFO  :
Transferred:      174.006 KiB / 174.006 KiB, 100%, 57.875 KiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:         3.6s

2024/06/11 20:22:53 DEBUG : 8 go routines active
2024/06/11 20:22:53 Failed to copy: PUT https://voicerecordingblobs.blob.core.windows.net/testcxone/2024-03-22-04-04-08_463365507182_VOICE_e48a6036-c494-4c92-823d-c13ac50b57cb.mp4
--------------------------------------------------------------------------------
RESPONSE 400: 400 The MD5 value specified in the request did not match with the MD5 value calculated by the server.
ERROR CODE: Md5Mismatch
--------------------------------------------------------------------------------
<?xml version="1.0" encoding="utf-8"?><Error><Code>Md5Mismatch</Code><Message>The MD5 value specified in the request did not match with the MD5 value calculated by the server.
RequestId:c7bb7748-b01e-0002-205e-bc9343000000
Time:2024-06-12T00:22:50.6820152Z</Message><UserSpecifiedMd5>/iyP3ZdtOlmCvp0oC/LqXA==</UserSpecifiedMd5><ServerCalculatedMd5>E+50zGiLMXXDw3aSbMDpXg==</ServerCalculatedMd5></Error>
--------------------------------------------------------------------------------

asdffdsa · June 12, 2024, 12:32am

welcome to the forum,

to get a deeper look, copy a single file using --dump=headers --retries=1

and post the output of rclone config redacted

ncw · June 12, 2024, 3:58pm

It would be worth trying to copy one of the files that you had a problem with to local disk using rclone, so something like

rclone copy -vv cxonetest:/ao-prod-sea-storage/SEA_4607836/2024-05-21/2024-03-22-04-04-08_463365507182_VOICE_e48a6036-c494-4c92-823d-c13ac50b57cb.mp4 c:\temp\

If that fails too then it is likely the MD5 is wrong on the source server.

Is the AWS bucket using KMS? The hashes are knows to be wrong in that case. Rclone should be dealing with that if you've configured it correctly.

Can you post a redacted copy of your config with rclone config redacted please as suggested above - thanks.

carlos.martinez · June 12, 2024, 5:09pm

Thanks for the quick response,

For the --dump=headers --retries=1, this is what I get:

2024/06/12 13:05:01 NOTICE: Automatically setting -vv as --dump is enabled
2024/06/12 13:05:01 DEBUG : rclone: Version "v1.66.0" starting with parameters ["C:\\rclone-v1.66.0-windows-amd64\\rclone-v1.66.0-windows-amd64\\rclone.exe" "copy" "cxonetest:/ao-prod-sea-storage/SEA_4607836/2024-05-21" "azureblob:testcxone" "--dump=headers" "--retries=1"]
2024/06/12 13:05:01 DEBUG : Creating backend with remote "cxonetest:/ao-prod-sea-storage/SEA_4607836/2024-05-21"
2024/06/12 13:05:01 DEBUG : Using config file from "C:\\Users\\cmdic\\AppData\\Roaming\\rclone\\rclone.conf"
2024/06/12 13:05:01 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/06/12 13:05:01 DEBUG : Resolving service "s3" region "us-west-2"
2024/06/12 13:05:01 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/06/12 13:05:01 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/06/12 13:05:01 DEBUG : HTTP REQUEST (req 0xc0006f97a0)
2024/06/12 13:05:01 DEBUG : HEAD /SEA_4607836/2024-05-21 HTTP/1.1
Host: ao-prod-sea-storage.s3.amazonaws.com
User-Agent: rclone/v1.66.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240612T170501Z

2024/06/12 13:05:01 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/06/12 13:05:01 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/06/12 13:05:01 DEBUG : HTTP RESPONSE (req 0xc0006f97a0)
2024/06/12 13:05:01 DEBUG : HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Wed, 12 Jun 2024 17:05:00 GMT
Server: AmazonS3
X-Amz-Id-2: 9exXtaXqCZ0HB4aExI+5DAqsJHyPUwkY6yXRaiUgorDukctUzFkRSeDRG2DkE2CCoHWSuiRxpV8=
X-Amz-Request-Id: JTEE1G9NPZ5ZQVZD

2024/06/12 13:05:01 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/06/12 13:05:01 DEBUG : fs cache: renaming cache item "cxonetest:/ao-prod-sea-storage/SEA_4607836/2024-05-21" to be canonical "cxonetest:ao-prod-sea-storage/SEA_4607836/2024-05-21"
2024/06/12 13:05:01 DEBUG : Creating backend with remote "azureblob:testcxone"
2024/06/12 13:05:01 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/06/12 13:05:01 DEBUG : HTTP REQUEST (req 0xc000530fc0)
2024/06/12 13:05:01 DEBUG : GET /?delimiter=%2F&encoding-type=url&list-type=2&max-keys=1000&prefix=SEA_4607836%2F2024-05-21%2F HTTP/1.1
Host: ao-prod-sea-storage.s3.amazonaws.com
User-Agent: rclone/v1.66.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240612T170501Z
Accept-Encoding: gzip

2024/06/12 13:05:01 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/06/12 13:05:01 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/06/12 13:05:01 DEBUG : HTTP REQUEST (req 0xc00056ab40)
2024/06/12 13:05:01 DEBUG : GET /common/discovery/instance?api-version=1.1&authorization_endpoint=https%3A%2F%2Flogin.microsoftonline.com%2F15c0e2d9-ea17-4f45-aa46-23564bdf04de%2Foauth2%2Fv2.0%2Fauthorize HTTP/1.1
Host: login.microsoftonline.com
User-Agent: rclone/v1.66.0
Accept-Encoding: gzip

2024/06/12 13:05:01 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/06/12 13:05:01 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/06/12 13:05:01 DEBUG : HTTP RESPONSE (req 0xc000530fc0)
2024/06/12 13:05:01 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Wed, 12 Jun 2024 17:05:01 GMT
Server: AmazonS3
X-Amz-Bucket-Region: us-west-2
X-Amz-Id-2: hAuzuMUIL/D7QrNu3JYRFK8u4AqtbCHSwNSRYpEtiFh9/S2iD0c8Zf4umiY/UX4o6kl1BZOeSP0=
X-Amz-Request-Id: JTE8QGBE9BXTZQV9

2024/06/12 13:05:01 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/06/12 13:05:02 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/06/12 13:05:02 DEBUG : HTTP RESPONSE (req 0xc00056ab40)
2024/06/12 13:05:02 DEBUG : HTTP/1.1 200 OK
Content-Length: 980
Access-Control-Allow-Methods: GET, OPTIONS
Access-Control-Allow-Origin: *
Cache-Control: max-age=86400, private
Content-Type: application/json; charset=utf-8
Date: Wed, 12 Jun 2024 17:05:00 GMT
P3p: CP="DSP CUR OTPi IND OTRi ONL FIN"
Set-Cookie: fpc=AnJ5LnRSC1BDq4g89tfbnVM; expires=Fri, 12-Jul-2024 17:05:01 GMT; path=/; secure; HttpOnly; SameSite=None
Set-Cookie: esctx=PAQABBwEAAAApTwJmzXqdR4BN2miheQMY9crkbsiGDPIDBV9QxfNiNdeTe5wYFGnpfwYQS-864X6OL47HKtn560stkI1Qdoz-hHDwm44EP6SB8H5xjyf9H-5XUneSdSWI49iBOaSejYK9UXrJnfhJmHNEx-1hdx5OFSmijxUY_HBf8PgmdKEd7qH-9Drhc1-izchB6VR9QacgAA; domain=.login.microsoftonline.com; path=/; secure; HttpOnly; SameSite=None
Set-Cookie: x-ms-gateway-slice=estsfd; path=/; secure; samesite=none; httponly
Set-Cookie: stsservicecookie=estsfd; path=/; secure; samesite=none; httponly
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Ms-Ests-Server: 2.1.18261.3 - SCUS ProdSlices
X-Ms-Request-Id: 2b0e1416-a2e3-4595-9008-b43c3bb61a00
X-Ms-Srs: 1.P
X-Xss-Protection: 0

2024/06/12 13:05:02 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/06/12 13:05:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/06/12 13:05:02 DEBUG : HTTP REQUEST (req 0xc00056bc20)
2024/06/12 13:05:02 DEBUG : GET /15c0e2d9-ea17-4f45-aa46-23564bdf04de/v2.0/.well-known/openid-configuration HTTP/1.1
Host: login.microsoftonline.com
User-Agent: rclone/v1.66.0
Accept-Encoding: gzip

2024/06/12 13:05:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/06/12 13:05:02 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/06/12 13:05:02 DEBUG : HTTP RESPONSE (req 0xc00056bc20)
2024/06/12 13:05:02 DEBUG : HTTP/1.1 200 OK
Content-Length: 1753
Access-Control-Allow-Methods: GET, OPTIONS
Access-Control-Allow-Origin: *
Cache-Control: max-age=86400, private
Content-Type: application/json; charset=utf-8
Date: Wed, 12 Jun 2024 17:05:01 GMT
P3p: CP="DSP CUR OTPi IND OTRi ONL FIN"
Set-Cookie: fpc=Ag-2n_NhORdDvePw3IP8Ito; expires=Fri, 12-Jul-2024 17:05:01 GMT; path=/; secure; HttpOnly; SameSite=None
Set-Cookie: esctx=PAQABBwEAAAApTwJmzXqdR4BN2miheQMYenOR4uEr-JQYvHiaSwI9XIUCvm6gWnAkt59PEdJysSYKJfEDcNcJgkohSGqn2UnDm32nuwUvJp7_2P0O_E2nduBh0r_WHbnDXdIIcftlMnVDvjIAVF5H_JKgFu8CzK5VcwSSYtlvzlYkuGBFZ-ORnfkLPPLUfssaK_tVxaHFE44gAA; domain=.login.microsoftonline.com; path=/; secure; HttpOnly; SameSite=None
Set-Cookie: x-ms-gateway-slice=estsfd; path=/; secure; samesite=none; httponly
Set-Cookie: stsservicecookie=estsfd; path=/; secure; samesite=none; httponly
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Ms-Ests-Server: 2.1.18261.4 - NCUS ProdSlices
X-Ms-Request-Id: dc1cb49e-5353-43f2-b487-41b2ae745a00
X-Ms-Srs: 1.P
X-Xss-Protection: 0

2024/06/12 13:05:02 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/06/12 13:05:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/06/12 13:05:02 DEBUG : HTTP REQUEST (req 0xc0005318c0)
2024/06/12 13:05:02 DEBUG : POST /15c0e2d9-ea17-4f45-aa46-23564bdf04de/oauth2/v2.0/token HTTP/1.1
Host: login.microsoftonline.com
User-Agent: rclone/v1.66.0
Content-Length: 210
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Accept-Encoding: gzip

2024/06/12 13:05:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/06/12 13:05:02 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/06/12 13:05:02 DEBUG : HTTP RESPONSE (req 0xc0005318c0)
2024/06/12 13:05:02 DEBUG : HTTP/1.1 200 OK
Content-Length: 1392
Cache-Control: no-store, no-cache
Content-Type: application/json; charset=utf-8
Date: Wed, 12 Jun 2024 17:05:01 GMT
Expires: -1
P3p: CP="DSP CUR OTPi IND OTRi ONL FIN"
Pragma: no-cache
Set-Cookie: fpc=Ah8VJXjoFnNMnAQVeL4a0MUMmOfNAQAAAD3M-90OAAAA; expires=Fri, 12-Jul-2024 17:05:01 GMT; path=/; secure; HttpOnly; SameSite=None
Set-Cookie: x-ms-gateway-slice=estsfd; path=/; secure; samesite=none; httponly
Set-Cookie: stsservicecookie=estsfd; path=/; secure; samesite=none; httponly
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Ms-Ests-Server: 2.1.18261.4 - WUS3 ProdSlices
X-Ms-Request-Id: 58648482-fc4a-458c-af93-be466a645100
X-Ms-Srs: 1.P
X-Xss-Protection: 0

2024/06/12 13:05:02 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/06/12 13:05:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/06/12 13:05:02 DEBUG : HTTP REQUEST (req 0xc00056a240)
2024/06/12 13:05:02 DEBUG : GET /testcxone?comp=list&delimiter=%2F&include=metadata&maxresults=5000&prefix=&restype=container HTTP/1.1
Host: voicerecordingblobs.blob.core.windows.net
User-Agent: rclone/v1.66.0
Accept: application/xml
Authorization: XXXX
x-ms-version: 2023-08-03
Accept-Encoding: gzip

2024/06/12 13:05:02 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/06/12 13:05:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/06/12 13:05:03 DEBUG : HTTP RESPONSE (req 0xc00056a240)
2024/06/12 13:05:03 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Wed, 12 Jun 2024 17:05:01 GMT
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
X-Ms-Request-Id: e0226ce7-401e-0006-0cea-bc1e44000000
X-Ms-Version: 2023-08-03

2024/06/12 13:05:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/06/12 13:05:03 DEBUG : 2024-03-22-04-04-08_463365507182_VOICE_e48a6036-c494-4c92-823d-c13ac50b57cb.mp4: Need to transfer - File not found at Destination
2024/06/12 13:05:03 DEBUG : Azure container testcxone: Waiting for checks to finish
2024/06/12 13:05:03 DEBUG : Azure container testcxone: Waiting for transfers to finish
2024/06/12 13:05:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/06/12 13:05:03 DEBUG : HTTP REQUEST (req 0xc000386480)
2024/06/12 13:05:03 DEBUG : GET /SEA_4607836/2024-05-21/2024-03-22-04-04-08_463365507182_VOICE_e48a6036-c494-4c92-823d-c13ac50b57cb.mp4 HTTP/1.1
Host: ao-prod-sea-storage.s3.amazonaws.com
User-Agent: rclone/v1.66.0
Accept-Encoding: gzip
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240612T170503Z

2024/06/12 13:05:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/06/12 13:05:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/06/12 13:05:03 DEBUG : HTTP RESPONSE (req 0xc000386480)
2024/06/12 13:05:03 DEBUG : HTTP/1.1 200 OK
Content-Length: 59394
Accept-Ranges: bytes
Content-Type: application/x-www-form-urlencoded
Date: Wed, 12 Jun 2024 17:05:03 GMT
Etag: "fe2c8fdd976d3a5982be9d280bf2ea5c"
Last-Modified: Tue, 21 May 2024 12:48:38 GMT
Server: AmazonS3
X-Amz-Expiration: expiry-date="Fri, 21 Jun 2024 00:00:00 GMT", rule-id="SEA-Delete-LifecycleRule"
X-Amz-Id-2: SxNWvCmusq11ycAJScK7wu3DLnVqWoyW+sb43hNzlrgp1FVtAPS85jFgxD3lv5gNEff5wtlY8Og=
X-Amz-Request-Id: WE9JNBTX5CF4Z440
X-Amz-Server-Side-Encryption: aws:kms
X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id: arn:aws:kms:us-west-2:737494165703:key/mrk-effe8ee4f4ff4be6823cf6c6f756c900
X-Amz-Tagging-Count: 2
X-Amz-Version-Id: YqCZnn6Cp2HLgca6NAt6BtMulQRPZnyn

2024/06/12 13:05:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/06/12 13:05:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/06/12 13:05:03 DEBUG : HTTP REQUEST (req 0xc0008207e0)
2024/06/12 13:05:03 DEBUG : PUT /testcxone/2024-03-22-04-04-08_463365507182_VOICE_e48a6036-c494-4c92-823d-c13ac50b57cb.mp4 HTTP/1.1
Host: voicerecordingblobs.blob.core.windows.net
User-Agent: rclone/v1.66.0
Content-Length: 59394
Accept: application/xml
Authorization: XXXX
Content-Type: application/octet-stream
x-ms-blob-content-md5: /iyP3ZdtOlmCvp0oC/LqXA==
x-ms-blob-content-type: application/x-www-form-urlencoded
x-ms-blob-type: BlockBlob
x-ms-meta-mtime: 2024-05-21T12:48:38.000000000Z
x-ms-version: 2023-08-03
Accept-Encoding: gzip

2024/06/12 13:05:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/06/12 13:05:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/06/12 13:05:03 DEBUG : HTTP RESPONSE (req 0xc0008207e0)
2024/06/12 13:05:03 DEBUG : HTTP/1.1 400 The MD5 value specified in the request did not match with the MD5 value calculated by the server.
Content-Length: 405
Content-Type: application/xml
Date: Wed, 12 Jun 2024 17:05:02 GMT
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
X-Ms-Error-Code: Md5Mismatch
X-Ms-Request-Id: e0227043-401e-0006-41ea-bc1e44000000
X-Ms-Version: 2023-08-03

2024/06/12 13:05:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/06/12 13:05:03 ERROR : 2024-03-22-04-04-08_463365507182_VOICE_e48a6036-c494-4c92-823d-c13ac50b57cb.mp4: Failed to copy: PUT https://voicerecordingblobs.blob.core.windows.net/testcxone/2024-03-22-04-04-08_463365507182_VOICE_e48a6036-c494-4c92-823d-c13ac50b57cb.mp4
--------------------------------------------------------------------------------
RESPONSE 400: 400 The MD5 value specified in the request did not match with the MD5 value calculated by the server.
ERROR CODE: Md5Mismatch
--------------------------------------------------------------------------------
<?xml version="1.0" encoding="utf-8"?><Error><Code>Md5Mismatch</Code><Message>The MD5 value specified in the request did not match with the MD5 value calculated by the server.
RequestId:e0227043-401e-0006-41ea-bc1e44000000
Time:2024-06-12T17:05:02.9638384Z</Message><UserSpecifiedMd5>/iyP3ZdtOlmCvp0oC/LqXA==</UserSpecifiedMd5><ServerCalculatedMd5>E+50zGiLMXXDw3aSbMDpXg==</ServerCalculatedMd5></Error>
--------------------------------------------------------------------------------
2024/06/12 13:05:03 ERROR : Attempt 1/1 failed with 1 errors and: PUT https://voicerecordingblobs.blob.core.windows.net/testcxone/2024-03-22-04-04-08_463365507182_VOICE_e48a6036-c494-4c92-823d-c13ac50b57cb.mp4
--------------------------------------------------------------------------------
RESPONSE 400: 400 The MD5 value specified in the request did not match with the MD5 value calculated by the server.
ERROR CODE: Md5Mismatch
--------------------------------------------------------------------------------
<?xml version="1.0" encoding="utf-8"?><Error><Code>Md5Mismatch</Code><Message>The MD5 value specified in the request did not match with the MD5 value calculated by the server.
RequestId:e0227043-401e-0006-41ea-bc1e44000000
Time:2024-06-12T17:05:02.9638384Z</Message><UserSpecifiedMd5>/iyP3ZdtOlmCvp0oC/LqXA==</UserSpecifiedMd5><ServerCalculatedMd5>E+50zGiLMXXDw3aSbMDpXg==</ServerCalculatedMd5></Error>
--------------------------------------------------------------------------------
2024/06/12 13:05:03 INFO  :
Transferred:       58.002 KiB / 58.002 KiB, 100%, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         2.8s

2024/06/12 13:05:03 DEBUG : 9 go routines active
2024/06/12 13:05:03 Failed to copy: PUT https://voicerecordingblobs.blob.core.windows.net/testcxone/2024-03-22-04-04-08_463365507182_VOICE_e48a6036-c494-4c92-823d-c13ac50b57cb.mp4
--------------------------------------------------------------------------------
RESPONSE 400: 400 The MD5 value specified in the request did not match with the MD5 value calculated by the server.
ERROR CODE: Md5Mismatch
--------------------------------------------------------------------------------
<?xml version="1.0" encoding="utf-8"?><Error><Code>Md5Mismatch</Code><Message>The MD5 value specified in the request did not match with the MD5 value calculated by the server.
RequestId:e0227043-401e-0006-41ea-bc1e44000000
Time:2024-06-12T17:05:02.9638384Z</Message><UserSpecifiedMd5>/iyP3ZdtOlmCvp0oC/LqXA==</UserSpecifiedMd5><ServerCalculatedMd5>E+50zGiLMXXDw3aSbMDpXg==</ServerCalculatedMd5></Error>
--------------------------------------------------------------------------------

For the rclone config redacted

[azureblob]
type = azureblob
account = XXX
tenant = XXX
client_id = XXX
client_secret = XXX

[cxonetest]
type = s3
provider = AWS
access_key_id = XXX
secret_access_key = XXX
region = us-west-2
endpoint = https://s3.amazonaws.com
location_constraint = us-west-2
### Double check the config for sensitive info before posting publicly

carlos.martinez · June 12, 2024, 5:12pm

Hi,

I tried copying to the local disk and the same happens, the thing is if I copy to the local disk I can use --ignore=checksum and it works, but when I try to copy it directly to the Azure Blob it just fails (it seems it doesn't accept the ignore flag).

Is the AWS bucket using KMS? --Not sure, How do I check it? I'm not the owner of the bucket, our phone company owns that and they post there the phone recordings so we can move it to our storage

asdffdsa · June 12, 2024, 5:25pm

from the dump

X-Amz-Server-Side-Encryption: aws:kms
X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id: arn:aws:kms:us-west-2:737494165703:key/mrk-effe8ee4f4ff4be6823cf6c6f756c900

to your remote config, might try adding
server_side_encryption = aws:kms

asdffdsa · June 12, 2024, 5:31pm

and might also need
sse_kms_key_id=arn:aws:kms:us-west-2:737494165703:key/mrk-effe8ee4f4ff4be6823cf6c6f756c900

carlos.martinez · June 12, 2024, 5:53pm

That worked! Thanks

system · June 15, 2024, 5:54pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.