Yandex.Cloud (S3 API), unexpected MD5 hash mismatch (talk with tech support)

Snufkin · July 10, 2020, 9:30pm

What is the problem you are having with rclone?

MD5 hash mismatch after files upload

What is your rclone version (output from `rclone version`)

1.52.0, 1.52.1, 1.52.2

Which OS you are using and how many bits (eg Windows 7, 64 bit)

FreeBSD 11.4, 64 bit

Which cloud storage system are you using? (eg Google Drive)

Yandex.Cloud (S3 API)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

/path/to/rclone/rclone --config /path/to/rclone/rclone.conf --log-file /path/to/logs/rclone.log --no-update-modtime --timeout 30m --bwlimit 6M --checksum --exclude-from /path/to/rclone/exl.rule -vv --stats-one-line check /long/path/to/VIDEO_TS yandexcloud:/bucket-name/long/path/to/VIDEO_TS

The rclone config contents with secrets removed.

[yandexcloud]
type = s3
provider = Other
env_auth = false
access_key_id = <...>
secret_access_key = <...>
endpoint = storage.yandexcloud.net
storage_class = STANDARD_IA

A log from the command with the `-vv` flag

2020/07/09 21:30:34 DEBUG : rclone: Version "v1.52.2" starting with parameters ["/path/to/rclone/rclone" "--config" "/path/to/rclone/rclone.conf" "--log-file" "/path/to/logs/rclone.log" "--no-update-modtime" "--timeout" "30m" "--bwlimit" "6M" "--checksum" "--exclude-from" "/path/to/rclone/exl.rule" "-vv" "--stats-one-line" "check" "/long/path/to/VIDEO_TS" "yandexcloud:/bucket-name/long/path/to/VIDEO_TS"]
2020/07/09 21:30:34 DEBUG : Using config file from "/path/to/rclone/rclone.conf"
2020/07/09 21:30:34 INFO  : Starting bandwidth limiter at 6MBytes/s
2020/07/09 21:30:34 DEBUG : fs cache: renaming cache item "yandexcloud:/bucket-name/long/path/to/VIDEO_TS" to be canonical "yandexcloud:bucket-name/long/path/to/VIDEO_TS"
2020/07/09 21:30:34 DEBUG : S3 bucket bucket-name path long/path/to/VIDEO_TS: Waiting for checks to finish
2020/07/09 21:30:34 DEBUG : VIDEO_TS.BUP: MD5 = 9ecacb7b6f6fc7f2080b09ea913ccc26 OK
2020/07/09 21:30:34 DEBUG : VIDEO_TS.BUP: OK
2020/07/09 21:30:34 DEBUG : VIDEO_TS.IFO: MD5 = 9ecacb7b6f6fc7f2080b09ea913ccc26 OK
2020/07/09 21:30:34 DEBUG : VIDEO_TS.IFO: OK
2020/07/09 21:30:34 DEBUG : VIDEO_TS.VOB: MD5 = 741261a71ed8873a90b73374220ea569 OK
2020/07/09 21:30:34 DEBUG : VIDEO_TS.VOB: OK
2020/07/09 21:30:34 DEBUG : VTS_01_0.BUP: MD5 = 384630d9636da1f6a82b4e7093000ddd OK
2020/07/09 21:30:34 DEBUG : VTS_01_0.BUP: OK
2020/07/09 21:30:34 DEBUG : VTS_01_0.IFO: MD5 = 384630d9636da1f6a82b4e7093000ddd OK
2020/07/09 21:30:34 DEBUG : VTS_01_0.IFO: OK
2020/07/09 21:30:35 DEBUG : VTS_01_1.VOB: MD5 = 27ba7feaa6eaffda8b4c51be0375333d (Local file system at /long/path/to/VIDEO_TS)
2020/07/09 21:30:35 DEBUG : VTS_01_1.VOB: MD5 = 36bd5ce679ce97325b9973c6a850a6ac (S3 bucket bucket-name path long/path/to/VIDEO_TS)
2020/07/09 21:30:35 ERROR : VTS_01_1.VOB: MD5 differ
2020/07/09 21:30:36 DEBUG : VTS_02_0.BUP: MD5 = 0575c6db8e20fb8e388062eddfb79e2c OK
2020/07/09 21:30:36 DEBUG : VTS_02_0.BUP: OK
2020/07/09 21:30:36 DEBUG : VTS_02_0.IFO: MD5 = 0575c6db8e20fb8e388062eddfb79e2c OK
2020/07/09 21:30:36 DEBUG : VTS_02_0.IFO: OK
2020/07/09 21:30:37 DEBUG : VTS_02_1.VOB: MD5 = aa8f09e25e1c0326451654de8cc5396a (Local file system at /long/path/to/VIDEO_TS)
2020/07/09 21:30:37 DEBUG : VTS_02_1.VOB: MD5 = 5410878ab81ad276863192eaabb7232d (S3 bucket bucket-name path long/path/to/VIDEO_TS)
2020/07/09 21:30:37 ERROR : VTS_02_1.VOB: MD5 differ
2020/07/09 21:30:37 NOTICE: S3 bucket bucket-name path long/path/to/VIDEO_TS: 2 differences found
2020/07/09 21:30:37 NOTICE: S3 bucket bucket-name path long/path/to/VIDEO_TS: 2 errors while checking
2020/07/09 21:30:37 NOTICE: S3 bucket bucket-name path long/path/to/VIDEO_TS: 7 matching files
2020/07/09 21:30:37 DEBUG : 7 go routines active
2020/07/09 21:30:37 Failed to check with 3 errors: last error was: 2 differences found

Looking through the rclone logs, I found a bunch of unexpected records that briefly look like this.

07 July rclone 1.52.2 - Massive 50 GB upload
30 June rclone 1.52.2 - Massive 50 GB upload
23 June rclone 1.52.1 - Regular small upload
16 June rclone 1.52.1 - Regular small upload
09 June rclone 1.52.0 - Regular small upload
02 June rclone 1.52.0 - Regular small upload

On June 07, to collect more detailed information, I narrowed down uploads to the single folder (with home made dvd image). In total, there are 9 files in this folder, and only 2 files are bigger than 200 MB (default for multipart).
I tried rclone 1.52.2, 1.51.0 and v1.50.0 and got an MD5 mismatch in all cases, followed by file uploads.

Latest answer from Yandex.Cloud technical support doesn't shed much light on the root cause.

rclone не использует алгоритмы расчета MD5 облака, если они отличаются от простого расчета MD5 всего файла.
Т.е. для небольших файлов, загружающихся как simple object используется ETag, который совместимо рассчитывается как MD5 от файла; для больших файлов используется multipart upload, и контрольная сумма рассчитывается на стороне клиента и сохраняется в метаданных.
Проблема как раз возникает для больших файлов, где алгоритмы облака не используются. Метаданные объекта, судя по нашим исследованиям, сохраняются адекватно.

Here is Google translation of what they said.

rclone does not use cloud MD5 calculation algorithms if they are different from a simple MD5 calculation of the entire file.
That is, for small files loading as a simple object, an ETag is used, which is compatible as MD5 from the file; for large files, multipart upload is used, and the checksum is calculated on the client side and stored in metadata.
The problem just arises for large files where cloud algorithms are not used. According to our investigation, object metadata is adequately stored.

The ticket is still open with a couple of questions to the tech support guys.

What is the reason for MD5 mismatch even for older rclone versions? It matches before June 23 even for objects larger 200 MB, but failed to match after.
Before the massive upload of large files on July 7, there was similair upload on June 30. And the multipart assumption is not suitable for it, since the log shows file uploads even for small jpg photos from 3 to 7 MB in size.

I would be very grateful for any hint to figure out what happened after all, and on which side the ball is.

ncw · July 10, 2020, 10:29pm

I would download the files from Yandex and check the checksum with rclone md5sum for them.

You might have to use --ignore-checksum .

This will tell you a lot about where the problem is.

If the downloaded files match the local md5sum then it is the Yandex metadata that is corrupted.

If the downloaded files match the Yandex checksum then there has been some corruption.

This could be at Yandex or on your local disk or in transmission. I wouldn't like to say which is more likely but all are possible. A transmission error would require your computer to have bad Ram. A local disk error would require corruption on your hard disk or bad Ram. Corruption at Yandex is probably unlikely since they probably store 3 copies of the data. Not impossible though.

Examining the differences between the local and remote files can be instructive too.

You might want to run memtest86 on your computer. You can also run stressdisk https://github.com/ncw/stressdisk to check the disk out.

Snufkin · July 11, 2020, 2:06pm

Thank you, @ncw, for sharing your thoughts and ideas on this complicated issue.

Your broad view of data integrity question gave me good starting point to create bullet-proof test case and find out which one of two guns is still smoking.

This was the very first step I took to check if my backup data was corrupted. Hashes for both local and downloaded files turned out to be the same. And this became a clear sign that, most likely, only metadata in the cloud (ETag) contains an error.

The following is a brief description of test case scenario that I carried out.

Prerequisites

Focus on a single file VTS_01_1.VOB stored on FreeBSD NAS
Create twin copy VTS_01_1dp.VOB and keep it on MS Windows PC only.

Sequence of steps

1. Calculate local MD5

1st env: FreeBSD NAS
tools: openssl dgst -MD5, rclone md5sum local:
file: VTS_01_1.VOB

2nd env: MS Windows PC
tool: HashTab
file: VTS_01_1dp.VOB

Step result
Identical for both files, in all environments and with all tools.

27ba7feaa6eaffda8b4c51be0375333d  VTS_01_1.VOB
27ba7feaa6eaffda8b4c51be0375333d  VTS_01_1dp.VOB

2. Upload to Yandex.Cloud

1st env: FreeBSD NAS
tool: rclone copy local: remote:
file: VTS_01_1.VOB

2nd env: MS Windows PC
tool: WinSCP copy
file: VTS_01_1dp.VOB

Step result
Both files were sent to the cloud from independent sources using independent tools.

3. Get cloud MD5 (ETag object property)

1st env: FreeBSD NAS
tool: rclone md5sum remote:
files: VTS_01_1.VOB, VTS_01_1dp.VOB

2nd env: MS Windows PC
tool: aws s3api list-objects
files: VTS_01_1.VOB, VTS_01_1dp.VOB

Step result
Identical for both files, in an independent environment, using independent tools.

36bd5ce679ce97325b9973c6a850a6ac  VTS_01_1.VOB	
36bd5ce679ce97325b9973c6a850a6ac  VTS_01_1dp.VOB

Overall result

Hash identical files, being uploaded to the Yandex.Cloud via independent paths and using independent tools, get an identical and invalid ETag (MD5) object property.

36bd5ce679ce97325b9973c6a850a6ac  invalid cloud ETag
27ba7feaa6eaffda8b4c51be0375333d  correct MD5

Solution

If I understand the test results correctly, there is no way to use rclone --checksum option while working with Yandex.Cloud.

Corrupted hash resulted in corrupted trust to the particular cloud service.

Anyway, my dialogue with Yandex tech support guys is not yet complete, and I will keep rclone community informed.

ncw · July 11, 2020, 5:37pm

Interesting. Are the files server side encrypted? The hashes are known to be wrong in that case.

It would be useful if you could post the headers from a HEAD or GET request. You can get rclone to show you this by doing something like

rclone cat remote:file.vob --head 1 -vv --dump headers

I'm sure we can work around this as rclone is capable of storing its own hashes as metadata.

Snufkin · July 11, 2020, 6:53pm

I did not configure any encryption for these files either locally (using rclone) or on the cloud side. Actually, Yandex supports Key Management Service similar to Amazon S3.

Here is log records

2020/07/11 21:08:52 DEBUG : rclone: Version "v1.52.2" starting with parameters ["/path/to/rclone/rclone" "--config" "/path/to/rclone/rclone.conf" "--log-file" "/path/to/logs/rclone.log" "--head" "1" "-vv" "--dump" "headers" "cat" "yandexcloud:backet-name/long/path/to/VIDEO_TS/VTS_01_1.VOB"]
2020/07/11 21:08:52 DEBUG : Using config file from "/path/to/rclone/rclone.conf"
2020/07/11 21:08:52 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2020/07/11 21:08:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/07/11 21:08:52 DEBUG : HTTP REQUEST (req 0xc0001ab000)
2020/07/11 21:08:52 DEBUG : HEAD /backet-name/long/path/to/VIDEO_TS/VTS_01_1.VOB HTTP/1.1
Host: storage.yandexcloud.net
User-Agent: rclone/v1.52.2
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20200711T180852Z

2020/07/11 21:08:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/07/11 21:08:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/07/11 21:08:52 DEBUG : HTTP RESPONSE (req 0xc0001ab000)
2020/07/11 21:08:52 DEBUG : HTTP/1.1 200 OK
Content-Length: 248956928
Accept-Ranges: bytes
Connection: keep-alive
Content-Type: application/octet-stream
Date: Sat, 11 Jul 2020 18:08:52 GMT
Etag: "36bd5ce679ce97325b9973c6a850a6ac-48"
Keep-Alive: timeout=60
Last-Modified: Sat, 11 Jul 2020 07:43:06 GMT
Server: nginx
X-Amz-Meta-Md5chksum: J7p/6qbq/9qLTFG+A3UzPQ==
X-Amz-Meta-Mtime: 1581013835.787353
X-Amz-Request-Id: 44e82aed7db673a6
X-Amz-Storage-Class: COLD
X-Yc-S3-Cloud-Id: b1gon2ju6u8a40luppvc
X-Yc-S3-Folder-Id: b1git8d9qt0un3fcrh3q

2020/07/11 21:08:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/07/11 21:08:52 DEBUG : fs cache: renaming cache item "yandexcloud:backet-name/long/path/to/VIDEO_TS"
2020/07/11 21:08:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/07/11 21:08:52 DEBUG : HTTP REQUEST (req 0xc000110900)
2020/07/11 21:08:52 DEBUG : GET /backet-name?delimiter=%2F&max-keys=1000&prefix=<long/path/to>VIDEO_TS%2F HTTP/1.1
Host: storage.yandexcloud.net
User-Agent: rclone/v1.52.2
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20200711T180852Z
Accept-Encoding: gzip

2020/07/11 21:08:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/07/11 21:08:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/07/11 21:08:52 DEBUG : HTTP RESPONSE (req 0xc000110900)
2020/07/11 21:08:52 DEBUG : HTTP/1.1 200 OK
Content-Length: 3761
Connection: keep-alive
Content-Type: application/xml; charset=UTF-8
Date: Sat, 11 Jul 2020 18:08:52 GMT
Keep-Alive: timeout=60
Server: nginx
X-Amz-Request-Id: 0ac293c2ba5c53d8
X-Yc-S3-Cloud-Id: b1gon2ju6u8a40luppvc
X-Yc-S3-Folder-Id: b1git8d9qt0un3fcrh3q

2020/07/11 21:08:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/07/11 21:08:52 DEBUG : VIDEO_TS.BUP: Excluded
2020/07/11 21:08:52 DEBUG : VIDEO_TS.IFO: Excluded
2020/07/11 21:08:52 DEBUG : VIDEO_TS.VOB: Excluded
2020/07/11 21:08:52 DEBUG : VTS_01_0.BUP: Excluded
2020/07/11 21:08:52 DEBUG : VTS_01_0.IFO: Excluded
2020/07/11 21:08:52 DEBUG : VTS_01_1dp.VOB: Excluded
2020/07/11 21:08:52 DEBUG : VTS_02_0.BUP: Excluded
2020/07/11 21:08:52 DEBUG : VTS_02_0.IFO: Excluded
2020/07/11 21:08:52 DEBUG : VTS_02_1.VOB: Excluded
2020/07/11 21:08:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/07/11 21:08:52 DEBUG : HTTP REQUEST (req 0xc00053b100)
2020/07/11 21:08:52 DEBUG : GET /backet-name/long/path/to/VIDEO_TS/VTS_01_1.VOB HTTP/1.1
Host: storage.yandexcloud.net
User-Agent: rclone/v1.52.2
Authorization: XXXX
Range: bytes=0-0
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20200711T180852Z

2020/07/11 21:08:52 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/07/11 21:08:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/07/11 21:08:52 DEBUG : HTTP RESPONSE (req 0xc00053b100)
2020/07/11 21:08:52 DEBUG : HTTP/1.1 206 Partial Content
Content-Length: 1
Accept-Ranges: bytes
Connection: keep-alive
Content-Range: bytes 0-0/248956928
Content-Type: application/octet-stream
Date: Sat, 11 Jul 2020 18:08:52 GMT
Etag: "36bd5ce679ce97325b9973c6a850a6ac-48"
Keep-Alive: timeout=60
Last-Modified: Sat, 11 Jul 2020 07:43:06 GMT
Server: nginx
X-Amz-Meta-Md5chksum: J7p/6qbq/9qLTFG+A3UzPQ==
X-Amz-Meta-Mtime: 1581013835.787353
X-Amz-Request-Id: 206c6bbcfa471013
X-Amz-Storage-Class: COLD
X-Yc-S3-Cloud-Id: b1gon2ju6u8a40luppvc
X-Yc-S3-Folder-Id: b1git8d9qt0un3fcrh3q

2020/07/11 21:08:52 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/07/11 21:08:52 DEBUG : 4 go routines active

I’m not sure that it’s worth your effort to solve this artificial problem, which, in my opinion, is a plain bug of the cloud service.

ncw · July 14, 2020, 3:05pm

In here we have the Etag that Yandex sends - in this case it is an invalid Etag for a multipart upload. In which case rclone will read the Etag that it wrote in X-Amz-Meta-Md5chksum

You can convert this to hex with python

>>> etag="J7p/6qbq/9qLTFG+A3UzPQ=="
>>> etag.decode("base64").encode("hex")
'27ba7feaa6eaffda8b4c51be0375333d'
>>>

Which is the correct MD5

The 36bd5ce679ce97325b9973c6a850a6ac-48 without the -48 suffix is the invalid Etag. Rclone checks etags against a regexp so it shouldn't accept a -48 suffixed Etag as an md5sum.

I wonder if it is being returned without the -48 suffix in the directory listing - if so that would be a bug at Yandex.

If rclone is somehow not recognising the -48 then that would be a bug in rclone.

Can you do a directory listing of that directory so rclone lsf s3:path/to/dir -vv --dump bodies and find the directory XML. You'll find an entry for the file in question which will have the Etag in hex in it (like this ETag>"992a16dbee992d34ce27ca2e8a67648e-1"</ETag>). I would guess that is incorrect that value so is 36bd5ce679ce97325b9973c6a850a6ac without the -48 but I might be wrong in which case I need to dig into rclone more.

Snufkin · July 14, 2020, 7:47pm

Ah, so here it is. I could not understand how to perform the conversion.

So, let's take a look at the query results

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
	<Name>backet-name</Name>
	<Prefix>long/path/to/VIDEO_TS/</Prefix>
	<Delimiter>/</Delimiter>
	<MaxKeys>1000</MaxKeys>
	<IsTruncated>false</IsTruncated>
	...
	<Contents>
		<Key>long/path/to/VIDEO_TS/VTS_01_1.VOB</Key>
		<LastModified>2020-07-11T07:43:06.627Z</LastModified>
		<ETag>&#34;36bd5ce679ce97325b9973c6a850a6ac&#34;</ETag>
		<Size>248956928</Size>
		<StorageClass>COLD</StorageClass>
	</Contents>
	<Contents>
		<Key>long/path/to/VIDEO_TS/VTS_01_1dp.VOB</Key>
		<LastModified>2020-07-11T07:52:17.533Z</LastModified>
		<ETag>&#34;36bd5ce679ce97325b9973c6a850a6ac&#34;</ETag>
		<Size>248956928</Size>
		<StorageClass>COLD</StorageClass>
	</Contents>
	...
	<Marker></Marker>
</ListBucketResult>

And here is the response to the request using aws s3api CLI

aws s3api list-objects --endpoint-url=https://storage.yandexcloud.net --bucket bucket-name --prefix  'long/path/to/VIDEO_TS'
{
    "Contents": [
        {
            "Key": "long/path/to/VIDEO_TS/VTS_01_1.VOB",
            "LastModified": "2020-07-11T07:43:06.627Z",
            "ETag": "\"36bd5ce679ce97325b9973c6a850a6ac\"",
            "Size": 248956928,
            "StorageClass": "COLD"
        },
        {
            "Key": "long/path/to/VIDEO_TS/VTS_01_1dp.VOB",
            "LastModified": "2020-07-11T07:52:17.533Z",
            "ETag": "\"36bd5ce679ce97325b9973c6a850a6ac\"",
            "Size": 248956928,
            "StorageClass": "COLD"
        }
    ]
}

Curiously, ETag cannot always be used as a checksum.

The entity tag (ETag) is a hash of the object that might not be an MD5 digest of the object data. Whether the ETag is an MD5 digest depends on how the object was created and encrypted. Because the ETag isn't always an MD5 digest, it can't always be used for verifying the integrity of uploaded files.

Ref link: How can I check the integrity of an object uploaded to Amazon S3?

A note about ongoing dialog with Yandex tech support.
It went in a circle, and they still do not see anything except rclone and its versions.
In reply, I sent them the test results published in post #3, when the file is uploaded in another way by another program.

ncw · July 15, 2020, 12:39pm

Thanks, that confirms what I was thinking.

So the problem is that this Etag, which is for a multipart upload, has exactly the same form as a valid MD5SUM . AWS takes care to suffix these with an -NNN so they are not valid MD5SUMs.

In the HEAD request this ETag is returned

Which is suffixed with the -NNN as we expect.

So the bug in Yandex is that the directory listing XML does not return the same value for the Etag as the HEAD request, and the one in the HEAD request is the correct one.

So why is this a problem for rclone? Rclone expects Etags whether read from directory listings or read from HEAD requests to either be

Valid MD5SUMs made from 32 hex digits surrounded by quotes
Some other string which doesn't match that which it ignores.

So Yandex sending back 32 hex digits as an Etag for a multipart upload is confusing rclone.

If you want to send them something to take notice of, send them the results of the HEAD request (you can do this with aws s3api head-object) vs the results of the aws s3api list-objects. The Etag should be identical for both and it clearly isn't.

Snufkin · July 15, 2020, 6:19pm

Brilliantly! I'm pretty sure Sherlock Holmes and Miss Marple are your distant relatives.

Below is the proof.

aws s3api list-objects --endpoint-url=https://storage.yandexcloud.net --bucket evablesseder --prefix 'long/path/to/VIDEO_TS/VTS_01_1.VOB'
{
    "Contents": [
        {
            "Key": "long/path/to/VIDEO_TS/VTS_01_1.VOB",
            "LastModified": "2020-07-11T07:43:06.627Z",
            "ETag": "\"36bd5ce679ce97325b9973c6a850a6ac\"",
            "Size": 248956928,
            "StorageClass": "COLD"
        }
    ]
}

aws s3api head-object --endpoint-url=https://storage.yandexcloud.net --bucket evablesseder --key 'long/path/to/VIDEO_TS/VTS_01_1.VOB'
{
    "AcceptRanges": "bytes",
    "LastModified": "Sat, 11 Jul 2020 07:43:06 GMT",
    "ContentLength": 248956928,
    "ETag": "\"36bd5ce679ce97325b9973c6a850a6ac-48\"",
    "ContentType": "application/octet-stream",
    "Metadata": {
        "Md5chksum": "J7p/6qbq/9qLTFG+A3UzPQ==",
        "Mtime": "1581013835.787353"
    },
    "StorageClass": "COLD"
}

I am going to continue the dialogue with Yandex tech support and get a bug fix from them. It may take a few days, may be weeks, but in any case I will let rclone community know what they propose to do.

ncw · July 15, 2020, 7:16pm

More like a dog with a bone who just won't let it go until the juicy marrow is extracted!

Excellent. That should be enough to convince Yandex there is a problem.

If both Etag's return the -48 version then rclone will work fine I think!

Great! Let us know what happens.

Snufkin · July 16, 2020, 6:30pm

Yandex tech support confirmed the bug in their S3 API.

Во время анализа с нашей стороны мы были сфокусированы и проверяли именно GET/HEAD Object, а вот запрос листинга, который и содержит ошибку, был нами упущен.
Фикс взяли в работу, постараемся выпустить как можно быстрее.

Google translate

During the analysis on our part, we were focused and checked exactly the GET/HEAD Object, but the listing request, which contains the error, was missed by us.
We are already working on bug fix, will try to release it as soon as possible.

And personal thanks to @ncw from Yandex development team

Большое спасибо за участие и помощь в анализе проблемы!

ncw · July 16, 2020, 10:35pm

Great! Look forward to the fix

And well done for persevering with it.

Snufkin · July 23, 2020, 6:21pm

Yandex.Cloud development team reported, and I, for my part, confirm that bug has been fixed and patch has been applied in the production environment.

I would like to thank @ncw for his excellent support in this difficult case.

This topic can be marked solved.

ncw · July 23, 2020, 9:26pm

Great, glad it has been fixed

system · July 26, 2020, 9:26pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.