Dropbox API Issue - failed to open source: not an encrypted file - bad magic string (Response/Header Issue)

Help and Support
41

Ole could it be the range header not working as expect resulting in the start of the file always been returned? For an un-encrypted file it likely would not error the download but the file would be junk if using multi threads or multi requests and ok if single thread / single download request. For an encrypted file it would instead break the encrypted data because you are going back to the start of the stream.

Not sure if someone has an easy way to test it to see but could be worth it to see if a say a file directly downloaded from Dropbox webui vs the same file downloaded but not decrypted by reclone was binary same or if the binary where it differs matches the start of the file.

The file would need to be big enough to trigger multi download threads to be a valid test

42

All the symptoms above seem to point to a problem with Range requests. In particular rclone copy not working unless --multithread-streams is 0. The mount not working for large files agrees with this also.

This means that this dropbox thread (as posted above by @andressantamaria2003)

Is very likely the problem.

I have replied to that thread.

I think this is most likely a dropbox bug.

3 Likes
43

You are right, I am suspecting something like this too.

Or perhaps something like this:

rclone mkdir Dropbox:testfolder
echo "Hello Dropbox" | rclone rcat Dropbox:testfolder/hello.txt
rclone mount Dropbox:testfolder /some/local/folder --multi-thread-cutoff=5B
rclone check --download Dropbox:testfolder /some/local/folder

@ncw Do you think something like the above could provoke the issue? (to firmly place the issue at Dropbox)

44

I talk with dropbox support and they said that rclone doesn't report any error and me to contact with rclone support.

45

I managed to reproduce this with a small file file.txt contents shello and using

rclone cat --offset 3 dropbox:file.txt -vv --dump bodies

This made this request

2023/04/25 11:20:46 DEBUG : HTTP REQUEST (req 0xc0009fec00)
2023/04/25 11:20:46 DEBUG : POST /2/files/download HTTP/1.1
Host: content.dropboxapi.com
User-Agent: rclone/v1.63.0-beta.6980.470db6bcd
Content-Length: 0
Authorization: XXXX
Content-Type: application/octet-stream
Dropbox-Api-Arg: {"path":"id:qqS0afUHwS0AAAAAAA6Nig"}
Range: bytes=3-6

And this response which appears to have muddled up the body with the header

2023/04/25 11:20:47 DEBUG : HTTP RESPONSE (req 0xc0009fec00)
2023/04/25 11:20:47 Unsolicited response received on idle HTTP channel starting with "\"\n \n\x18application/octet-stream\x12\x04llo\n"; err=<nil>
2023/04/25 11:20:47 DEBUG : HTTP/1.1 206 Partial Content
Content-Length: 4
Accept-Encoding: identity,gzip
Accept-Ranges: bytes
Content-Range: bytes 3-6/7
Content-Security-Policy: sandbox allow-forms allow-scripts
Content-Type: application/grpc
Date: Tue, 25 Apr 2023 10:20:46 GMT
Dropbox-Api-Result: {"name": "file.txt", "path_lower": "/file.txt", "path_display": "/file.txt", "id": "id:qqS0afUHwS0AAAAAAA6Nig", "client_modified": "2022-11-09T12:59:37Z", "server_modified": "2022-11-09T12:59:41Z", "rev": "5ed093ae1400f082af73a", "size": 7, "is_downloadable": true, "content_hash": "483068e5287bde73b67923dae8c200d461d80997108030841a082e628489993b"}
Etag: W/"5ed093ae1400f082af73a"
Original-Content-Length: 4
Server: envoy
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
X-Dropbox-Request-Id: e186d6db20684aee82e1123611632a3a
X-Dropbox-Response-Origin: far_remote
X-Robots-Tag: noindex, nofollow, noimageindex
3 Likes
47

Some news? same problem here

48

I think we just need to wait now for Dropbox response or solution. :slight_smile:

1 Like
49

Are you able to do this request without the decryption amd compare it to the original file to see if the data in the response is correctly offset and the issue is just a difference in the header as some claim or if the data is actually not offset and the issue is the data not the content type inthe header.

honestly i would not think the content type should really effect rclone unless it is using some sort of manual parsing of the http response instead of a standard lib for making http calls and getting response streams

50

I actually noticed this starting maybe a day ago? I figured I had some odd bad files mingled in, but when I woke up this morning, I had about ~1000 emails about files reporting this message as I was rescanning a library so something is definitely afoot as I can't imagine I suddenly have that much bad media.

51

It looks like Dropbox have failed to unwrap the GRPC used internally. The data is OK just that the HTTP response is corrupted. We'll have to wait for dropbox to fix it.

Note that this bug appeared in the integration tests I run daily at 06:00 UTC today 2023-04-25 so it hasn't been around long, and hopefully they will fix it soon.

4 Likes
52

@ncw - thanks as that's helpful as we play the waiting game then!

1 Like
53

What does soon mean? hehe hours, days, weeks...?

54

That's really a dropbox question and there's no way to answer that.

1 Like
55

@ncw Do you know any possibilities to workaround the issue until fixed?
(That is some flag/setting that will disable range requests in mounts)

56

I was monitoring dropbox mount 24/7 and this problem started exactly yesterday at 22:00 UTC+2 (there was just few corrupted responses, and few correct responses) But after 30 minutes - all files started showing wrong response.

57

Do they know about this issue?

58

Yes

Thanks for the report! We're looking into it. I'll follow up here once I have an update on this.

see Dropbox Forum

Plus a ticket i have open with Dropbox Support:

Hi,

Thanks for the report! We're investigating this API issue. I'll follow up here once I have an update.

Regards,

1 Like
59

I know this is basically the wrong Forum for this specific question, but the current issue above leads me to this question.
I'm currently migrating from Google to DB this is why TV and Movies still work, but Anime and Pron is broken. I just tried to MergerFS DB:Google but since the files are still present in DB it still tries to load them and therefor it doesn't work. Any ideas for this usecase without the need to interfere manually in the future?

60

Keep google drive as backup and if there is any problem in future move to google libraries

61

That's what I'm asking for, automatically. I can't just manually switch around. Or at least, don't want to.
I just somehow need MergerFS to understand that there is an issue so it uses the second mount automatically on the fly.