Google Drive server side copy remove mime-type

KinGuy · July 24, 2019, 7:49am

While I was doing some cleanup in my Drive I found some strange Rclone / Drive behavior,
Some file types lost their mime-type and became "Binary" (application/octet-stream) after doing a server side copy. I lost the ability to see a preview for these files or play them in my browser. I had to download them in order to see them.

I made a test case with MP4 video file, JPG photo and PNG photo. I found that if I copy the video or the JPG photo (with JPEG extension) it will lose the correct mime-type. If I copy the PNG or JPG (with JPG extension) I retain the correct mime-type.

some screenshots and console output (running on freenas 11.2):

$ rclone copy drive:test/test_jpg.jpeg drive:test/copy -vv
2019/07/24 10:35:22 DEBUG : rclone: Version "v1.47.0" starting with parameters ["rclone" "copy" "drive:test/test_jpg.jpeg" "drive:test/copy" "-vv"]
2019/07/24 10:35:22 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2019/07/24 10:35:25 DEBUG : test_jpg.jpeg: Couldn't find file - need to transfer
2019/07/24 10:35:27 INFO : test_jpg.jpeg: Copied (server side copy)
2019/07/24 10:35:27 DEBUG : 4 go routines active
2019/07/24 10:35:27 DEBUG : rclone: Version "v1.47.0" finishing with parameters ["rclone" "copy" "drive:test/test_jpg.jpeg" "drive:test/copy" "-vv"]

$ rclone copy drive:test/test_mp4.mp4 drive:test/copy -vv
2019/07/24 10:35:38 DEBUG : rclone: Version "v1.47.0" starting with parameters ["rclone" "copy" "drive:test/test_mp4.mp4" "drive:test/copy" "-vv"]
2019/07/24 10:35:38 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2019/07/24 10:35:41 DEBUG : test_mp4.mp4: Couldn't find file - need to transfer
2019/07/24 10:35:45 INFO : test_mp4.mp4: Copied (server side copy)
2019/07/24 10:35:45 DEBUG : 4 go routines active
2019/07/24 10:35:45 DEBUG : rclone: Version "v1.47.0" finishing with parameters ["rclone" "copy" "drive:test/test_mp4.mp4" "drive:test/copy" "-vv"]

Anyone else encountered this problem?
Do you know how to solve this?
Is there a way to change the mime-type of the file with Rclone?

KinGuy · July 24, 2019, 7:50am

A screenshot after the copy:

ncw · July 24, 2019, 10:16am

Can you try this with the latest beta - I think this is fixed now.

KinGuy · July 24, 2019, 8:15pm

Tested the latest beta - it worked!

I did 3 different tests:

Copied the files using the old version (V1.47.0) - It created "Binary" files.
Copied the files using the latest beta (v1.48.0-074-g8e8b78d7-beta) - It created good files and kept the original mime-type.
Copied the "Binary" files from test 1 using the latest beta - It kept their mime-type as "Binary"...

Is there a way back from all these "Binary" files I have (I have thousands of them now)?
Do you know of a way to let Google Drive (or rclone) "guess" the mime-type from the file extension?

ncw · July 24, 2019, 10:23pm

Great!

Hmm, there is an API to set the mime type - you can use the patch API

You could use rclone lsf or rclone lsjson to find the IDs of the files then write a simple curl script to change them.

KinGuy · July 25, 2019, 8:58am

I tried to use the "Try this API" and it returns "200" message but the type is not changing.
After reading in depth, I found this:

So I guess I must create a new revision of the files, but I don't know how to do this without re uploading the files (it will take months...).
I'll update if I find a way to create a new revision and change it.

ncw · July 25, 2019, 9:26am

What you can do is do a server side copy and change the mime type while you are doing it. This is what rclone was doing and caused server side copies to go wrong for very large files...

So what you could do is put back the deleted line and put the mime type you want there

MimeType:     "video/mp4",

Then do a server side copy... I think that should fix it.

You can also do that with the drive API.

I'm not sure why the original fs.MimeTypeFromName(remote) didn't give the correct mime type though - maybe your computer doesn't have a complete mime types file or something like that.

What do these command say on your computer

$ xdg-mime query filetype image.jpg
image/jpeg
$ file -b --mime-type image.jpg
image/jpeg

KinGuy · July 25, 2019, 9:55am

its:

$ file -b --mime-type "...GoPro-101-2.mp4"
video/mp4

Plus, it didn't happen while uploading to Drive, only on server side copy. So I don't think the problem is in my computer. I think it got something to do with the implementation of the REST API.

I'm still playing with the file copy API and found that I have to manually declare the mimetype as "video/mp4" in order to change the copied file to the correct type. If I leave it blank it won't work.
I still couldn't find a way to make a new revision without copying the file...

ncw · July 25, 2019, 7:20pm

The bug, now fixed, was to do with rclone supplying a mime type in a server side copy when it shouldn't have. I conjecture that on your machine it was detecting the mime types wrong for some reason (this uses an OS database).

Yes, that is kind of what I expected...

KinGuy · July 27, 2019, 8:53pm

Hi,
To solve the problem I made a some python functions to find the files and copy them with a corrected mime-type. You can see it here:
https://github.com/KinGuy/driveMimeRepair

I hope it could help for other people with the same problem...

system · July 30, 2019, 8:53pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.