Google Drive server side copy: Can't copy a google document

#1

Hi,

I am trying to copy the content of Folder 1 into Folder 2 on the very same Google Drive Account. An error message tells me that a copy of a Google Document can’t be created.

$ ./rclone.exe -V
rclone v1.36

Here is my log:

$ ./rclone.exe copy -v "gdrive:z tmp/test folder 1" "gdrive:z tmp/test folder 2"
2017/05/22 15:32:01 INFO  : Google drive root 'z tmp/test folder 2': Modify window is 1ms
2017/05/22 15:32:02 ERROR : test sheet.xlsx: Failed to copy: can't copy a Google document
2017/05/22 15:32:02 ERROR : test slide.pptx: Failed to copy: can't copy a Google document
2017/05/22 15:32:02 INFO  : Google drive root 'z tmp/test folder 2': Waiting for checks to finish
2017/05/22 15:32:02 INFO  : Google drive root 'z tmp/test folder 2': Waiting for transfers to finish
2017/05/22 15:32:02 ERROR : Attempt 1/3 failed with 2 errors and: can't copy a Google document
2017/05/22 15:32:05 ERROR : test sheet.xlsx: Failed to copy: can't copy a Google document
2017/05/22 15:32:05 ERROR : test slide.pptx: Failed to copy: can't copy a Google document
2017/05/22 15:32:05 INFO  : Google drive root 'z tmp/test folder 2': Waiting for checks to finish
2017/05/22 15:32:05 INFO  : Google drive root 'z tmp/test folder 2': Waiting for transfers to finish
2017/05/22 15:32:05 ERROR : Attempt 2/3 failed with 2 errors and: can't copy a Google document
2017/05/22 15:32:06 ERROR : test sheet.xlsx: Failed to copy: can't copy a Google document
2017/05/22 15:32:06 ERROR : test slide.pptx: Failed to copy: can't copy a Google document
2017/05/22 15:32:06 INFO  : Google drive root 'z tmp/test folder 2': Waiting for checks to finish
2017/05/22 15:32:06 INFO  : Google drive root 'z tmp/test folder 2': Waiting for transfers to finish
2017/05/22 15:32:06 ERROR : Attempt 3/3 failed with 2 errors and: can't copy a Google document
2017/05/22 15:32:06 Failed to copy: can't copy a Google document

Setting drive formats does not help
--drive-formats ods,odt,odp

Any help with this is very appreciated. Thanks!

#2

Can I add any information needed or did I miss any habits in my question?

#3

Try to disable the document conversion feature in gdrive webui.

Settings -> General (I suppose. I have only the german ui available)

Screen Shot 2017-06-02 at 11.44.47.png964×199 18.7 KB

#4

Thank you for your efforts. That item was deactivated already.

Just to try, I activated conversion and got the same result.

#5

I wonder if it is trying to do a server side copy of the doc. I wonder what would happen if you duplicated that remote to appear as a different remote so that your can call it like this:

Rclone copy grive: gdriveclone: -v

Google docs can only be exported. I would have thought it would have tried to implicately export it but I guess not.

#6

I duplicated the very same remote and it works:
rclone.exe copy -v "gdrive:z tmp/test folder 1" "gdrive2:z tmp/test folder 2"

I get the exported files now, great. Thanks, calisro!

#7

great. You may want to open an issue for this on git. In my opinion it should do this automatically when copying to the same remote.

#8

I am experiencing this problem as well.

Was a git issue opened up for this?

I think that this type of operation shouldn’t just be an exported copy, but an actual copy.
I believe Google Drive does support copy operations for Google Docs (at least I certainly can do it through the web UI). There are also there APIs available:
https://developers.google.com/drive/api/v3/reference/files/copy
I don’t see any language in these operations that Google Docs files are not supported.

Might it be possible to permit actual copies of Google Docs files if we’re doing server-side copies?

#9

I don’t remember a specific issue for copying documents, but there are currently multiple open issues regarding the document handling. The latest one is #2483, which has the same issue.

I’m working on rewriting the drive backend in #2479 to allow copying and other features.

#10

This should be fixed by @B4dM4n’s pull request here: https://github.com/ncw/rclone/pull/2479

(I think).

I’m not going to merge that until 1.43 is out the door though.

#11

Very cool, those commits do look like they might handle the issue.

I would happily test this all out as soon as a beta build is out.

#12

There is a beta you can try:
https://beta.rclone.org/branch/v1.42-162-g6a66967c-drive-documents/

Keep in mind this is not tested thoroughly yet and therefore be considered experimental.

#13

I’m ok with experimental, happy to debug.

I suppose the one question I’d ask is: do you have any reason to suspect it could malfunction and delete any of the source space it’s trying to copy?

#14

This is is big change how the drive backend handles document files.

I’m not expecting any corrupted uploads or downloads, since this part is unchanged, but there could be some wrongly named files due to the new file extension handling. This could lead to unintentionally deleted or overwritten files during a transfer.

Otherwise I have no reason to assume other major problems.

#15

Understood, thanks.

Just to crosscheck, when you caution about a risk of deleted/overwritten files, is there any chance that could happen to the source of the copy/rsync?

I’m not too worried if the destination has trouble, but if the source could break, then that’s a bit more concerning and I’ll have to setup a much tighter sandbox for testing.

#16

The source is used read only during copy or sync, so there is no danger of deleting files there.

#17

Lovely, I’ll take a crack at this beta in the coming week.

#18

@B4dM4n First of all thank you for all the work put into Rclone.
But did this ever get successfully resolved? I am using version 1.43.1-2 and it still does not seem to be copy gdocs on server side.

I am not familiar with the source code however using this api
https://developers.google.com/drive/api/v3/reference/files/copy#try-it

It can make copies of gdocs based on the ID. Which means it must be possible. Any updates or thoughts you have would be greatly appreciated.

#19

Thank you for pushing this through. My initial tests with this look great, and Google documents are being copied server side. This is great, thank you.

I have confirmed this with Google Docs, Sheets, Slides, and Drawings.
Some Google document types don’t appear to be supported.
The following types seem to be ignored: Forms, My Maps, Sites, Apps Scripts.
Data Studio files are copied over, but they are broken (no longer are detected or open up in Data Studio).
Colab notebooks appear to copy over just fine and open in Colab, but the yellow colab icon is replaced with the generic blue file icon (not a big deal, but it makes Colab files less easily discernible visually). There are some types of Google documents that I haven’t tested yet (Fusion Tables & Jamboard).

Finally, I don’t know if this is expected behavior, but it seems that “rclone copy” and “rclone sync” are unable to determine that a Google document is identical in the destination. If I “rclone copy drive:src drive:dst” twice, any Google documents will end up in the ‘dst’ directory twice (unwanted behavior), while normal files will not be copied over if they already exist (expected behavior). I don’t know if there is an easy way to fix this.

#20

Rclone will ignore any types of files it can’t export.

In theory you could server side copy these just fine (which is what you are doing). but the rest of rclone can’t do anything with them (eg download them) which is why they aren’t supported.

If you could find out why, that would be very interesting

Can you please make a new issue on github about that and we can look at fixing.