Rclone mount for Google Drive does not show as web links the google documents of the "Google My Map" (.gmap) type

lxng · February 21, 2022, 12:56pm

Hi,
I was told to post it here rather than to submit a bug it the github repository:

What is the problem you are having with rclone?

When mounting a Google Drive via "rclone mount" with paramenter ' --drive-export-formats "url" ' the online Google documents (like Google Sheets, Google Docs etc.) are supposed to be seen as files with the .url extension.
This is according to the official rclone documentation.
This is true for all of the Google Documents types except "Google My Maps" file. The "Google My Maps" files are not seen on the mounted drive at all.
I assume, they are somehow missing from the list of known Google Documents file types, whereas they don't have any particular difference and they can (and should) be represented as *.url files as well.

The following line in the log below shows what I mean:
2022/02/21 15:25:48 DEBUG : My Map 1: Ignoring unknown document type "application/vnd.google-apps.map"

What is your rclone version (output from `rclone version`)

rclone v1.58.0-beta.5999.f22b703a5

os/version: Microsoft Windows 11 Home Single Language 21H2 (64 bit)
os/kernel: 10.0.22000.527 (x86_64)
os/type: windows
os/arch: amd64
go/version: go1.17.6
go/linking: dynamic
go/tags: cmount

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

os/version: Microsoft Windows 11 Home Single Language 21H2 (64 bit)

Which cloud storage system are you using? (e.g. Google Drive)

Google Drive

The command you were trying to run (e.g. `rclone copy /tmp remote:tmp`)

rclone mount GDrive-Lxng: D:\Lxng.local\Clouds\GDrive-Lxng -vv --log-file rclone-mount.log --drive-export-formats "url" --dir-cache-time 1m0s --poll-interval 1m0s --config C:\Users\Lxng\rclone-mount\rclone.conf

A log from the command with the `-vv` flag (e.g. output from `rclone -vv copy /tmp remote:tmp`)

The service rclone has been started.
2022/02/21 15:25:41 DEBUG : /: Statfs:
2022/02/21 15:25:41 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4456448 Bfree:1098452 Bavail:1098452 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2022/02/21 15:25:41 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/02/21 15:25:41 DEBUG : /: >Getattr: errc=0
2022/02/21 15:25:41 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/02/21 15:25:41 DEBUG : /: >Getattr: errc=0
2022/02/21 15:25:41 DEBUG : /: Opendir:
2022/02/21 15:25:41 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/02/21 15:25:41 DEBUG : /: >OpenFile: fd=/ (r), err=
2022/02/21 15:25:41 DEBUG : /: >Opendir: errc=0, fh=0x0
2022/02/21 15:25:41 DEBUG : /: Releasedir: fh=0x0
2022/02/21 15:25:41 DEBUG : /: >Releasedir: errc=0
2022/02/21 15:25:47 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/02/21 15:25:47 DEBUG : /: >Getattr: errc=0
2022/02/21 15:25:47 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/02/21 15:25:47 DEBUG : /: >Getattr: errc=0
2022/02/21 15:25:47 DEBUG : /: Opendir:
2022/02/21 15:25:47 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/02/21 15:25:47 DEBUG : /: >OpenFile: fd=/ (r), err=
2022/02/21 15:25:47 DEBUG : /: >Opendir: errc=0, fh=0x0
2022/02/21 15:25:47 DEBUG : /: Readdir: ofst=0, fh=0x0
2022/02/21 15:25:48 DEBUG : My Map 1: Ignoring unknown document type "application/vnd.google-apps.map"
2022/02/21 15:25:48 DEBUG : /: >Readdir: items=40, errc=0
2022/02/21 15:25:48 DEBUG : /: Releasedir: fh=0x0
2022/02/21 15:25:48 DEBUG : /: >Releasedir: errc=0
2022/02/21 15:25:48 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/02/21 15:25:48 DEBUG : /: >Getattr: errc=0
2022/02/21 15:25:48 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/02/21 15:25:48 DEBUG : /: >Getattr: errc=0
2022/02/21 15:25:48 DEBUG : /: Opendir:
2022/02/21 15:25:48 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/02/21 15:25:48 DEBUG : /: >OpenFile: fd=/ (r), err=
2022/02/21 15:25:48 DEBUG : /: >Opendir: errc=0, fh=0x0
2022/02/21 15:25:48 DEBUG : /: Statfs:
2022/02/21 15:25:48 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4456448 Bfree:1098452 Bavail:1098452 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2022/02/21 15:25:48 DEBUG : /: Releasedir: fh=0x0
2022/02/21 15:25:48 DEBUG : /: >Releasedir: errc=0
2022/02/21 15:25:48 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/02/21 15:25:48 DEBUG : /: >Getattr: errc=0
2022/02/21 15:25:48 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/02/21 15:25:48 DEBUG : /: >Getattr: errc=0
2022/02/21 15:25:48 DEBUG : /: Opendir:
2022/02/21 15:25:48 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/02/21 15:25:48 DEBUG : /: >OpenFile: fd=/ (r), err=
2022/02/21 15:25:48 DEBUG : /: >Opendir: errc=0, fh=0x0
2022/02/21 15:25:48 DEBUG : /: Releasedir: fh=0x0
2022/02/21 15:25:48 DEBUG : /: >Releasedir: errc=0
2022/02/21 15:25:48 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/02/21 15:25:48 DEBUG : /: >Getattr: errc=0
2022/02/21 15:25:48 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/02/21 15:25:48 DEBUG : /: >Getattr: errc=0
2022/02/21 15:25:48 DEBUG : /: Opendir:
2022/02/21 15:25:48 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/02/21 15:25:48 DEBUG : /: >OpenFile: fd=/ (r), err=
2022/02/21 15:25:48 DEBUG : /: >Opendir: errc=0, fh=0x0
2022/02/21 15:25:48 DEBUG : /: Readdir: ofst=0, fh=0x0
2022/02/21 15:25:48 DEBUG : /: >Readdir: items=40, errc=0
2022/02/21 15:25:48 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/02/21 15:25:48 DEBUG : /: >Getattr: errc=0
2022/02/21 15:25:48 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/02/21 15:25:48 DEBUG : /: >Getattr: errc=0
2022/02/21 15:25:48 DEBUG : /: Opendir:
2022/02/21 15:25:48 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/02/21 15:25:48 DEBUG : /: >OpenFile: fd=/ (r), err=
2022/02/21 15:25:48 DEBUG : /: >Opendir: errc=0, fh=0x1
2022/02/21 15:25:48 DEBUG : /: Releasedir: fh=0x1
2022/02/21 15:25:48 DEBUG : /: >Releasedir: errc=0
2022/02/21 15:25:48 DEBUG : /: Releasedir: fh=0x0
2022/02/21 15:25:48 DEBUG : /: >Releasedir: errc=0
2022/02/21 15:25:48 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/02/21 15:25:48 DEBUG : /: >Getattr: errc=0
2022/02/21 15:25:48 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/02/21 15:25:48 DEBUG : /: >Getattr: errc=0
2022/02/21 15:25:48 DEBUG : /: Opendir:
2022/02/21 15:25:48 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/02/21 15:25:48 DEBUG : /: >OpenFile: fd=/ (r), err=
2022/02/21 15:25:48 DEBUG : /: >Opendir: errc=0, fh=0x0
2022/02/21 15:25:48 DEBUG : /Paanajarvi.url: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/02/21 15:25:48 DEBUG : /Paanajarvi.url: >Getattr: errc=0
2022/02/21 15:25:48 DEBUG : /Paanajarvi.url: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/02/21 15:25:48 DEBUG : /Paanajarvi.url: >Getattr: errc=0
2022/02/21 15:25:48 DEBUG : /Paanajarvi.url: OpenEx: flags=0x0
2022/02/21 15:25:48 DEBUG : /Paanajarvi.url: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/02/21 15:25:48 DEBUG : Paanajarvi.url: Open: flags=O_RDONLY
2022/02/21 15:25:48 DEBUG : Paanajarvi.url: >Open: fd=Paanajarvi.url (r), err=
2022/02/21 15:25:48 DEBUG : /Paanajarvi.url: >OpenFile: fd=Paanajarvi.url (r), err=
2022/02/21 15:25:48 DEBUG : /Paanajarvi.url: >OpenEx: errc=0, fh=0x1
2022/02/21 15:25:48 DEBUG : /Paanajarvi.url: Flush: fh=0x1
2022/02/21 15:25:48 DEBUG : /Paanajarvi.url: >Flush: errc=0
2022/02/21 15:25:48 DEBUG : /Paanajarvi.url: Release: fh=0x1
2022/02/21 15:25:48 DEBUG : /Paanajarvi.url: >Release: errc=0
The service rclone has been stopped.
2022/02/21 15:27:06 DEBUG : Google drive root '': Destroy:
2022/02/21 15:27:06 DEBUG : Google drive root '': >Destroy:
2022/02/21 15:27:06 DEBUG : Not calling host.Unmount as mount already Destroyed
2022/02/21 15:27:06 DEBUG : Unmounted successfully

asdffdsa · February 21, 2022, 1:47pm

hello and welcome to the forum,

this is not a rclone bug.
google drive api does not allow maps to be exported.

has been discussed in the forum a few times
and if you search the internet you should find that same answer.

if you find that the gdrive api has changed, let us know.

lxng · February 21, 2022, 2:40pm

Hi,
I may not be understanding the deep essence of the problem.
However, I am not taking about exporting Google My Maps. I have a different (and actually the opposite) goal: not to export online documents from Google, but rather only get links to them, so that I can click on the link and open it in the web browser.
Exactly for this purpose rclone provides the argument ' --drive-export-formats "url" '. With this argument and with this value of the argument Goole Document files are NOT exported. Instead, only information about them is read and a virtual "link" file is created. The contents of this file is just a web link that leads to the Google document on the internet.
Based on what I can see in the log, the Google My Map files are found, and their type is properly detected.
And so I surmise that as they are found, their weblinks are also property detected. And if so, an ".url" (or "link.html", or "desktop") file could also be created.

As an example, this is a sample contents of the url-file, provided by rclone for a Google Docs document:

[InternetShortcut]
URL=https://docs.google.com/document/d/XXXXXXXXXXXXXXXXXXXXXXXXX/edit?usp=drivesdk

Is it possible to get the same for a My Maps file?

Animosity022 · February 21, 2022, 4:07pm

No, only the supported formats are here:

Google drive (rclone.org)

lxng · February 21, 2022, 5:53pm

Oh, how unfortunate.
The documentations states a seemingly different thing:
"Google documents can also be exported as link files. These files will open a browser window for the Google Docs website of that document when opened. The link file extension has to be specified as a --drive-export-formats parameter. They will match all available Google Documents."
I must have misenterpreded this statement as I thought they are "all Google documents available in the cloud drive". I assumed that if their contents is not exported or sent anywhere, nor even has it to be read, then no limitations would be applied.
Sorry to learn that this was not so

Animosity022 · February 21, 2022, 6:02pm

A Google Document != Google My Map.

Google Docs - Wikipedia

lxng · February 21, 2022, 6:25pm

Yeah, that's true. Good point.
However, why not to show ALL unexportable files as web-links? Just wondering what the reason for the existing limitation is.

ncw · February 21, 2022, 7:58pm

Currently rclone only shows files it can export and skips ones it doesn't.

It would make sense for rclone to show everything in URL mode, you are correct.

I'm not sure how hard that change would be, probably not very if you wanted to have a go.

lxng · February 21, 2022, 8:05pm

It is great to know that the existing limitation is not something fundamental.
I doubt that my level of programming skills is enough for this task. But at least now I understand the state of things.

ncw · February 22, 2022, 5:42pm

I had a quick go at this - can you see if it works for you @lxng ? This should show links for all the documents regardless of whether they can be exported.

v1.58.0-beta.6005.fa1ace54b.fix-drive-export on branch fix-drive-export (uploaded in 15-30 mins)

lxng · February 22, 2022, 9:05pm

These are my observations:
With the argument --drive-export-formats "url" all online documents are shown as links (and can be opened as links). Including My Maps, which is excellent!
However, with the argument --drive-export-formats "docx" now all online documents are show as exported files, not only .docx, but also .xlsx. .svg etc.
In the previous version with the latter argument value, only .docx files were show this way, and all other files were hidden.
Also, in the previous version the value "docx,url" showed .docx files as .docx, and all other - as urls.
In the new version the value "docx,url" makes rclone show all online files as urls (including docx).

ncw · February 23, 2022, 7:53pm

Thanks for your testing. I can see I messed that up slightly!

Can you give this a go instead?

v1.58.0-beta.6005.0356bc0c5.fix-drive-export on branch fix-drive-export (uploaded in 15-30 mins)

lxng · February 24, 2022, 4:08pm

Hi Nick,
Now with this new build the behaviour is as follows:

--drive-export-formats "url" : all online files are displayed as urls (which is OK)
--drive-export-formats "docx,url" : Google files that can be exported as MS Word, are displayed as .docx, all other - as urls (also OK)
--drive-export-formats "docx,pptx,url" : Google files that match MS Word and PowerPoint documents are shown as .docx and .pptx respectively, and all other files are urls (OK)
--drive-export-formats "docx" : all Google documents are shown as exported files (docx, pptx, xlsx and svg)
--drive-export-formats "docx,pptx" : again, all Google files are exported
--drive-export-formats "" : all Google files are exported (seems OK)
no --drive-export-formats argument at all : all Google files are exported (seems OK)
Thus, once there is no "url" in the list of types in the --drive-export-formats argument, this argument does not matter anymore and all files are exported. This is probably not how it is supposed to work. In iterms 4 and 5 I would expect that only the exportable files of the specified formats should be visible.

ncw · February 25, 2022, 10:20am

I just checked this against rclone 1.57.0 and the patch didn't introduce any changes to rclone's handling of docs, other than exporting unexportable items when url is set.

What --drive-export-formats does is set the export format, but there is a default export format for each type, so you always see all exportable docs.

If you add url in there somewhere then it will export any types not mentioned previously as URLs including unexportable types.

Does that make sense?

Here are the diffs for the different options on a directory of google documents with rclone 1.57.0 and this patch

$ export FORMATS=url ; diff -u <(rclone-v1.57.0 lsf drive:GDocs --drive-export-formats "$FORMATS" | sort) <(rclone lsf drive:GDocs --drive-export-formats "$FORMATS" | sort)
--- /dev/fd/63	2022-02-25 10:14:21.245661418 +0000
+++ /dev/fd/62	2022-02-25 10:14:21.249661382 +0000
@@ -7,6 +7,7 @@
 Global Load Balancing.html
 .~lock.Docs.docx#
 .~lock.Sheets.xlsx#
+Maps.url
 Sheets2.url
 Sheets-copy10.url
 Sheets-copy11.url
$ export FORMATS=docx,url ; diff -u <(rclone-v1.57.0 lsf drive:GDocs --drive-export-formats "$FORMATS" | sort) <(rclone lsf drive:GDocs --drive-export-formats "$FORMATS" | sort)
--- /dev/fd/63	2022-02-25 10:14:31.425572185 +0000
+++ /dev/fd/62	2022-02-25 10:14:31.425572185 +0000
@@ -7,6 +7,7 @@
 Global Load Balancing.html
 .~lock.Docs.docx#
 .~lock.Sheets.xlsx#
+Maps.url
 Sheets2.url
 Sheets-copy10.url
 Sheets-copy11.url
$ export FORMATS=docx,pptx,url ; diff -u <(rclone-v1.57.0 lsf drive:GDocs --drive-export-formats "$FORMATS" | sort) <(rclone lsf drive:GDocs --drive-export-formats "$FORMATS" | sort)
--- /dev/fd/63	2022-02-25 10:14:46.117442604 +0000
+++ /dev/fd/62	2022-02-25 10:14:46.121442568 +0000
@@ -7,6 +7,7 @@
 Global Load Balancing.html
 .~lock.Docs.docx#
 .~lock.Sheets.xlsx#
+Maps.url
 Sheets2.url
 Sheets-copy10.url
 Sheets-copy11.url
$ export FORMATS=docx ; diff -u <(rclone-v1.57.0 lsf drive:GDocs --drive-export-formats "$FORMATS" | sort) <(rclone lsf drive:GDocs --drive-export-formats "$FORMATS" | sort)
$ export FORMATS=docx,pptx ; diff -u <(rclone-v1.57.0 lsf drive:GDocs --drive-export-formats "$FORMATS" | sort) <(rclone lsf drive:GDocs --drive-export-formats "$FORMATS" | sort)

lxng · February 25, 2022, 10:55am

To me this behaviour (with the new logic to process urls) is ok. If it is in line with what has been before, then everything seems fine to me.

ncw · February 25, 2022, 4:08pm

Great!

Thank you for great testing

I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.58

Rclone mount for Google Drive does not show as web links the google documents of the "Google My Map" (.gmap) type

What is the problem you are having with rclone?

What is your rclone version (output from rclone version)