Unable to move Native Google Docs of the same name

[larsen161]

I'm curious, why is rclone not able to move files of the same name when these are native drive files. Google Drive supports files of the same name to exist even when they are non-google formats.

From what I can tell, rclone is using file names instead of Drive File IDs which are unique to every file in drive. If if ls this folder for example I'm seeing files named File gdoc in rclone-testing.docx when this is not a docx fie but native Google Doc format. Every file has a unique ID.

Then when I run something like the following, the 3 Doc files are reduced down into just 1 instance of them and the other two are gone.

rclone move larsen161:/ larsen161-shareddrive:/ --create-empty-src-dirs --drive-server-side-across-configs

[Animosity022]

What is your rclone version (output from rclone version)

Which OS you are using and how many bits (eg Windows 7, 64 bit)

The rclone config contents with secrets removed.

Paste config here

A log from the command with the -vv flag

Paste  log here

[larsen161]

steve@gam:~:[demo.cloudbakers.com]$ rclone move demo-cloudbakers-steve:/ demo-cloudbakers-steve-shareddrive:/ --drive-server-side-across-configs -vv
2020/08/28 10:45:21 DEBUG : rclone: Version "v1.52.3" starting with parameters ["rclone" "move" "demo-cloudbakers-steve:/" "demo-cloudbakers-steve-shareddrive:/" "--drive-server-side-across-configs" "-vv"]
2020/08/28 10:45:21 DEBUG : Using config file from "/home/steve/.config/rclone/rclone.conf"
2020/08/28 10:45:21 DEBUG : fs cache: renaming cache item "demo-cloudbakers-steve:/" to be canonical "demo-cloudbakers-steve:"
2020/08/28 10:45:21 DEBUG : fs cache: renaming cache item "demo-cloudbakers-steve-shareddrive:/" to be canonical "demo-cloudbakers-steve-shareddrive:"
2020/08/28 10:45:21 DEBUG : demo-cloudbakers-steve-shareddrive: Loaded invalid token from config file - ignoring
2020/08/28 10:45:21 DEBUG : demo-cloudbakers-steve-shareddrive: Saved new token in config file
2020/08/28 10:45:22 DEBUG : Google drive root '': read info from team drive "Destination-Domain-to-Domain"
2020/08/28 10:45:22 NOTICE: File gdoc in rclone-testing.docx: Duplicate object found in source - ignoring
2020/08/28 10:45:22 NOTICE: File gdoc in rclone-testing.docx: Duplicate object found in source - ignoring
2020/08/28 10:45:22 NOTICE: Screen Shot 2020-08-26 at 11.19.13 PM.png: Duplicate object found in source - ignoring
2020/08/28 10:45:22 DEBUG : Google drive root '': Waiting for checks to finish
2020/08/28 10:45:22 DEBUG : Google drive root '': Waiting for transfers to finish
2020/08/28 10:45:23 INFO  : Screen Shot 2020-08-26 at 11.19.13 PM.png: Moved (server side)
2020/08/28 10:45:23 INFO  : File gdoc in rclone-testing.docx: Moved (server side)
2020/08/28 10:45:24 INFO  : File gpresentation in rclone-testing.pptx: Moved (server side)
2020/08/28 10:45:25 INFO  : File gsheet in rclone-testing.xlsx: Moved (server side)
2020/08/28 10:45:25 INFO  : There was nothing to transfer
2020/08/28 10:45:25 INFO  : 
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 4 / 4, 100%
Renamed:                4
Elapsed time:         2.8s

2020/08/28 10:45:25 DEBUG : 15 go routines active

This seems like a fundamental flaw with rclone that it sees these files as docx files and having the same name which causes issues in a classic server with a file system but this is the Google Drive, names are just metadata and the true file names are in fact unique

File gdoc in rclone-testing
1ciL89w_aL0-h24RhdkWNRFKUsm5rWekq655GiiXSpok
169D2JeCbTZi2I4rOJmy1DaHPOTqNl8pk_sancOivw-I
1Gv2ycuFg1f7EHtXK9M5z7oYsEp7CGVRha3sNKJ2Q6BU

Screen Shot 2020-08-26 at 11.19.13 PM.png
1LjJzikDcticSjiT001kWaxRBk2Y2qr6N
1bup7M4t_g8bwi_X_hxd9yRAXTmv6gkiW

[larsen161]

Here is the equivalent gam command that moves these files that are named the same into the destination shared drive as expected.

steve@gam:~:[demo.cloudbakers.com]$ gam config debug_level 1 user steve update drivefile name "File gdoc in rclone-testing" teamdriveparentid 0AEHUxvVJEu7cUk9PVA
connect: (www.googleapis.com, 443)
send: b'GET /discovery/v1/apis/admin/directory_v1/rest HTTP/1.1\r\nHost: www.googleapis.com\r\nuser-agent: GAMADV-XTD3 5.08.17 - https://github.com/taers232c/GAMADV-XTD3 / Ross Scroggs <ross.scroggs@gmail.com> / Python 3.8.5 final / Linux-5.3.0-1032-gcp-x86_64-with-glibc2.4 x86_64 /\r\nauthorization: Bearer ya29.a0AfHMDZ2yo\r\naccept-encoding: gzip, deflate\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json; charset=UTF-8
header: Vary: Origin
header: Vary: X-Origin
header: Vary: Referer
header: Date: Fri, 28 Aug 2020 11:00:23 GMT
header: Server: ESF
header: Content-Length: 243589
header: X-XSS-Protection: 0
header: X-Frame-Options: SAMEORIGIN
header: X-Content-Type-Options: nosniff
connect: (www.googleapis.com, 443)
send: b'GET /discovery/v1/apis/drive/v3/rest HTTP/1.1\r\nHost: www.googleapis.com\r\nuser-agent: Python-httplib2/0.18.1 (gzip)\r\naccept-encoding: gzip, deflate\r\nif-none-match: "-2NioU2H8y8siEzrBOV_qzRI6kQ/iw5KltrEyyVmddHJpTcj8RnsLIs"\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Expires: Fri, 28 Aug 2020 11:05:23 GMT
header: Date: Fri, 28 Aug 2020 11:00:23 GMT
header: Cache-Control: public, max-age=300, must-revalidate, no-transform
header: ETag: "-2NioU2H8y8siEzrBOV_qzRI6kQ/l7m-JIhSvmu0HFIAKZgPpq4u-fg"
header: Vary: Origin
header: Vary: X-Origin
header: Content-Type: application/json; charset=UTF-8
header: Content-Encoding: gzip
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: Content-Security-Policy: frame-ancestors 'self'
header: X-XSS-Protection: 1; mode=block
header: Server: GSE
header: Transfer-Encoding: chunked
connect: (oauth2.googleapis.com, 443)
send: b'POST /token HTTP/1.1\r\nHost: oauth2.googleapis.com\r\nContent-Length: 865\r\ncontent-type: application/x-www-form-urlencoded\r\nuser-agent: GAMADV-XTD3 5.08.17 - https://github.com/taers232c/GAMADV-XTD3 / Ross Scroggs <ross.scroggs@gmail.com> / Python 3.8.5 final / Linux-5.3.0-1032-gcp-x86_64-with-glibc2.4 x86_64 /\r\naccept-encoding: gzip, deflate\r\n\r\n'
send: b'assertion=eyJ0eXqGV9TuHS6AnrRBiHIgLq8IavdKPl-uso_ixsTAfGIVYy_DKAkPUorN4f5WgrSbJmAf7xyr7H37oEwTK8NWk80-TXukKyWdqvXnECz7qx3Sh-Q&grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: application/json; charset=utf-8
header: Vary: Origin
header: Vary: X-Origin
header: Vary: Referer
header: Content-Encoding: gzip
header: Date: Fri, 28 Aug 2020 11:00:23 GMT
header: Server: scaffolding on HTTPServer2
header: Cache-Control: private
header: X-XSS-Protection: 0
header: X-Frame-Options: SAMEORIGIN
header: X-Content-Type-Options: nosniff
header: Transfer-Encoding: chunked
Getting all Drive Files/Folders that match query ('me' in owners and name = 'File gdoc in rclone-testing') for steve@demo.cloudbakers.com
send: b'GET /drive/v3/files?q=%27me%27+in+owners+and+name+%3D+%27File+gdoc+in+rclone-testing%27&fields=nextPageToken%2Cfiles%28id%2CdriveId%29&pageSize=1000&prettyPrint=true&alt=json HTTP/1.1\r\nHost: www.googleapis.com\r\naccept: application/json\r\naccept-encoding: gzip, deflate\r\nuser-agent: GAMADV-XTD3 5.08.17 - https://github.com/taers232c/GAMADV-XTD3 / Ross Scroggs <ross.scroggs@gmail.com> / Python 3.8.5 final / Linux-5.3.0-1032-gcp-x86_64-with-glibc2.4 x86_64 / (gzip)\r\nx-goog-api-client: gdcl/1.9.3 gl-python/3.8.5\r\ncontent-length: 0\r\nauthorization: Bearer ya29.a0AfjX7j8Cqsf_6\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Expires: Fri, 28 Aug 2020 11:00:24 GMT
header: Date: Fri, 28 Aug 2020 11:00:24 GMT
header: Cache-Control: private, max-age=0, must-revalidate, no-transform
header: Vary: Origin
header: Vary: X-Origin
header: Content-Type: application/json; charset=UTF-8
header: Content-Encoding: gzip
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: Content-Security-Policy: frame-ancestors 'self'
header: X-XSS-Protection: 1; mode=block
header: Server: GSE
header: Transfer-Encoding: chunked
Got 2 Drive Files/Folders that matched query ('me' in owners and name = 'File gdoc in rclone-testing') for steve@demo.cloudbakers.com...
User: steve@demo.cloudbakers.com, Update 2 Drive Files/Folders
send: b'GET /drive/v3/files/0AEHUxvVJEu7cUk9PVA?fields=id%2CmimeType%2CdriveId&supportsAllDrives=true&prettyPrint=true&alt=json HTTP/1.1\r\nHost: www.googleapis.com\r\naccept: application/json\r\naccept-encoding: gzip, deflate\r\nuser-agent: GAMADV-XTD3 5.08.17 - https://github.com/taers232c/GAMADV-XTD3 / Ross Scroggs <ross.scroggs@gmail.com> / Python 3.8.5 final / Linux-5.3.0-1032-gcp-x86_64-with-glibc2.4 x86_64 / (gzip)\r\nx-goog-api-client: gdcl/1.9.3 gl-python/3.8.5\r\ncontent-length: 0\r\nauthorization: Bearer ya29.a0AfgjX7j8Cqsf_6\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Expires: Fri, 28 Aug 2020 11:00:24 GMT
header: Date: Fri, 28 Aug 2020 11:00:24 GMT
header: Cache-Control: private, max-age=0, must-revalidate, no-transform
header: Vary: Origin
header: Vary: X-Origin
header: Content-Type: application/json; charset=UTF-8
header: Content-Encoding: gzip
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: Content-Security-Policy: frame-ancestors 'self'
header: X-XSS-Protection: 1; mode=block
header: Server: GSE
header: Transfer-Encoding: chunked
send: b'GET /drive/v3/files/1Gv2ycuFg1f7EHtXK9M5z7oYsEp7CGVRha3sNKJ2Q6BU?fields=parents&supportsAllDrives=true&prettyPrint=true&alt=json HTTP/1.1\r\nHost: www.googleapis.com\r\naccept: application/json\r\naccept-encoding: gzip, deflate\r\nuser-agent: GAMADV-XTD3 5.08.17 - https://github.com/taers232c/GAMADV-XTD3 / Ross Scroggs <ross.scroggs@gmail.com> / Python 3.8.5 final / Linux-5.3.0-1032-gcp-x86_64-with-glibc2.4 x86_64 / (gzip)\r\nx-goog-api-client: gdcl/1.9.3 gl-python/3.8.5\r\ncontent-length: 0\r\nauthorization: Bearer ya29.a0AfgjX7j8Cqsf_6\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Expires: Fri, 28 Aug 2020 11:00:24 GMT
header: Date: Fri, 28 Aug 2020 11:00:24 GMT
header: Cache-Control: private, max-age=0, must-revalidate, no-transform
header: Vary: Origin
header: Vary: X-Origin
header: Content-Type: application/json; charset=UTF-8
header: Content-Encoding: gzip
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: Content-Security-Policy: frame-ancestors 'self'
header: X-XSS-Protection: 1; mode=block
header: Server: GSE
header: Transfer-Encoding: chunked
send: b'PATCH /drive/v3/files/1Gv2ycuFg1f7EHtXK9M5z7oYsEp7CGVRha3sNKJ2Q6BU?enforceSingleParent=false&keepRevisionForever=false&useContentAsIndexableText=false&addParents=0AEHUxvVJEu7cUk9PVA&removeParents=1sjKQZnZjl8rP-bjCV0H69iLxq6j1pAF8&fields=id%2Cname%2CmimeType&supportsAllDrives=true&prettyPrint=true&alt=json HTTP/1.1\r\nHost: www.googleapis.com\r\naccept: application/json\r\naccept-encoding: gzip, deflate\r\nuser-agent: GAMADV-XTD3 5.08.17 - https://github.com/taers232c/GAMADV-XTD3 / Ross Scroggs <ross.scroggs@gmail.com> / Python 3.8.5 final / Linux-5.3.0-1032-gcp-x86_64-with-glibc2.4 x86_64 / (gzip)\r\nx-goog-api-client: gdcl/1.9.3 gl-python/3.8.5\r\ncontent-type: application/json\r\ncontent-length: 2\r\nauthorization: Bearer ya29.a0AfjX7j8Cqsf_6\r\n\r\n'
send: b'{}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
header: Pragma: no-cache
header: Expires: Mon, 01 Jan 1990 00:00:00 GMT
header: Date: Fri, 28 Aug 2020 11:00:25 GMT
header: Vary: Origin
header: Vary: X-Origin
header: Content-Type: application/json; charset=UTF-8
header: Content-Encoding: gzip
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: Content-Security-Policy: frame-ancestors 'self'
header: X-XSS-Protection: 1; mode=block
header: Server: GSE
header: Transfer-Encoding: chunked
  User: steve@demo.cloudbakers.com, Drive File: File gdoc in rclone-testing, Updated (1/2)
send: b'GET /drive/v3/files/169D2JeCbTZi2I4rOJmy1DaHPOTqNl8pk_sancOivw-I?fields=parents&supportsAllDrives=true&prettyPrint=true&alt=json HTTP/1.1\r\nHost: www.googleapis.com\r\naccept: application/json\r\naccept-encoding: gzip, deflate\r\nuser-agent: GAMADV-XTD3 5.08.17 - https://github.com/taers232c/GAMADV-XTD3 / Ross Scroggs <ross.scroggs@gmail.com> / Python 3.8.5 final / Linux-5.3.0-1032-gcp-x86_64-with-glibc2.4 x86_64 / (gzip)\r\nx-goog-api-client: gdcl/1.9.3 gl-python/3.8.5\r\ncontent-length: 0\r\nauthorization: Bearer ya29.a0AfgjX7j8Cqsf_6\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Expires: Fri, 28 Aug 2020 11:00:25 GMT
header: Date: Fri, 28 Aug 2020 11:00:25 GMT
header: Cache-Control: private, max-age=0, must-revalidate, no-transform
header: Vary: Origin
header: Vary: X-Origin
header: Content-Type: application/json; charset=UTF-8
header: Content-Encoding: gzip
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: Content-Security-Policy: frame-ancestors 'self'
header: X-XSS-Protection: 1; mode=block
header: Server: GSE
header: Transfer-Encoding: chunked
send: b'PATCH /drive/v3/files/169D2JeCbTZi2I4rOJmy1DaHPOTqNl8pk_sancOivw-I?enforceSingleParent=false&keepRevisionForever=false&useContentAsIndexableText=false&addParents=0AEHUxvVJEu7cUk9PVA&removeParents=1sjKQZnZjl8rP-bjCV0H69iLxq6j1pAF8&fields=id%2Cname%2CmimeType&supportsAllDrives=true&prettyPrint=true&alt=json HTTP/1.1\r\nHost: www.googleapis.com\r\naccept: application/json\r\naccept-encoding: gzip, deflate\r\nuser-agent: GAMADV-XTD3 5.08.17 - https://github.com/taers232c/GAMADV-XTD3 / Ross Scroggs <ross.scroggs@gmail.com> / Python 3.8.5 final / Linux-5.3.0-1032-gcp-x86_64-with-glibc2.4 x86_64 / (gzip)\r\nx-goog-api-client: gdcl/1.9.3 gl-python/3.8.5\r\ncontent-type: application/json\r\ncontent-length: 2\r\nauthorization: Bearer ya29.a0AfgjX7j8Cqsf_6\r\n\r\n'
send: b'{}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
header: Pragma: no-cache
header: Expires: Mon, 01 Jan 1990 00:00:00 GMT
header: Date: Fri, 28 Aug 2020 11:00:26 GMT
header: Vary: Origin
header: Vary: X-Origin
header: Content-Type: application/json; charset=UTF-8
header: Content-Encoding: gzip
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: Content-Security-Policy: frame-ancestors 'self'
header: X-XSS-Protection: 1; mode=block
header: Server: GSE
header: Transfer-Encoding: chunked
  User: steve@demo.cloudbakers.com, Drive File: File gdoc in rclone-testing, Updated (2/2)

[Animosity022]

While Google Drive allows for duplicate file names, rclone does not.

https://rclone.org/drive/#duplicated-files

You'd need to run dedupe to clean up duplicate files so you can copy.

Rclone allows for mounts and various things that duplicate file names would break, which would be a fundamental flaw in most OS's that use mounts

[larsen161]

Sounds like a feature request then to better support Drive to Drive migrations.

[Animosity022]

It's not quite that simple, you can check out:

rclone sync complains of duplicate objects in destination, ignores and fails to remove them

Albeit that's slightly different, feel free to open a feature request for an enhancement for Drive as it's really an edge case since mounts would not function with duplicate files.

[larsen161]

Hmm, edge case for me it is not. It's actually core functionality when you look at G Suite to G Suite migrations which we do 100s of a year. Mounting is taboo when using Drive.
You do not want to mount files from Drive unless you use Backup and Sync and/or Drive Files Stream clients and these handle objects of the same name in the same folder when on a file system without any impact.

[Animosity022]

You’d be surprised as the primary use case for 95% of the posts here are mounting.

[larsen161]

Mounting Google Drive or the other storage systems? I have a feeling there's a big difference if you look specifically how Drive is used with rclone.

[Animosity022]

By the amount of posts I answer on a daily basis, the primary use case is mounting a Google Drive with rclone. That's my data from the forums and github issues I see mostly.

There are quite a number of other cloud mounts that are used by the clear winner in use is Google.