FYI Sep 30 - New Google Drive Shortcuts and Folder Structure

Just an FYI. More detail on the new folder/file structure.

2 Likes

@ncw this will create chaos, isn't ?

As far as my understanding: If we put same file in different directory, google will count them as one. Am I correct?

Currently you can have the same file in multiple directories (by using SHIFT-Z to create a copy of the file in the web app IIRC)

This means that the file has multiple parent directories.

This isn't a good abstraction for several reasons

  • the file only has one name so it has to have the same name in all directories
  • it makes removing files more difficult - if a file has multiple parents you want to remove one parent, not delete the file

Rclone doesn't deal well with these files anyway so it probably won't make much difference to rclone though there was stuff about sharing I wasn't sure of the consequence.

I think I'll need to do something to rclone so it understands the shortcuts though. Currently it will probably ignore them. Maybe it should copy the item pointed to...

2 Likes

shortcut vs soft link vs hard link, hmm

Re Shift-Z, yep! The first few times you delete what you thought was a 'link' in a Shift-Z folder and discover that it deleted the original files = :scream: :sob:

At the moment if I copy Shift-Z folder/files to any other remote (My Drive, Shared Drive, local drive, etc) rclone creates a full copy of the folders/files in the destination. Would be great if rclone could still do that by default. And would be fabulous, if possible, to allow shortcuts/softlinks on filesystems that support them (complicated, I imagine?).

2 Likes

Yes I think copying the destination is probably the correct default for softlinks.

I tried a shortcut and rclone does ignore them at the moment with a DEBUG message like this

2020/03/28 17:57:19 DEBUG : file2 shortcut.txt: Ignoring unknown document type "application/vnd.google-apps.shortcut"

So that is probably the first thing to fix. Would you mind putting this into a new issue on github for me so I don't forget? Put a link back here - thanks

2 Likes

Came here for this bizarre change. Make a copy doesn't save it to your Drive either; it simply creates another shortcut.

Any way to download such shared folders now using rclone, @ncw?

Open it in the web app to discover its ID then supply that to --drive-root-folder-id - that should do as a work-around!

3 Likes

To a new remote?

Just supply that flag to your existing remote and it should work.

Yes, more carnage, but I can see why they are doing it. The file in multiple folder model was amazingly powerful but caused so much trouble - say, for example a file is in two folders and I want to zip up one of those folders and delete that folder, leaving the other. Once over the other one remained... then it changed so the other instance(s) got silently deleted.... aargh.

It will be carnage, but has a fighting chance of working because at least the shortcut model is closer to the 'traditional' file system paradigm so there can be some sort of expectation what normal behaviour will be.

Maybe the next harmonisation/dumbing down will be to prevent having multiple files in the same directory/label with the same filename? I just see the continual, terrifying, fundamental changes to the nature of the Google Drive filesystem. It takes some dedication from the developers to keep rclone in step with all of these continually evolving cloud services - even if you were to believe every storage firm wanted to co-operate.

Another work around is the directory you've been shared should appear in --drive-shared-with-me, supply that directory name to --includes as foldername/** and do your copy normally. ex:
rclone --drive-shared-with-me --includes foldername/** copy remote1: remote2:

1 Like

A file having multiple parents is one of those features which seems like a good idea but causes no end of chaos, so I think removing it is a good plan. However its removal will cause a bit of disruption and I've no idea how Google are going to convert those files with multiple parents into shortcuts. How will they choose which parent is the real parent? I predict some messy fallout from that!

1 Like

Issue posted: #4121.

The issue got made here #4098.

I've had a go at this - please give it a whirl and report back

https://beta.rclone.org/branch/v1.51.0-163-g8b5786ab-fix-4098-drive-shortcuts-beta/ (uploaded in 15-30 mins)

OK that turned out to be a lot harder than I anticipated!

https://beta.rclone.org/branch/v1.51.0-163-gf22af086-fix-4098-drive-shortcuts-beta/ (uploaded in 15-30 mins)

Here are the docs I had to make for it

Shortcuts

In March 2020 Google introduced a new feature in Google Drive called drive shortcuts (API). These will (by September 2020) replace the ability for files or folders to be in multiple folders at once.

Shortcuts are files that link to other files on Google Drive somewhat like a symlink in unix, except they point to the underlying file data (eg the inode in unix terms) so they don't break if the source is renamed or moved about.

Be default rclone treats these as follows.

For shortcuts pointing to files:

  • When listing a file shortcut appears as the destination file.
  • When downloading the contents of the destination file is downloaded.
  • When updating shortcut file with a non shortcut file, the shortcut is removed then a new file is uploaded in place of the shortcut.
  • When server side moving (renaming) the shortcut is renamed, not the destination file.
  • When server side copying the shortcut is copied, not the contents of the shortcut.
  • When deleting the shortcut is deleted not the linked file.
  • When setting the modification time, the modification time of the linked file will be set.

For shortcuts pointing to folders:

  • When listing the shortcut appears as a folder and that folder will contain the contents of the linked folder appear (including any sub folders)
  • When downloading the contents of the linked folder and sub contents are downloaded
  • When uploading to a shortcut folder the file will be placed in the linked folder
  • When server side moving (renaming) the shortcut is renamed, not the destination folder
  • When server side copying the contents of the linked folder is copied, not the shortcut.
  • When deleting with rclone rmdir or rclone purge the shortcut is deleted not the linked folder.
  • NB When deleting with rclone remove or rclone mount the contents of the linked folder will be deleted.

It isn't currently possible to create shortcuts with rclone.

Shortcuts can be completely ignored with the --drive-skip-shortcuts flag
or the corresponding skip_shortcuts configuration setting.

Hopefully this one will actually build - sorry!

https://beta.rclone.org/branch/v1.51.0-171-g5d91b677-fix-4098-drive-shortcuts-beta/ (uploaded in 15-30 mins)

This feature change is going to be fun. Managed to pull out a list(1) of over 100 folders I have that are themselves in multiple other folders - I don't really fancy Google deciding which will be folders and which links, and that is before thinking about multihomed files.

But, Google's Android help topic identified above for these new links that will supercede objects in multiple folders is still suggesting users consider the obsolete method as an alternative!

Try these alternatives:

  • Instead of a shortcut, add the folder to an additional location. On the Google Drive website, select the folder and press SHIFT + Z.

(1)

rclone lsd -R drive: | sort -k1,4 | sed "s/.*\///" | uniq -cd | sort -rk1,1 > filename.txt
# Get list of folders with timestamps | sort by timestamp
# |extract foldername from full path so groups of folder
# locations are identical | identify folders with multiple
# instances, select just them and do counts | sort
# by frequency, and output to file

in case it is of use to anyone else. I couldn't see an obvious way to do it from Google's interface. Command depends on rclone lsd outputting folder creation dates which are same for multiple instances within other folders.

Hello everyone,

Just gave this a quick reading. My initial impressions are, (a) what a clusterf*ck coming from Google, and (b) kudos to @ncw for trying to fix and accommodate it all the best possible way.

Just one question: if I never "Shift-Z" anything on any of my Google drive accounts (which I'm pretty sure I didn't), can I just ignore all of this? Or is there anyway this can come and bite me in the rear when I'm not looking?

Thanks in advance for the clarifications,
-- Durval

Thank you very much for the beta rclone. It's work very well.
On two days test.
I found one question.

If one shortcut invalid like this
Imgur

ex 1:

gd1:
0418/music
0418/music_old <=invalid
0418/folder1
0418/folder2

rclone gd1:0418/ gd2:
rclone will not work

ex 2:

gd1:
0418/music/music_old <=invalid
0418/folder1
0418/folder2

rclone will work but folder music will be skip.

gd2 list lost folder music. but folder1 and folder2 will copy well.

gd2:
0418/folder1
0418/folder2