Polling for changes in a shortcut in the same drive

What is the problem you are having with rclone?

I have a Shared Drive that has Shortcuts to other Shared Drives in my account (which I own, too).
Mounting the Drive with all the shortcuts is a big problem because changes don't appear in the mount.

I read Google drive shortcuts and polling updates/cache and Nick already said

Poll changes won't detect changes on a different drive - we only poll the base drive. Even if it was a shortcut within the same drive, I'm not sure whether the updates would work properly....

but this makes a shortcut very useless in a mount.
Would it be possible to add a option to make polling updates to Shared Drives in the same Drive possible?
Maybe this is not possible for shortcuts which link outside of the account, but maybe for Shared Drives in the same account.

If this is not possible at all, maybe you should add a warning to rclone mount

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.0

  • os/version: Microsoft Windows Server 2022 Standard 21H2 (64 bit)
  • os/kernel: 10.0.20348.740 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.17.8
  • go/linking: dynamic
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount remote_acc: F:\mnt\remote_acc --tpslimit 9 --dir-cache-time 1000h --poll-interval 15s --cache-dir "F:\rclone cache\remote_acc" --vfs-cache-mode full --vfs-cache-max-size 50G --vfs-cache-max-age 24h  -vv --log-file F:\log.txt

The rclone config contents with secrets removed.

[remote_acc]
type = drive
client_id = REDACTED.apps.googleusercontent.com
client_secret = REDACTED
scope = drive
token = {"access_token":"REDACTED","token_type":"Bearer","refresh_token":"REDACTED","expiry":"REDACTED"}
team_drive = 0A......

A log from the command with the -vv flag

https://pastebin.com/raw/ZiqbSnrX

Would you be willing to name all the folder IDs you wanted polled? If not, rclone would have to traverse the entire remote looking for shortcuts which would be a pain.

Thanks for the quick reply.

Is it not possible for rclone to get the ID of the shortcutted folder?
I don't know exactly what Google responds with, but I don't think Google would miss out on that.

But yeah, if not possible in any other way, I wouldn't mind setting the IDs of the polled folders manually.

Thanks.

It is, but rclone would have to traverse the entire drive to find the shortcuts first, which would be very time consuming on startup.

I just had a look at the API for this

It looks like it only works if you give it a shared drive (team drive) ID. If you leave the ID out then you get changes for normal google drive. So you can't give it a folder ID.

So you'd need to give it IDs of shared drives.

This would make rclone's job very hard..

What you want is something more like the combine backend (currently still in development here Add new `Combine` Remote · Issue #5600 · rclone/rclone · GitHub ) and use this to mount the relevant shared drives. I don't think this supports changenotify yet, but it could relatively easily.

Ah, I see, makes sense.
I tried the beta and it's exactly what I'm looking for :slight_smile:

There is one problem tho:
I have crypted files in my Shared Drives, but the Shared Drives themself don't have crypted names.
The crypt works fine on the AllDrives and it can also decrypt the Shared Drive if it has a crypted name, but the problem is that I don't see the Shared Drives with the unencrypted names.
Dumb question, but is there a way to bypass renaming all my Shared Drives, or is this necessary?

Thank you so much, will donate to you if money is left over :smiley:

I see...

I can think of two possibilities,

  1. rename (in the config file only) the remotes into their crypted names. You can find the encrypted names using rclone backend encode secret: "unencrypted name"
  2. change the aliases to point to a crypted version of each drive

Assuming your crypt remote is called secret: and your main drive remote is called drive:

[ssasdds]
type = alias
remote = secret,remote="drive,team_drive=0AXXXXXXX,root_folder_id=:":

Now that I had more time to test it, there is still a problem with polling.
It says that it polls the root (which is hardcoded afaict), but the mount doesn't show any changes.
I mounted it with the following command (not crypted):

rclone mount AllDrives2: F:\mnt\remote_acc --tpslimit 9 --dir-cache-time 1000h --poll-interval 15s --cache-dir "F:\rclone cache\remote_acc" --vfs-cache-mode full --vfs-cache-max-size 50G --vfs-cache-max-age 24h  -vv --log-file=F:\log.txt

The log file says this:
https://paste.ee/r/N1Qbm

The config looks like this:

[remote_acc]
type = drive
client_id = REDACTED.apps.googleusercontent.com
client_secret = REDACTED
scope = drive
token = {"access_token":"REDACTED","token_type":"Bearer","refresh_token":"REDACTED","expiry":"REDACTED"}
root_folder_id = 

[_Test]
type = alias
remote = remote_acc,team_drive=0AREDACTED,root_folder_id=:

[AllDrives2]
type = combine
upstreams = "_Test=_Test:"



The renaming of the Shared Drives via config works very well tho!

Edit:
I oversaw that you wrote "I don't think this supports changenotify yet, but it could relatively easily.".
So, does Google have to add it in or do you have to add it to rclone?
Would be very nice if that is possible.

Its something I need to add to the combine backend. Actually I looked at the code and it looks like I added it already, it just doesn't work properly yet!

I've attempted to fix it here - please give it a go!

v1.59.0-beta.6171.7dcf7cf55.fix-5600-combine on branch fix-5600-combine (uploaded in 15-30 mins)

Oh, nice!
Could you also add Windows binaries into the beta, please? :sweat_smile:
Thanks!

Try this

https://beta.rclone.org/branch/fix-5600-combine/v1.59.0-beta.6174.910778ba2.fix-5600-combine/

Mount works perfectly with this!
Thank you so much for your efforts :slight_smile:

1 Like

I got this error with this ls command in the Shared Drives root which makes no sense to me:

rclone ls remote_acc_all:td --config "F:\mnt\conf.conf" -vv

C:\Users\Administrator>rclone ls remote_acc_all:td --config "F:\mnt\conf.conf" -vv
2022/06/01 13:21:57 DEBUG : rclone: Version "v1.59.0-beta.6174.910778ba2.fix-5600-combine" starting with parameters ["rclone" "ls" "remote_acc_all:td" "--config" "F:\\mnt\\conf.conf" "-vv"]
2022/06/01 13:21:57 DEBUG : Creating backend with remote "remote_acc_all:td"
2022/06/01 13:21:57 DEBUG : Using config file from "F:\\mnt\\conf.conf"
2022/06/01 13:21:57 DEBUG : Creating backend with remote "td:"
2022/06/01 13:21:57 DEBUG : Creating backend with remote "remote_acc,team_drive=0AFTREDACTED,root_folder_id=:"
2022/06/01 13:21:57 DEBUG : remote_acc: detected overridden config - adding "{1tAz4}" suffix to name
2022/06/01 13:21:57 DEBUG : fs cache: renaming cache item "remote_acc,team_drive=0AFTREDACTED,root_folder_id=:" to be canonical "remote_acc{1tAz4}:"
2022/06/01 13:21:57 DEBUG : fs cache: renaming cache item "td:" to be canonical "remote_acc{1tAz4}:"
2022/06/01 13:21:57 Failed to create file system for "remote_acc_all:td": is root directory

The config is:

[remote_acc]
type = drive
client_id = REDACTED.apps.googleusercontent.com
client_secret = REDACTED
scope = drive
token = {"access_token":"REDACTED","token_type":"Bearer","refresh_token":"REDACTED","expiry":"REDACTED"}
root_folder_id = 

[td]
type = alias
remote = remote_acc,team_drive=0AFTREDACTED,root_folder_id=:

[remote_acc_all]
type = combine
upstreams = "td=td:"

Is this normal behaviour?
It works fine if I do this command where it goes through all Shared Drives

rclone ls remote_acc_all: --config "F:\mnt\conf.conf" -vv (without specifying the td)

Edit: Looks like nothing is working except mounting if a path is given.

Oops a little bug!

Try this

v1.59.0-beta.6174.9aa5ce515.fix-5600-combine on branch fix-5600-combine (uploaded in 15-30 mins)

Works perfectly now, thank you very much, I left you a donation in Bitcoins :slight_smile:

Great and thank you :slight_smile:

Looks like I found another issue with subfolders:

>rclone lsjson remote_acc_all: -vv
2022/06/03 12:32:48 DEBUG : rclone: Version "v1.59.0-beta.6174.9aa5ce515.fix-5600-combine" starting with parameters ["rclone" "lsjson" "remote_acc_all:" "-vv"]
2022/06/03 12:32:48 DEBUG : Creating backend with remote "remote_acc_all:"
2022/06/03 12:32:48 DEBUG : Using config file from "C:\\Users\\Administrator\\AppData\\Roaming\\rclone\\rclone.conf"
2022/06/03 12:32:48 DEBUG : Creating backend with remote "remote_acc_fonts:"
2022/06/03 12:32:48 DEBUG : Creating backend with remote "remote_acc,team_drive=REDACTED,root_folder_id=:"
2022/06/03 12:32:48 DEBUG : remote_acc: detected overridden config - adding "{pqTKq}" suffix to name
2022/06/03 12:32:48 DEBUG : fs cache: renaming cache item "remote_acc,team_drive=REDACTED,root_folder_id=:" to be canonical "remote_acc{pqTKq}:"
2022/06/03 12:32:48 DEBUG : fs cache: renaming cache item "remote_acc_fonts:" to be canonical "remote_acc{pqTKq}:"
[
{"Path":"Fonts","Name":"Fonts","Size":-1,"MimeType":"inode/directory","ModTime":"2022-06-03T12:32:48.609+02:00","IsDir":true}
]
2022/06/03 12:32:48 DEBUG : 2 go routines active

>rclone lsjson remote_acc_all:Fonts -vv
2022/06/03 12:32:50 DEBUG : rclone: Version "v1.59.0-beta.6174.9aa5ce515.fix-5600-combine" starting with parameters ["rclone" "lsjson" "remote_acc_all:Fonts" "-vv"]
2022/06/03 12:32:50 DEBUG : Creating backend with remote "remote_acc_all:Fonts"
2022/06/03 12:32:50 DEBUG : Using config file from "C:\\Users\\Administrator\\AppData\\Roaming\\rclone\\rclone.conf"
2022/06/03 12:32:50 DEBUG : Creating backend with remote "remote_acc_fonts:"
2022/06/03 12:32:50 DEBUG : Creating backend with remote "remote_acc,team_drive=REDACTED,root_folder_id=:"
2022/06/03 12:32:50 DEBUG : remote_acc: detected overridden config - adding "{pqTKq}" suffix to name
2022/06/03 12:32:50 DEBUG : fs cache: renaming cache item "remote_acc,team_drive=REDACTED,root_folder_id=:" to be canonical "remote_acc{pqTKq}:"
2022/06/03 12:32:50 DEBUG : fs cache: renaming cache item "remote_acc_fonts:" to be canonical "remote_acc{pqTKq}:"
[
{"Path":"Random Fonts","Name":"Random Fonts","Size":-1,"MimeType":"inode/directory","ModTime":"2022-06-03T10:26:20.085Z","IsDir":true,"ID":"1_AVYJlUVBuabatcg9MWiOiRFA4hrqMAY"}
]
2022/06/03 12:32:50 DEBUG : 4 go routines active

>rclone lsjson "remote_acc_all:Fonts/Random Fonts" -vv
2022/06/03 12:32:52 DEBUG : rclone: Version "v1.59.0-beta.6174.9aa5ce515.fix-5600-combine" starting with parameters ["rclone" "lsjson" "remote_acc_all:Fonts/Random Fonts" "-vv"]
2022/06/03 12:32:52 DEBUG : Creating backend with remote "remote_acc_all:Fonts/Random Fonts"
2022/06/03 12:32:52 DEBUG : Using config file from "C:\\Users\\Administrator\\AppData\\Roaming\\rclone\\rclone.conf"
2022/06/03 12:32:52 DEBUG : Creating backend with remote "remote_acc_fonts:"
2022/06/03 12:32:52 DEBUG : Creating backend with remote "remote_acc,team_drive=REDACTED,root_folder_id=:"
2022/06/03 12:32:52 DEBUG : remote_acc: detected overridden config - adding "{pqTKq}" suffix to name
2022/06/03 12:32:52 DEBUG : fs cache: renaming cache item "remote_acc,team_drive=REDACTED,root_folder_id=:" to be canonical "remote_acc{pqTKq}:"
2022/06/03 12:32:52 DEBUG : fs cache: renaming cache item "remote_acc_fonts:" to be canonical "remote_acc{pqTKq}:"
2022/06/03 12:32:52 Failed to create file system for "remote_acc_all:Fonts/Random Fonts": couldn't list directory: context canceled


>rclone version
rclone v1.59.0-beta.6174.9aa5ce515.fix-5600-combine
- os/version: Microsoft Windows Server 2022 Standard 21H2 (64 bit)
- os/kernel: 10.0.20348.740 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.18.2
- go/linking: static
- go/tags: cmount

The config looks like this:


[remote_acc]
type = drive
client_id = REDACTED.apps.googleusercontent.com
client_secret = REDACTED
scope = drive
token = {"access_token":"REDACTED","token_type":"Bearer","refresh_token":"REDACTED","expiry":"REDACTED"}
team_drive = REDACTED
root_folder_id = 

[remote_acc_fonts]
type = alias
remote = remote_acc,team_drive=REDACTED,root_folder_id=:

[remote_acc_all]
type = combine
upstreams = "Fonts=remote_acc_fonts:"

Could you please have a look at this? :slight_smile:
Thanks!

That's weird! I can reproduce that no problems. It will work if you put a / on the end of the path.

I'll fix and post an update in a day or two!

1 Like

Any updates yet? :slight_smile:

I fixed that now - try

v1.59.0-beta.6174.91b67758c.fix-5600-combine on branch fix-5600-combine (uploaded in 15-30 mins)

Works perfectly, thank you very much again!