Rclone + GDrive no longer returning anything

Help and Support
1

What is the problem you are having with rclone?

I have a nightly clone from GDrive to an rclone folder for about 12 people. This has been working flawlessly for well over a year. but on May 28, it stopped. It didn't stop running, the clone from GDrive to local storage suddenly gets "nothing" from Google. No error, no authorization failure, nothing. IT's like the remote folders are blank. I've triped checked all the google side service authorizations and everything is OK.

The log from the May 28 run shows rclone deleting everything on the local side sync (thank god it's a one way sync!). It's like the entire remote folder structure is gone; which it is not. We've verified with each of the users that they still have loads of files/folders on Gdrive.

May 28th also happens to coincide with upgrading from 1.48 to 1.52. It was right after the upgrade that this started to fail. I tried downgrading back to1.48, but get the same error. (don't know if this is relate.) (Personal identifiable information below has been redacted)

What is your rclone version (output from rclone version)

rclone v1.52.1
os/arch: linux/amd64
go version: go1.14.4

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux vl007 4.4.0-178-generic #208-Ubuntu SMP Sun Apr 5 23:45:10 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

/usr/local/bin/rclone  --config=/usr/local/etc/rclone/rclone.conf --drive-impersonate notify@xxxxx.com --log-level DEBUG  lsf gdrive:

The rclone config contents with secrets removed.

[gdrive]
type = drive
#scope = drive.readonly
scope = drive
service_account_file = /usr/local/etc/rclone/gdrive.json
client_id = xxxxxxxxxxxxxxxxxxxxx
root_folder_id = xxxxxxxxxxx-xxxxxxx

A log from the command with the -vv flag

www-data@vl007:~$  /usr/local/bin/rclone  --config=/usr/local/etc/rclone/rclone.conf --drive-impersonate notify@xxxxxx.com -vv lsf gdrive:
2020/06/15 16:34:40 DEBUG : rclone: Version "v1.52.1" starting with parameters ["/usr/local/bin/rclone" "--config=/usr/local/etc/rclone/rclone.conf" "--drive-impersonate" "notify@xxxxxx.com" "-vv" "lsf" "gdrive:"]
2020/06/15 16:34:40 DEBUG : Using config file from "/usr/local/etc/rclone/rclone.conf"
2020/06/15 16:34:40 DEBUG : 6 go routines active
2

Are you able to check if the service account has lost permissions? Can yo try with a regular user account?

3

I know what has happened. Add the flag --drive-root-folder-id "" and it should all be well.

This is a bug that I need to fix and somehow slipped of my radar for the 1.52 release. Th problem is that rclone is caching the root folder ID in the config file so effectively rclone is looking at the wrong directory :frowning:

I'll work on a fix for this

4

This fix for this is annoying. I made a mistake caching the root ID of the drive into root_folder_id because the user can set that too. The second mistake was caching the root_folder_id when --drive-impersonate is in use - that is what leads to the problem here.

So now I don't know if the user set root_folder_id or not.

Potential solutions

  1. if --drive-impersonate is set ignore root_folder_id.
    • this fixes all the problems provided the user didn't really want --drive-impersonate with a root_folder_id - it will fail silently for this unusual case by showing the wrong directory.
    • it is a nice simple fix
  2. If --drive-impersonate and root_folder_id is set, then check the user can access the root folder - if not then clear the root_folder_id
    • this fix would work for this issue
    • this fix would allow unusual setups with root_folder_id and impersonate to carry on working with the cost of an extra transaction
    • this still has the opportunity to come up with the wrong folder if the impersonated user happens to have access to the stored root_folder_id
  3. If impersonate and root_folder_id are set then stop with an error and make the user take a corrective action
    • 100% reliable
    • inconvenient!
    • maybe too intrusive for a point release
    • for people who have upgraded from pre 1.51 to this new version then they won't have a cached root_folder_id so it will be seamless, for all other users of --impersonate it will cause their scripts to break

Any thoughts on the right course of action here? I don't think an obvious right choice jumps out at me!

1 Like
5

--drive-root-folder-id "" works ! Thanks.

You are right.. I was going to ask why does my config file get this value called root folder id after running. Discovered that late last night.

IMHO, your first option is the best. If I am running impersonate, that's where I want to start. Also, when that option is set, do not update root drive id in the conf file.
You could add, if the root-folder-id is not equal to the root-folder-id for the impersonation, issue a warning, but keep working on the impersonate folder?

6

Great

Yes that is right.

That is the mistake I made originally! If I'd done that then all would have been fine.

That will issue a warning if you use rclone without --impersonate, then use it with --impersonate as there will be a cached root_folder_id. Do you think that is a likely scenario?

7

I implemented the simplest possible thing

This

  • ignores root_folder_id if using impersonate and logs a NOTICE message that it is doing so
  • does not cache root_folder_id if using impersonate

Do you think you could test that out for me? I don't have access to a google account which allows this.

I'd be grateful if you could try using rclone with the --drive-impersonate flag then using it without, then using it with the --drive-impersonate flag it should show it working. Just listing a directory is enough - thank you :slight_smile:

https://beta.rclone.org/branch/v1.52.1-095-gf1772a2d-fix-drive-impersonate-beta/ (uploaded in 15-30 mins)

8

I assume you also want me to run without -drive-root-folder-id "" ?

With drive-impersonate

vl007:~$ /usr/local/bin/rclone-v1.52.1-095 --config=/usr/local/etc/rclone/rclone.conf --drive-alternate-export -vv --drive-impersonate user@xxxxxx.com  lsf gdrive:/

2020/06/18 11:18:25 DEBUG : rclone: Version "v1.52.1-095-gf1772a2d-fix-drive-impersonate-beta" starting with parameters ["/usr/local/bin/rclone-v1.52.1-095" "--confi\
g=/usr/local/etc/rclone/rclone.conf" "--drive-alternate-export" "-vv" "--drive-impersonate" "user@xxxxxx.com" "lsf" "gdrive:/"]
2020/06/18 11:18:25 DEBUG : Using config file from "/usr/local/etc/rclone/rclone.conf"
2020/06/18 11:18:25 NOTICE: Google drive root '': Ignoring cached root_folder_id when using --drive-impersonate
2020/06/18 11:18:25 DEBUG : fs cache: renaming cache item "gdrive:/" to be canonical "gdrive:"
Before 2019/
Client FIles/
Personal／Misc/
Checklist Template.docx     
Basic Research.docx      
School/
Templates/
Untitled document-2.docx
2020/06/18 11:18:26 DEBUG : 6 go routines active

Without drive-impersonate

vl007:~$ /usr/local/bin/rclone-v1.52.1-095 --config=/usr/local/etc/rclone/rclone.conf --drive-alternate-export -vv  lsf gdrive:/         

2020/06/18 11:18:41 DEBUG : rclone: Version "v1.52.1-095-gf1772a2d-fix-drive-impersonate-beta" starting with parameters ["/usr/local/bin/rclone-v1.52.1-095" "--confi\
g=/usr/local/etc/rclone/rclone.conf" "--drive-alternate-export" "-vv" "lsf" "gdrive:/"]                             
2020/06/18 11:18:41 DEBUG : Using config file from "/usr/local/etc/rclone/rclone.conf"
2020/06/18 11:18:41 DEBUG : fs cache: renaming cache item "gdrive:/" to be canonical "gdrive:"
2020/06/18 11:18:42 DEBUG : 6 go routines active

With drive-impersonate

vl007:~$ /usr/local/bin/rclone-v1.52.1-095 --config=/usr/local/etc/rclone/rclone.conf --drive-alternate-export -vv --drive-impersonate user@xxxxxx.com  lsf gdrive:/ 

2020/06/18 11:18:53 DEBUG : rclone: Version "v1.52.1-095-gf1772a2d-fix-drive-impersonate-beta" starting with parameters ["/usr/local/bin/rclone-v1.52.1-095" "--confi\
g=/usr/local/etc/rclone/rclone.conf" "--drive-alternate-export" "-vv" "--drive-impersonate" "user@xxxxxx.com" "lsf" "gdrive:/"]         
2020/06/18 11:18:53 DEBUG : Using config file from "/usr/local/etc/rclone/rclone.conf"
2020/06/18 11:18:53 NOTICE: Google drive root '': Ignoring cached root_folder_id when using --drive-impersonate
2020/06/18 11:18:53 DEBUG : fs cache: renaming cache item "gdrive:/" to be canonical "gdrive:"
Before 2019/
Client FIles/
Personal／Misc/
Checklist Template.docx     
Basic Research.docx       
School/
Templates/
Untitled document-2.docx
2020/06/18 11:18:54 DEBUG : 6 go routines active
1 Like
9

Yes.

Your tests look great, thank you.

I've merged this to master now which means it will be in the latest beta in 15-30 mins and released in v1.52.2 which should be quite soon (days)

10

With v1.52.2 my apparently unusual combination of

impersonate = <email address>
root_folder_id = appDataFolder

has stopped working and I can no longer access my data. I had to roll back to v1.52.1.

It seems that you need a special case for root_folder_id = appDataFolder...

11

I hoped that would be a setup so unusual that nobody would be using it, but I knew, deep down, that would break someones setup somewhere!

Did you notice the log message?

I made a work-around for appDataFolder here - can you give it a go please?

https://beta.rclone.org/branch/v1.52.2-131-g5fb392aa-fix-drive-impersonate-beta/ (uploaded in 15-30 mins)

12

Indeed. I think I'm the only one using it. :slight_smile:

I seem to remember that if you want to use a service account then you basically have to use impersonate, to get the proper quotas. It's a small step from there to using root_folder_id = appDataFolder...

Anyway, it's on you for using a config file as a cache. :wink:

Also, I'm in the process of moving my data out of the Application Data folder, primarily to use the new shortcuts feature, so you'll probably get less annoyance from me in the future.

Yes I did. Good job! :+1:

In fact, the log message seems to be duplicated:

2020/06/25 18:48:04 NOTICE: Google drive root 'rclone': Ignoring cached root_folder_id when using --drive-impersonate
2020/06/25 18:48:05 NOTICE: Google drive root 'rclone': Ignoring cached root_folder_id when using --drive-impersonate

It works very well. Thank you. Will there be a Docker image any time soon?

13

Thank you for testing

I've merged this to master now which means it will be in the latest beta in 15-30 mins and also in the :beta docker image about then.

I've queued it for v1.52.3 also!

14

Hey, I just wanted to chime in and say my google drive mounts all broke today.

I rolled back to an earlier version and things started working again. Yes, I use "impersonate" and "root_folder_id".

TBH, I'm struggling to understand why that is such an unusual combination. It seems normal to me that you want to name the root folder, and the user who owns it.

15

hello, you are supplying little information.
the more you share, the more we can help.

What is the problem you are having with rclone?

What is your rclone version (output from rclone version)

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Paste command here

The rclone config contents with secrets removed.

Paste config here

A log from the command with the -vv flag

Paste  log here

also,
what version broke?
what version worked?
did you try all the suggestions in the post?
did you try the beta?

16

I'm working on this at the moment, sorry for the inconvenience.

I've been trying to work-around a mistake I made caching the root_folder_id but I've been making things worse...

See: https://github.com/rclone/rclone/issues/4419 for the plan

1 Like
17

I had a go at fixing this again

v1.52.2-249-ga26ef2e3-fix-4419-drive-root-folder-id-beta on branch fix-4419-drive-root-folder-id (uploaded in 15-30 mins)

This removes all the caching of root_folder_id and just gives the user a hint in the debug messages that setting it would speed things up.

This should hopefully fix your problems @WalkingDude - I'd appreciate it if you'd give it a test.

18

Hi @ncw

Thanks for doing this. It failed, but in a strange way.

I have included the whole command I run, in case I'm just plain doing it wrong.

There are three things in play:

  1. root_folder_id= in rclone.conf
  2. impersonate= in rclone.conf
  3. Path under 'drivename:/path' (which maybe should be 'drivename:path'

It mounted the path. 'ls' showed the top level folders.

Attempting to access a top level folder caused it to show the message
path: Transport endpoint is not connected

By that time rclone had quit, but the mountpoint still existed.
All this is under ubuntu 20.04, in case that helps.

+ exec /usr/bin/rclone mount -v -v -v -P --rc --user-agent xxxxxxxxxxxx --allow-non-empty --cache-db-purge --write-back-cache --allow-other --uid 1001 --gid 1001 --umask 002 --no-modtime --drive-use-trash --dir-cache-time 100h --timeout 1h --attr-timeout 1m --buffer-size 1G --vfs-read-chunk-size 4M --vfs-read-chunk-size-limit off Rustdrive:/Collections /export/media/Rust/RustDrive
+ sleep 10
2020/07/26 01:56:20 DEBUG : rclone: Version "v1.52.2-249-ga26ef2e3-fix-4419-drive-root-folder-id-beta" starting with parameters ["/usr/bin/rclone" "mount" "-v" "-v" "-v" "-P" "--rc" "--user-agent" "ocelotmedia" "--allow-non-empty" "--cache-db-purge" "--write-back-cache" "--allow-other" "--uid" "1001" "--gid" "1001" "--umask" "002" "--no-modtime" "--drive-use-trash" "--dir-cache-time" "100h" "--timeout" "1h" "--attr-timeout" "1m" "--buffer-size" "1G" "--vfs-read-chunk-size" "4M" "--vfs-read-chunk-size-limit" "off" "Rustdrive:/Collections" "/export/media/Rust/RustDrive"]
2020/07/26 01:56:20 NOTICE: Serving remote control on http://127.0.0.1:5572/
2020/07/26 01:56:20 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2020/07/26 01:56:22 DEBUG : fs cache: renaming cache item "Rustdrive:/Collections" to be canonical "Rustdrive:Collections"
2020/07/26 01:56:22 DEBUG : Google drive root 'Collections': Mounting on "/export/media/Rust/RustDrive"
2020/07/26 01:56:22 DEBUG : : Root:
2020/07/26 01:56:22 DEBUG : : >Root: node=/, err=<nil>
2020/07/26 01:56:25 DEBUG : /: Attr:
2020/07/26 01:56:25 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2020/07/26 01:56:25 DEBUG : : Statfs:
2020/07/26 01:56:25 DEBUG : : >Statfs: stat={Blocks:289983601161 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2020/07/26 01:56:26 DEBUG : /: Attr:
2020/07/26 01:56:26 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2020/07/26 01:56:27 DEBUG : /: Lookup: name="tracefs"
2020/07/26 01:56:27 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2020/07/26 01:56:27 DEBUG : /: Lookup: name="Rust"
2020/07/26 01:56:27 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2020/07/26 01:56:27 DEBUG : /: Lookup: name="Rust"
2020/07/26 01:56:27 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2020/07/26 01:56:27 DEBUG : /: Lookup: name="Rustdrive:Collections"
2020/07/26 01:56:27 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2020/07/26 01:56:27 DEBUG : /: ReadDirAll:
2020/07/26 01:56:27 DEBUG : /: >ReadDirAll: item=2, err=<nil>
2020/07/26 01:56:27 DEBUG : /: Lookup: name="Packages"
2020/07/26 01:56:27 DEBUG : /: >Lookup: node=Packages/, err=<nil>
2020/07/26 01:56:27 DEBUG : Packages/: Attr:
2020/07/26 01:56:27 DEBUG : Packages/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2020/07/26 01:56:27 DEBUG : /: Lookup: name="Archive"
2020/07/26 01:56:27 DEBUG : /: >Lookup: node=Archive/, err=<nil>
2020/07/26 01:56:27 DEBUG : Archive/: Attr:
2020/07/26 01:56:27 DEBUG : Archive/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2020/07/26 01:56:29 DEBUG : /: Attr:
2020/07/26 01:56:29 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2020/07/26 01:56:29 DEBUG : /: ReadDirAll:
2020/07/26 01:56:29 DEBUG : /: >ReadDirAll: item=2, err=<nil>
2020/07/26 01:56:29 DEBUG : /: Lookup: name="Packages"
2020/07/26 01:56:29 DEBUG : /: >Lookup: node=Packages/, err=<nil>
2020/07/26 01:56:29 DEBUG : Packages/: Attr:
2020/07/26 01:56:29 DEBUG : Packages/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2020/07/26 01:56:29 DEBUG : /: Lookup: name="Archive"
2020/07/26 01:56:29 DEBUG : /: >Lookup: node=Archive/, err=<nil>
2020/07/26 01:56:29 DEBUG : Archive/: Attr:
2020/07/26 01:56:29 DEBUG : Archive/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2020/07/26 01:56:30 DEBUG : /: Attr:
2020/07/26 01:56:30 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2020/07/26 01:56:30 DEBUG : /: ReadDirAll:
2020/07/26 01:56:30 DEBUG : /: >ReadDirAll: item=2, err=<nil>
2020/07/26 01:56:30 DEBUG : /: Lookup: name="Packages"
2020/07/26 01:56:30 DEBUG : /: >Lookup: node=Packages/, err=<nil>
2020/07/26 01:56:30 DEBUG : Packages/: Attr:
2020/07/26 01:56:30 DEBUG : Packages/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2020/07/26 01:56:30 DEBUG : Packages/: ReadDirAll:
2020/07/26 01:56:30 DEBUG : /: Attr:
2020/07/26 01:56:30 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2020/07/26 01:56:31 DEBUG : Packages/: >ReadDirAll: item=11, err=<nil>
2020/07/26 01:56:31 DEBUG : rclone: Version "v1.52.2-249-ga26ef2e3-fix-4419-drive-root-folder-id-beta" finishing with parameters ["/usr/bin/rclone" "mount" "-v" "-v" "-v" "-P" "--rc" "--user-agent" "ocelotmedia" "--allow-non-empty" "--cache-db-purge" "--write-back-cache" "--allow-other" "--uid" "1001" "--gid" "1001" "--umask" "002" "--no-modtime" "--drive-use-trash" "--dir-cache-time" "100h" "--timeout" "1h" "--attr-timeout" "1m" "--buffer-size" "1G" "--vfs-read-chunk-size" "4M" "--vfs-read-chunk-size-limit" "off" "Rustdrive:/Collections" "/export/media/Rust/RustDrive"]


[rustdrive]
type = drive
scope = drive
acknowledge_abuse = true
client_id = xxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com
client_secret = xxxxxxxxxxxxxxxxxxxxxxxx
chunk_size = 4M
use_trash = false
root_folder_id = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
service_account_file = $HOME/.config/rclone/xxxxxx-xxxxx-xxxxxxxxxxxx.json
impersonate = admin@xxxxxxx.xxx
19

It did didn't it!

Can you check that rclone lsf Rustdrive:/Collections shows the files you expect.

Your command was

/usr/bin/rclone mount -v -v -v -P --rc --user-agent ocelotmedia --allow-non-empty --cache-db-purge --write-back-cache --allow-other --uid 1001 --gid 1001 --umask 002 --no-modtime --drive-use-trash --dir-cache-time 100h --timeout 1h --attr-timeout 1m --buffer-size 1G --vfs-read-chunk-size 4M --vfs-read-chunk-size-limit off Rustdrive:/Collections /export/media/Rust/RustDrive

I don't know why rclone mount quit at that point. Can you try running it in the terminal?

This is a bit excessive --buffer-size 1G - this can potentially use 1G of memory per open file. I'd probably just remove that or set it much lower to 32M say.

20

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.