Rclone lsjson does not return all files

[plc (Paul C.)]

Hi

Sorry another newbie question

I'm trying to use rclone lsjson -R gdrive: for a Google drive. I have a huge directory shared with me (I think it contains more than 500k files) but the list returned by rclone (only 70k lines including directories) does not contains all the entries I can see when browsing the filetree with my browser.

I checked the json data returns which looks goot (not truncated).

Is it a known issue ? Any workaround ?

I'm using the lastes version 1.47 on a debian host

Thx

[calisro (Rob)]

Do you have an example file we can look at that is not listing in rclone but is listing in the webui? What kind of file is it? Can we locate that in the console and then do a rclone lsjson on that specific path with a log?

[ncw (Nick Craig-Wood)]

Is it a known issue ?

No this isn't a known issue. Can you try with and without --disable ListR and see if that makes a difference?

[plc (Paul C.)]

Do you have an example file we can look at that is not listing in rclone but is listing in the webui? What kind of file is it? Can we locate that in the console and then do a rclone lsjson on that specific path with a log?

Yes I found one file
rclone lsjson on this file works (see below)
I double checked by grepping the ID in my "full" list. It's not in this recursive listing

The logs gives

2019/05/22 22:12:00 DEBUG : rclone: Version "v1.47.0" starting with parameters ["rclone" "-vv" "lsjson" "gdrive:01 fichiers/events/A classer/rep/2018.11.15 presentation.pptx"]
2019/05/22 22:12:00 DEBUG : Using config file from "/xxx/.config/rclone/rclone.conf"
[
2019/05/22 22:12:04 DEBUG : 2018.11 xxx.docx: Excluded
2019/05/22 22:12:04 DEBUG : yyy.docx: Excluded
{"Path":"2018.11.15 presentation.pptx","Name":"2018.11.15 presentation.pptx","Size":-1,"MimeType":"application/vnd.openxmlformats-officedocument.presentationml.presentation","ModTime":"2018-11-19T17:16:21.985Z","IsDir":false,"ID":"97Z7kTo7ffeXyqppVDo79abcd7VjzkLQjG3yWGFCzzh0"}
]
2019/05/22 22:12:04 DEBUG : 4 go routines active
2019/05/22 22:12:04 DEBUG : rclone: Version "v1.47.0" finishing with parameters ["rclone" "-vv" "lsjson" "gdrive:01 fichiers/events/A classer/rep/2018.11.15 presentation.pptx"]

[plc (Paul C.)]

No this isn't a known issue. Can you try with and without --disable ListR and see if that makes a difference?

It's running

• 72194 lines without --disable ListR
• 920150 lines with --disable ListR

I also started a find -print on a mounted version of the drive. I'll post the results too but it slower

[ncw (Nick Craig-Wood)]

It's running

• 72194 lines without --disable ListR
• 920150 lines with --disable ListR

I presume the second ~920k is the correct answer?

I also started a find -print on a mounted version of the drive. I'll post the results too but it slower

I'd expect that to print 920k too.

So it looks like there is a bug in --fast-list aka ListR for the drive backend.

When you say this directory is shared with you, what do you mean? Are you using the --drive-shared-with-me flag?

Might you have something out of the ordinary like files with multiple parents?

If you diff the 70k and 920k listings can you notice any patterns? I'd sort them first then diff

diff -u <(sort <70klisting) <(sort <920klisting)

Maybe it is just one directory and all subsequent directories missing or something like that.