Make Mounted Google Drive to Search Faster in Windows Explorer

not sure that that means?
please post a debug log

https://forum.rclone.org/t/search-a-file-on-windows-explorer/26169

second commend, then it didn't work

I meant the "rc" commend line. The first line is just the mount commend line.

C:\Users\A\Desktop\rclone>rclone rc vfs/refresh recursive=true -vv
2021/09/05 15:04:42 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "rc" "vfs/refresh" "recursive=true" "-vv"]
2021/09/05 15:09:43 DEBUG : 2 go routines active
2021/09/05 15:09:43 Failed to rc: connection failed: Post "http://localhost:5572/vfs/refresh": net/http: timeout awaiting response headers
C:\Users\A\Desktop\rclone>PAUSE
Press any key to continue . . .

I checked the option "Don't use the index when searching in file folder for system files (searches might take longer)"

That didn't improve search speed.

the rclone rc command failed.

post the mount debug log, if it is very large in size, then post the top 20 lines.

as long as a command is running, the cmd window will not accept new commands.
to work around that prefix the command with start
start rclone mount ....

might want to add --no-console to the rclone mount command.

Ok before getting the debug log, I run a batch file for my commends. Maybe you help me to correct anything I am doing wrong.

@ECHO OFF
cd C:\Users\A\Desktop\rclone
start rclone mount --rc --vfs-cache-mode off --no-console --cache-dir D:/temp remote:/ C:/mount -vv
rclone rc vfs/refresh recursive=true -vv

  1. since you are not using a log file, would not use --no-console.
  2. start forks the rclone mount and returns immediately before the rclone mount is active.
    so there is a chance that rclone rc will try to access a mount that is not ready and error out.
    i would add timeout /t 20 between the rclone mount and the rclone rc
  3. would be better to use cd /d, not cd

@ECHO OFF
cd /d C:\Users\A\Desktop\rclone
start rclone mount --rc --vfs-cache-mode off --no-console --log-file=mylogfile.txt --cache-dir D:/temp remote: C:/mount -vv
timeout /t 20
rclone rc vfs/refresh recursive=true -vv

Log File

2021/09/06 19:24:41 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "mount" "--rc" "--vfs-cache-mode" "off" "--no-console" "--log-file=mylogfile.txt" "--cache-dir" "D:/temp" "remote:" "C:/mount" "-vv"]
2021/09/06 19:24:41 NOTICE: Serving remote control on http://localhost:5572/
2021/09/06 19:24:41 DEBUG : Creating backend with remote "remote:"
2021/09/06 19:24:41 DEBUG : Using config file from "C:\Users\A\.config\rclone\rclone.conf"
2021/09/06 19:24:42 DEBUG : Network mode mounting is disabled
2021/09/06 19:24:42 DEBUG : Mounting on "C:\mount" ("remote")
2021/09/06 19:24:42 DEBUG : Google drive root '': Mounting with options: ["-o" "attr_timeout=1" "-o" "uid=-1" "-o" "gid=-1" "--FileSystemName=rclone" "-o" "volname=remote"]
2021/09/06 19:24:42 DEBUG : Google drive root '': Init:
2021/09/06 19:24:42 DEBUG : Google drive root '': >Init:
2021/09/06 19:24:42 DEBUG : /: Statfs:
2021/09/06 19:24:42 DEBUG : Google drive root '': read info from Shared Drive "remote drive [1]"
2021/09/06 19:24:42 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2021/09/06 19:24:42 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/09/06 19:24:42 DEBUG : /: >Getattr: errc=0
2021/09/06 19:24:42 DEBUG : /: Readlink:
2021/09/06 19:24:42 DEBUG : /: >Readlink: linkPath="", errc=-40
The service rclone has been started.
2021/09/06 19:24:42 DEBUG : /: Statfs:
2021/09/06 19:24:42 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2021/09/06 19:24:42 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/09/06 19:24:42 DEBUG : /: >Getattr: errc=0
2021/09/06 19:24:42 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/09/06 19:24:42 DEBUG : /: >Getattr: errc=0
2021/09/06 19:24:42 DEBUG : /: Opendir:
2021/09/06 19:24:42 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2021/09/06 19:24:42 DEBUG : /: >OpenFile: fd=/ (r), err=
2021/09/06 19:24:42 DEBUG : /: >Opendir: errc=0, fh=0x0
2021/09/06 19:24:42 DEBUG : /: Releasedir: fh=0x0
2021/09/06 19:24:42 DEBUG : /: >Releasedir: errc=0
2021/09/06 19:25:00 DEBUG : rc: "vfs/refresh": with parameters map[recursive:true]
2021/09/06 19:25:00 DEBUG : : Reading directory tree
2021/09/06 19:25:09 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2021/09/06 19:25:09 DEBUG : pacer: Rate limited, increasing sleep to 1.004651028s
2021/09/06 19:25:09 DEBUG : pacer: Reducing sleep to 0s
2021/09/06 19:25:09 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2021/09/06 19:25:09 DEBUG : pacer: Rate limited, increasing sleep to 1.510359739s
2021/09/06 19:25:09 DEBUG : pacer: Reducing sleep to 0s
2021/09/06 19:25:09 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2021/09/06 19:25:09 DEBUG : pacer: Rate limited, increasing sleep to 1.385573975s
2021/09/06 19:25:10 DEBUG : pacer: Reducing sleep to 0s
2021/09/06 19:25:42 DEBUG : Google drive root '': Checking for changes on remote

look like rclone is getting throttled by gdrive.
that could be the reason search using windows explorer is so slow.

have you done this, if not, should do so and test again.
https://rclone.org/drive/#making-your-own-client-id

You have to use the search function via https://drive.google.com/ unless there are just a few files.

Windows Explorer has to crawl each file which takes forever when you have many files, because google limits it heavily to prevent abuses.

The throttling here seems to be happening while the refresh command is running. We need to know if it ever completes successfully. OP, you should see a status of "OK" at the end. Searching should not be slow at all once priming is done. I regularly search through tens of thousands of files and folders.

good point, the truth is i made a choice to pretend not to notice.
that i would miss something we know could not happen

so it must be that, i set a trap for you, yes, yes, now that i am thinking about it....
from a lurking vigilante in the background to a poster in public :wink:

i know of your experience with vfs/refresh and gdrive, i deflected, made mention about the need for a client id, without mention that the pacer began after that vfs/refresh

it all makes perfect sense to me!

now, back to reality.

in your rclone vfs/refresh commands, there are no flags to deal with pacer issues, correct?
given that the OP has not posted the config, do you think having a client id would resolve that or should the OP add some flags?

It shouldn't be necessary to run with additional flags, but I do have a few:

rclone rc vfs/refresh recursive=true --drive-pacer-burst 200 --drive-pacer-min-sleep 10ms --timeout 30m --user-agent *******

I never have any issues running a refresh, but I also never look at any logs.

The above runs through roughly 820 TB in four minutes.

Well, my rclone config is everything to default.

The google drive I am connecting is a shared drive and it usually runs into bandwidth limits. By reading what is written in this thread, I guess I can do nothing about the slow search when google is throttling my connection. It is kinda stupid that I cannot use Explorer to find files simply by file names.

that is why i do not use gdrive.

you can try the vfs/refresh command that was shared by @VBB

I have tried the below with what VBB's have suggested, but it seems the search speed did not improve.

For Google Drive since it's a polling remote, you want to use a high dir-cache-time.

I use:

--dir-cache-time 5000h

I tried added that but didn't improve search speed.

@ECHO OFF
cd /d C:\Users\A\Desktop\rclone
start rclone mount --rc --vfs-cache-mode off --no-console --log-file=mylogfile.txt --dir-cache-time 5000h --cache-dir D:/temp remote: C:/mount -vv
timeout /t 20
rclone rc vfs/refresh recursive=true --drive-pacer-burst 200 --drive-pacer-min-sleep 10ms --timeout 30m --user-agent *******

You need to share a log file when you make a change as we can't see what's going on.

earlier in the post, i shared a link from another topic with the same issue and contained a solution.
did you try that solution and it failed to solve your problem?
i never got a reply from you about that?
https://forum.rclone.org/t/search-a-file-on-windows-explorer/26169/15

also, i suggested the need to create and use a client id.
have you done that?

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.