I'm just curious if anyone has pointed a filename search indexer at an rclone mount, and if there are any caveats> I'm planning on trying this with the Void tools Everything indexer. I only want file name searching, but do wonder if I'll end up with unexpected excessive usage as a result. I plan to limit file system scans to 4 or 6 times a day.
Anyway, if anyone tried this and knows if it's fine or has an unexpected side effect, I'd be interested in hearing your experiences.
Thanks!
hi,
what backend are you using?
before running the indexer, you will want to prime the mount
Google Drive, Plex, Windows 10
and this
https://rclone.org/commands/rclone_mount/#vfs-performance
GDrive backend, using a variation of Animosity22's settings for the mount, with it set read-only. I am using vfs cache mode full, just in case the fractional reading mattered for the file system names, though it wasn't clear if it would, for this use case.
The priming bit is interesting, I had wondered about pre-loading the file system structure. I'll have to test it/think about it a little to see if it's worth doing or not, in terms of the refresh time for the file system structure.
Thanks!
vfs cache is for the contents of files, not for username and other meta data such as mod-time
let us know what you find.
So far, it's been promising. The priming operation did help my index file scan immensely, I'm thinking right now I'll automate the priming to occur at a schedule offset from the indexer refresh, and it seems like it'll work well, at least for now. The question for me becomes one of, how well does it scale as more items get put into the tree.
The interesting issue I'm having now is that the Everything indexer is trying to be a little too clever, and is doing something weird where instead of showing me the local file path, it's attempting to map the mount point into a UNC accessible path, which ends up not being helpful (I'm trying to run the indexer from a single point, it can can act as a client to itself for the remote index, but mapping the paths messes it up). Anyway, that's a separate issue on a different program.
perhaps try to mount to a folder instead of a drive letter.
which is how i always mount.
rclone.exe mount gdrive-a1b2: b:\mount\rclone\gdrive-a1b2
try to scan from your equivalent of
b:\mount\rclone\gdrive-a1b2
and
b:\mount\rclone\
for example, if i want to net share the mount, it works much better if i share b:\mount\rclone\, which is a ntfs folder, instead of the fake FUSE link of b:\mount\rclone\gdrive-a1b2
I don't think it will matter, the problem is the Everything search indexer is being a little too helpful. It knows that it's indexing a local path, and that the client requesting the search result is not itself, so it's trying to present the result as a UNC path so that the client could possibly reach the path. It's making an assumption that the same drive letter is not/may not be available from the remote, but presenting is as a UNC path with the drive letter might work (it won't, of course).
i.e. index location is T:
remote client, instead of getting a result as T:\something gets \server\T\something
It's not a bad idea in typical circumstances, it just doesn't really work that way in this one. I don't necessarily want to pull the file over the share that way (rclone->windows share). Though I suppose I could. So if it comes down to it, that might be a workaround. It would probably simplify some of the other things, I just wasn't completely sure of how great that would be re-sharing the share (and if that would mess with vfs caching performance- I was concerned if putting the rclone behind an SMB share might cause unexpected side effects, specifically that the share might make it look like the entire file was being requested at once).
I know Everything tries to pull extra ntfs metadata as well, so I also wondered if mounting under an actual ntfs path might cause weird things as well. Still, maybe it's time I explored those options as well.
Thanks!
keep in mind that the only metadata rclone uses is for files and only mod-time and size.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.
