Rclone mount can't read files from dir with a very large number of files and dirs

Help and Support
1

What is the problem you are having with rclone?

Using: rclone v1.69.0
My encrypted google drive is mounted on my system using the latest version of macFuse 4.9.1. Never had any issues, until now that I'm trying to read files from a very large collection of images. I cannot read files stored in a directory with many files/folders around 500k.

The files are stored in the following structure. Each image file is in its own "info" folder and all those "info" folders are located inside the images folder.

  • photos.library/
    • images/
      • M5Q2Z2JKV3WTK.info/
        • wallpaper.jpg
      • P5Q2S2JKV3WTJ.info/
        • tree.jpg
      • X5Q2ZUJKV3WTG.info/
        • ocean.jpg
      • Q5Q2Z2JKV3WTA.info/
        • waterfall.jpg
      • ...

Files cannot be read even after trying to access them directly using their absolute path. e.g "drive/photos.library/images/M5Q2Z2JKV3WTK.info/wallpaper.jpg". I was under the impression that this should not take long as is accessing the file directly and not listing or reading metadata form all the other files or dirs located in the images/ folder

The finder app just hangs/loading and becomes unresponsive, until I restart my system.

Run the command 'rclone version' and share the full output of the command.

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount --stats 1m -vv --vfs-cache-mode=off --local-unicode-normalization -o modules=iconv,from_code=UTF-8,to_code=UTF-8 ghub: ~/drive

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[gdrive]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = XXX
team_drive =

[ghub]
type = crypt
remote = gdrive:/Backups
password = XXX
password2 = XXX

A log from the command that you were trying to run with the -vv flag

...
...
...
2025/03/13 23:18:56 DEBUG : drive/photos.library/images/M5JN655R7XHZV.info: forgetting directory cache
2025/03/13 23:18:56 DEBUG : drive/photos.library/images/M5SNYS9DYW8WT.info: forgetting directory cache
2025/03/13 23:18:56 DEBUG : drive/photos.library/images/M5TM1JB4SLQQ8.info: forgetting directory cache
2025/03/13 23:18:56 DEBUG : drive/photos.library/images/M5Q2Z2JY50VJQ.info: forgetting directory cache
2025/03/13 23:18:56 DEBUG : drive/photos.library/images/M6LOHXD6V1HN9.info: forgetting directory cache
2025/03/13 23:18:56 DEBUG : drive/photos.library/images/M6NTMS19V3BXX.info: forgetting directory cache
2025/03/13 23:18:56 DEBUG : drive/photos.library/images/M6NTMS2ZTPVBZ.info: forgetting directory cache
2025/03/13 23:18:56 DEBUG : drive/photos.library: forgetting directory cache
2025/03/13 23:18:56 DEBUG : drive/photos.library/images: forgetting directory cache
2025/03/13 23:18:56 DEBUG : drive: forgetting directory cache

Sometimes I get this

2025/03/13 21:04:12 DEBUG : /drive/photos.library/images/M5Q2Z2JKV3WTK.info/wallpaper.jpg: Getattr: fh=0xFFFFFFFFFFFFFFFF
2025/03/13 21:04:12 DEBUG : /drive/photos.library/images/M5Q2Z2JKV3WTK.info/wallpaper.jpg: >Getattr: errc=-2
2

Update:

With the previous command the file would simply not open and keep loading for ever!

Now, I modified the command this time.
I increased the buffer-size to 128M and set vfs-cache-mode to minimal and now I'm able to read the files, but it still takes around 30 minutes loading until I can finally read the file directly. It looks like all files are being indexed??? because after 30 minutes of loading I can access any other random file in from any of the folders and it loads instantly.

Here is the new command used.

rclone mount --fast-list --dir-cache-time=12h --poll-interval=15m --buffer-size=128M --stats 1m -vv --vfs-cache-mode=minimal ghub: ~/ghub

Any suggestions how can I improve the command above so I don't have to wait that long when trying to read files.

note: having the remote always mounted is not an option for me, as I only mount the remote drive when I need to read the files.

Thank you in advance for any suggestions!

3

welcome to the forum,

does not work on mount command.

rclone, on its own, does not do that.
rclone simply provides the data that other applications ask for, such as finder, anti-virus and so on.
if finder tries to index the files and create thumbnails, that can be slow.

if you open a command prompt and try rclone ls ~/ghub:, what happens?

check out my summary of vfs caches
the dir cache is stored in memory, and once the mount is killed, the contents are lost.

might test --vfs-fast-fingerprint --buffer-size=0 --vfs-cache-mode=full

that might not be correct. the rclone debug log would provide the answer

4

Hello,

thank you for the suggestions and for helping understand the difference between dir-cache and vfs-cache. I tested with --vfs-fast-fingerprint --buffer-size=0 and I can access the files much faster now, I will continue testing with other values but is better now. Also, I try not to use --vfs-cache-mode=full, as I don't really want to cache the content of the files. This post (dir cache) has been really helpful too.

5

keep in mind that the next time you run the mount command, if the files are not in the cache, rclone will have to download them again.
the good news is that rclone uses chunked-reading, does not download the entire file.

to keep the file cache small, can use a short value for --vfs-cache-max-age
and/or run rclone in a simple script, something like

  1. rclone mount ...
  2. delete the files in the cache
1 Like