How to speed-up opening folders/traversing directories?

Help and Support
1

What is the problem you are having with rclone?

Slow opening folders, freezes desktop apps (KDE Dolphin for example)

What is your rclone version (output from rclone version)

rclone v1.54.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu, KDE Plasma 64 bit

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount gdrive: /home/user1/gdrive-rclone --cache-dir /tonk/cache-user1/gdrive-rclone --vfs-cache-mode full --vfs-cache-poll-interval 20m --vfs-cache-max-age 8760h --vfs-cache-max-size 1024G --attr-timeout 8700h --dir-cache-time 8760h --poll-interval 30s --multi-thread-streams 0

The rclone config contents with secrets removed.

[gdrive]
type = drive
client_id = 
client_secret =
token = {"secret"}
root_folder_id = secret?

A log from the command with the -vv flag

is this necessary in this case?
2

2h of reading the manual later I think if the above command be run with --rc (flag enabling remote control) then running rclone rc vfs/refresh -v --fast-list recursive=true will precache all directories making the traversals much faster. Still need to test it though and find a way to integrate it with systemd

what led me to this:

wish it was default behaviour though...

3

hello and welcome to the forum,

for systemd, this is the goto guide, from fellow rcloner @Animosity022
note that it will precache the mount,
systemd

1 Like
4

thanks a lot, I see that it doesn't use the --fast-list flag for precaching and also added _async=true. This may be a separate question, but is there anything inappropriate with using fast-list (also Google Drive)?

5

--fast-list does nothing on a mount.

1 Like
6

for some reason systemd fails when running the service with this line
ExecStartPost = rclone rc vfs/refresh recursive=true _async=true --rc-addr :5572 --rc-user user --rc-pass password
The unit rclone-gdrive-mount.service has entered the 'failed' state with result 'exit-code'.

not sure why, it is the same as in the example above

7

sorry, that is not correct,
for example, you added extra flags, changed some flags and the full path is missing for rclone executable.

best to use the settings from that systemd as is, no changes.
once that is working, then if needed, tweak the settings.

9

Okay, I got it to work after adding those lines to the systemd service file service section (from the example above). Thanks a lot for linking it.

Type=notify
Environment=RCLONE_CONFIG=/path/to/the/rclone.conf
KillMode=none
RestartSec=5

I assume RestartSec was needed to wait longer for the async job to maybe like wait for a response from somewhere or maybe KillMode was set to kill it at some point.
Not sure; hopefully, it will speed up listing of the directories now.

10

I guess there is no way to view the progress of the rclone rc vfs/refresh scan?

11

you could add --progress the refresh command.

1 Like
12

perfect, thanks again. Is it normal that after 2 minutes the transfer still stays at 0 Bytes?

Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
13

yes that is normal, as Bytes/s is when using move/sync/copy operations, for the file contents thare are transferred

whereas the refresh is reading the folder names, file names and metadata such as mod-time.
which is not considered data.

14

that makes sense, it was still worth trying, because it showed that the rclone rc vfs/refresh timed out after 5 minutes :open_mouth: rclone documentation mentioned that's a default (Documentation) and can be overwritten with the --timeout flag added to the rclone rc vfs/refresh command

15

Do you know if/where this precached data is saved? It would be great to confirm if it suceeded.

16

at this point, take a few minutes to read the documenation.

https://rclone.org/commands/rclone_mount/#vfs-file-caching

1 Like
17

thanks, just one more question. Does this vfs/refresh cache persist across reboots? Is it stored on disk or any part of it or does rclone need to run full scan each time again?

18

each time you rclone mount, you have to perform the refresh.

19

Hmm, I'm not sure if it may be google just throttling my account, but with over 5TB of data stored it can take hours to refresh. Is there really no way to cache this to disk? Isn't that what windows' Google Drive desktop client does by default?

20

@Animosity022 would know

21

I think you are talking about two different things here.

vfs/refresh is just for metadata so directory and file structure.

Thats impacted by dir-cache-time and on a mount since Google Drive is a polling remote, you can have that value high.

I use that to precache my directory / file structure in memory so when my machine boots and starts the rclone mount, the directory listing is fast.

So when I run a directory listing, it comes back instantly.

felix@gemini:/GD$ time ls -alR | wc -l
86201

real	0m1.474s
user	0m0.206s
sys	0m0.223s

The dir cache is lost when you reboot or restart the mount and the 'priming' of the mount needs to be done which is why I have it as part of my startup command to just make things faster for browsing. The time is takes is not dependent on the size but more so on the number of directories/files you have and how it's laid out.

Dir-cache-time has nothing directly to do with file based caching on disk with vfs-cache-mode full. That's for storing the actual data on the files and when you mount something with rclone, it only grabs when you ask it to grab. So if you read a file, it gets what you/the application has requested. The vfs-cache-mode data is persistent and sticks around on reboots.