How to get full directory tree cached on first mount?

Hello,

If I mount a remote for the first time, and this mount has thousands of files and folders, I see that browsing through the mount can be very slow because rclone has to get the directory contents on each folder load while I'm browsing. So just going through a few folders it has to pause and wait to load them with thousands of files. However, once these folder contents are cached, they load instantly the 2nd time around.

I want to know if there is some way I can tell rclone to mount this remote and cache the entire directory tree at that exact time that I am mounting it for the first time. This way browsing through the mount will be very quick and not laggy.

I hope I explained it good.

Thank you.

I know exactly what you mean. This bothered me too initially before i had learned much about rclone.
Thankfully it is fixable in most cases, and I can help you solve this via "precaching". This will effectively allow the mount to respond as fast as a normal HDD, and also be fully searched in seconds rather than 10-30 minutes. I can probably even give you some pre-made script to use (or use as a template at least).

But first in order to help you I need to know:

  • Your rclone version (run command: rclone version to check)
  • What OS (operating system) you use
  • What backend (ie. service provider) you use. For example - Google Drive.

Thank for replying. Yes that's exactly what I want, to precaching right from the beginning so the mount responds fast for searching files and going through folders.

Here is the answer to your questions:

rclone v1.48.0

  • os/arch: windows/amd64
  • go version: go1.12.3

Running on Windows 10 64-bit.

Google Drive backend.

Thanks if you can help.

welcome to the forum,
i am the assistant of the honorable thestigma,

please update to the latest version of rclone, as the the honorable, thestigma will ask that of you...

That setup is perfect.

Firstly, the most important part is that have a backend that supports polling. Without this would be very hard to have any sort of practical preaching while also being able to detect changes on the drive.

Secondly, my primary OS is also Windows, so you can just use my scripts with very little modification necessary.

So in short our process will be:

  • Set required flags to keep cache information long-term
  • As part of the mounting script, start a process that precaches the disk (I will share script)

These are the relevant flags:
--attr-timeout 8700h ^
(how long to keep information about files, like size, modtime ect.)
--dir-cache-time 8760h ^
(how long to keep directory information, list of files in a folder)
--poll-interval 30s ^
(how often to ask the Google server is anything on the drive has changed and needs to be also changed in the cache)
We will simply append these flags to your regular mount command.

So now I will share some batch scripts that you can modify to your need. Here is a simple example of how to mount a drive, and also call the precache script at the same time:
Note that you don't need to use all, or any the flags I do (except the 3 mentioned) - but I will leave them in to illustrate and you can ask more about them if interested.

Mountscript.bat

@echo off

::Variables
set driveletter=X
set remotename=TD1Crypt
set RCport=5572

::Window size (bootup)
mode con: cols=60 lines=20
title %remotename%-BOOTUP
cd c:\rclone

echo Spawning mount for %driveletter%: (new window)
echo.

:: Now let's mount the drive with some options
start /min "%remotename%" rclone mount %remotename%: %driveletter%: ^
--rc ^
--rc-web-gui ^
--rc-addr localhost:%RCport% ^
--rc-user test ^
--rc-pass test ^
--cache-dir "E:\!---Rclone_VFS_Cache" ^
--vfs-cache-mode writes ^
--vfs-cache-poll-interval 20m ^
--vfs-cache-max-age 8760h ^
--vfs-cache-max-size 1024G ^
--attr-timeout 8700h ^
--dir-cache-time 8760h ^
--poll-interval 30s ^
--allow-other ^
--multi-thread-streams 0

::Start cache warmup
call VFSCacheWarmup.bat %driveletter% %RCport%

The only thing you should really need to change here is:
set remotename=TD1Crypt
(use the name of your own remote here)
cd c:\rclone
(change this if you have rclone somewhere else)
--cache-dir "E:\!---Rclone_VFS_Cache" ^
(assuming you want to use write-caching at all, then set a folder here for temporary files)

As you might notice, at the end we also call another script, so you will need to make that too (in this example you would put it in the same fodler as the other script to make it work):

VFSCacheWarmup.bat

@echo off

::Variables
set driveletter=%1
set RCport=%2

:: Check that the folder is valid, otherwise wait until it is
echo Waiting for %driveletter%: to be ready ...

:LOOP1
vol %driveletter%: >nul 2>nul
if errorlevel 1 (
    echo " "| set /p dummyName=.
	timeout /t 1 > nul
	goto LOOP1
) else (
	echo.
    echo Drive %driveletter%: OK!,
)

echo.
echo Warming up cache...
echo This may take a few minutes to complete
echo You can use the cloud-drive normally while this runs
echo.

echo Awaiting completion-message from RC...
rclone rc vfs/refresh -v recursive=true --rc-addr localhost:%RCport% --rc-user test --rc-pass test > nul

::If exit code of above command was anything but normal (indicating an error) - trigger alarm and pause for diagnosis.
if not %ERRORLEVEL% equ 0 (
	echo Something went wrong during precaching. Is the mount still running? press any key to continue.
	pause
) else (
	echo Cache warmup for %driveletter%: OK!
)

echo Closing window in 5 seconds...
timeout /t 5 >nul

You should not need to change anything in this file. Just make sure to name it as suggested and put it in the same folder as the other script.

Finally - to run the whole thing - just double-click the mountscript. You might also perhaps want to run this automatically as soon as you log in ect. but that is another topic for another day :slight_smile:

Let me know if you have trouble with any of this and I will get you settled.

5 Likes

Yes, that wouldn't hurt. It never does :slight_smile:
Latest version is 1.49.5 (1.50 coming real soon).

@aidyn
Oh, and I need to also make one (very small) point about data integrity.
With this setup (specifically using a long attribute timeout) there is a rare and very spesific set of circumstances under which a file could be become corrupted. Most likely this will be irrelevant for you, but I have to make sure you understand so that it will not happen by accident.

If a file's size is altered by some other source outside the mount (another machine or some other rclone instance)
and
before 30 seconds have passed (or however long the polling interval is set for) - You also modify that same file though the mount (writes outside of mount is not a problem).
and
You perform one of the few write commands that will not be able to detect the wrong size
then
a corrupt file (unexpected end of file) could potentially occur.

For most people this is a non-issue because they are usually single-users. But if your drive is regularly written to by many people other than yourself then you might need to think about this and maybe upload via webinterface or script rather than the mount. But even in such cases it's a very small window within where the problem could occur - and you can even lower polling down to 10s if you wanted to do reduce this even further...

Just be aware of this is all im trying to say :slight_smile:

1 Like

Well this sounds perfect! I'm going to give it a try when I'm home later and I will let you know how it went for me. Thank you for this great script! :grinning:

Thanks for that info regarding corruption. In my situation with this specific mount is strictly for reading files and no writing at all. I even use the --read-only flag on this mount. I also always try to make sure I access any remote from a single place and not to have them open in different places. So hopefully it reduces any of the risks for corruption. I would be very careful if I am writing from 2 locations.

Ahh yes, I'm always lazy about updates. It's time to do that.

Under those circumstances (read-only) there is exactly 0% risk of this problem.
And as mentioned - any uploads outside of the mount are also completely safe as they never rely on the cache anyway.

Regarding upgrading it can't be simpler. Literally just replace the exe with the new exe. Tada! All done :stuck_out_tongue: All configurations will keep working as before.

Good to know!

Yeah, I've really got no excuse for that LOL. :upside_down_face:

@aidyn,
the problem of corruption is something to be concerned about.
perhaps here with a read-only mount, that is not such a problem.

if you find yourself, copying large amounts of data to the cloud, there is a chance local files will be modified or deleted during the long upload.
i sync large files created by veeam backup and i ran into corruption problems.

if you find that is happening to you, please see my rclone wiki

1 Like

I don't know about "concerned" as it's not something that will happen randomly, but it's definitely something to be aware of. That is why I explicitly state the (small) risk anytime I help anyone use precaching.

On a read-only mount it's a non-issue.

But on any "unpredictable external writes" setup, such as a multi-user (multi-writer) drive the risk should not be ignored - even if the time-frame for errors to happen is small. In that case it would typically be best to avoid uploading via mount completely and use an alternative - of which there are several good methods.

  • An upload script (which is even better if combined with union or mergerFS as you can pretend like you are writing to the mount and get all the convenience (and application compatibility) out of it, but in reality the actual uploading happens outside the mount and is thus not at risk of using old cache data. This is what I prefer myself (and will be even better after the union updates that hopefully will be coming soon). not only just as convenient but also more efficient and super-fast as all "uploading" from the users perspective happens at local HDD speeds. Best of all worlds IMO, but needs a little bit of setup and understanding.
  • Uploading via web-interface. This will just work out of the box without any setup and is now pretty intuitive to do thanks to the new GUI. Drag&drop :slight_smile:

I've yet to run into any corruption from this. Now I mainly use the mount-redirected to upload script method, but I ran many months before that using a mix of mount (for upload) and scripts - just being thoughtful about what I was doing and not triggering the required circumstances. That "just be careful" approach is not something I recommend obviously as everyone has bad days where we don't pay enough attention to what we are doing - but I think it illustrates that it's not something that will happen easily or randomly.

1 Like

@asdffdsa Thanks for the wiki link. I do take great caution to prevent problems like that from happening and I only ever write from one rclone instance at a time.

@thestigma I gotta say, your script works real nice! :grinning: Everything is so fast now, performing as if it's local! Its exactly what I wanted and hoped for. You are a true rclone master. Many thanks.

1 Like

Aaargh! Flattery - my only weakness! :wink:
Thank you for the kind words. Just happy to help improve your experience :slight_smile:

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.