Speeding up directory tree navigation on a mounted network drive

Help and Support
1

Hi guys! :wave:

I'm trying to facilitate this workflow:

  1. browse mounted network drive to find some project
  2. drag a local copy of it to my SSD
  3. edit local copy
  4. drag local copy back to network drive (overwriting original)

To move files between local / network drives I'll simply drag and drop using file explorer. No syncing or remote editing required. I'm a simple man.

Problem

Browsing the mounted network drive is slow, sometimes making my system unresponsive.

I learned from @thestigma in this answer that I can pre-cache the directory tree after mounting to speed things up. However, in my implementation (below) I'm not sure anything is actually being cached as browsing remains slow and --cache-dir is always empty.

Network

60 Mbps download / 18 Mbps upload.

Version

rclone v1.59.1
- os/version: Microsoft Windows 10 Home 21H2 (64 bit)
- os/kernel: 10.0.19044.2130 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.18.5
- go/linking: static
- go/tags: cmount

winfsp-1.11.22176

Cloud Provider

Wasabi

Config

[wasabi]
type = s3
provider = Wasabi
env_auth = false
access_key_id = XXXXXX...
secret_access_key = XXXXXX...
region = eu-west-1
endpoint = s3.eu-west-1.wasabisys.com
location_constraint = 
acl = private

Startup Script (.bat)

I'm trying to mount the network drive and then pre-cache the directory tree.

@echo off

:: Set variables.
set driveletter=W
set remotename=wasabi
set port=5572

title %remotename%-boot
cd D:\Prizi\Downloads\media\applications\rclone-v1.59.1-windows-amd64

:: Mount network drive.
echo Mounting %driveletter%:
start /min "%remotename%" rclone mount %remotename%: %driveletter%: ^
-vv ^
--rc ^
--rc-addr localhost:%port% ^
--rc-user test ^
--rc-pass test ^
--cache-dir "%cd%\vfs-cache" ^
--vfs-cache-mode writes ^
--vfs-cache-max-age 8760h ^
--vfs-cache-max-size 1024G ^
--attr-timeout 8700h ^
--dir-cache-time 8760h ^
--multi-thread-streams 0

:: Check if mount successful.
echo Checking if %driveletter%: is ready.
:LOOP1
vol %driveletter%: >nul 2>nul
if errorlevel 1 (
  echo " "| set /p dummyName=.
  timeout /t 1 > nul
  goto LOOP1
) else (
    echo %driveletter%: is ready to use!
)

:: Pre-cache the full directory tree.
echo Refreshing cache. This may take a few minutes.
rclone rc vfs/refresh -vv recursive=true --rc-addr localhost:%port% --rc-user test --rc-pass test > nul

:: If exit code is non-zero (error) pause for diagnosis.
if not %ERRORLEVEL% equ 0 (
  echo Pre-cache failed. Is the mount still running? press any key to continue.
  pause
) else (
  echo Pre-cache for %driveletter%: OK!
)

:: Keep the window open.
cmd /k

Output Log (Window 01)

Mounting W:
Checking if W: is ready.
.W: is ready to use!
Refreshing cache. This may take a few minutes.
2022/10/30 17:14:58 DEBUG : rclone: Version "v1.59.1" starting with parameters ["rclone" "rc" "vfs/refresh" "-vv" "recursive=true" "--rc-addr" "localhost:5572" "--rc-user" "test" "--rc-pass" "test"]
2022/10/30 17:15:00 DEBUG : 4 go routines active
Precache for W: OK!
D:\Prizi\Downloads\media\applications\rclone-v1.59.1-windows-amd64>

Output Log (Window 02)

2022/10/30 17:14:57 DEBUG : rclone: Version "v1.59.1" starting with parameters ["rclone" "mount" "wasabi:" "W:" "-vv" "--rc" "--rc-addr" "localhost:5572" "--rc-user" "test" "--rc-pass" "test" "--cache-dir" "D:\\Prizi\\Downloads\\media\\applications\\rclone-v1.59.1-windows-amd64\\vfs-cache" "--vfs-cache-mode" "writes" "--vfs-cache-max-age" "8760h" "--vfs-cache-max-size" "1024G" "--attr-timeout" "8700h" "--dir-cache-time" "8760h" "--multi-thread-streams" "0"]
2022/10/30 17:14:57 INFO  : Using --user test --pass XXXX as authenticated user
2022/10/30 17:14:57 NOTICE: Serving remote control on http://localhost:5572/
2022/10/30 17:14:57 DEBUG : Creating backend with remote "wasabi:"
2022/10/30 17:14:57 DEBUG : Using config file from "C:\\Users\\Prizi\\AppData\\Roaming\\rclone\\rclone.conf"
2022/10/30 17:14:57 INFO  : S3 root: poll-interval is not supported by this remote
2022/10/30 17:14:57 DEBUG : vfs cache: root is "D:\\Prizi\\Downloads\\media\\applications\\rclone-v1.59.1-windows-amd64\\vfs-cache"
2022/10/30 17:14:57 DEBUG : vfs cache: data root is "\\\\?\\D:\\Prizi\\Downloads\\media\\applications\\rclone-v1.59.1-windows-amd64\\vfs-cache\\vfs\\wasabi"
2022/10/30 17:14:57 DEBUG : vfs cache: metadata root is "\\\\?\\D:\\Prizi\\Downloads\\media\\applications\\rclone-v1.59.1-windows-amd64\\vfs-cache\\vfsMeta\\wasabi"
2022/10/30 17:14:57 DEBUG : Creating backend with remote "D:/Prizi/Downloads/media/applications/rclone-v1.59.1-windows-amd64/vfs-cache/vfs/wasabi/"
2022/10/30 17:14:57 DEBUG : fs cache: renaming cache item "D:/Prizi/Downloads/media/applications/rclone-v1.59.1-windows-amd64/vfs-cache/vfs/wasabi/" to be canonical "//?/D:/Prizi/Downloads/media/applications/rclone-v1.59.1-windows-amd64/vfs-cache/vfs/wasabi/"
2022/10/30 17:14:57 DEBUG : Creating backend with remote "D:/Prizi/Downloads/media/applications/rclone-v1.59.1-windows-amd64/vfs-cache/vfsMeta/wasabi/"
2022/10/30 17:14:57 DEBUG : fs cache: renaming cache item "D:/Prizi/Downloads/media/applications/rclone-v1.59.1-windows-amd64/vfs-cache/vfsMeta/wasabi/" to be canonical "//?/D:/Prizi/Downloads/media/applications/rclone-v1.59.1-windows-amd64/vfs-cache/vfsMeta/wasabi/"
2022/10/30 17:14:57 DEBUG : Network mode mounting is disabled
2022/10/30 17:14:57 DEBUG : Mounting on "W:" ("wasabi")
2022/10/30 17:14:57 DEBUG : S3 root: Mounting with options: ["-o" "attr_timeout=3.132e+07" "-o" "uid=-1" "-o" "gid=-1" "--FileSystemName=rclone" "-o" "volname=wasabi"]
2022/10/30 17:14:57 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/10/30 17:14:57 DEBUG : S3 root: Init:
2022/10/30 17:14:57 DEBUG : S3 root: >Init:
2022/10/30 17:14:57 DEBUG : /: Statfs:
2022/10/30 17:14:57 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2022/10/30 17:14:57 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/10/30 17:14:57 DEBUG : /: >Getattr: errc=0
2022/10/30 17:14:57 DEBUG : /: Readlink:
2022/10/30 17:14:57 DEBUG : /: >Readlink: linkPath="", errc=-40
2022/10/30 17:14:57 DEBUG : /: Getxattr: name="non-existant-a11ec902d22f4ec49003af15282d3b00"
2022/10/30 17:14:57 DEBUG : /: >Getxattr: errc=-40, value=""
The service rclone has been started.
2022/10/30 17:14:57 DEBUG : /: Statfs:
2022/10/30 17:14:57 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2022/10/30 17:14:57 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/10/30 17:14:57 DEBUG : /: >Getattr: errc=0
2022/10/30 17:14:57 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/10/30 17:14:57 DEBUG : /: >Getattr: errc=0
2022/10/30 17:14:57 DEBUG : /: Opendir:
2022/10/30 17:14:57 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/10/30 17:14:57 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2022/10/30 17:14:57 DEBUG : /: >Opendir: errc=0, fh=0x0
2022/10/30 17:14:57 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/10/30 17:14:57 DEBUG : /: >Getattr: errc=0
2022/10/30 17:14:57 DEBUG : /: Releasedir: fh=0x0
2022/10/30 17:14:57 DEBUG : /: >Releasedir: errc=0
2022/10/30 17:14:57 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/10/30 17:14:57 DEBUG : /: >Getattr: errc=0
2022/10/30 17:14:57 DEBUG : /: Opendir:
2022/10/30 17:14:57 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/10/30 17:14:57 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2022/10/30 17:14:57 DEBUG : /: >Opendir: errc=0, fh=0x0
2022/10/30 17:14:57 DEBUG : /: Releasedir: fh=0x0
2022/10/30 17:14:57 DEBUG : /: >Releasedir: errc=0
2022/10/30 17:14:57 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/10/30 17:14:57 DEBUG : /: >Getattr: errc=0
2022/10/30 17:14:57 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/10/30 17:14:57 DEBUG : /: >Getattr: errc=0
2022/10/30 17:14:57 DEBUG : /: Opendir:
2022/10/30 17:14:57 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/10/30 17:14:57 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2022/10/30 17:14:57 DEBUG : /: >Opendir: errc=0, fh=0x0
2022/10/30 17:14:57 DEBUG : /: Releasedir: fh=0x0
2022/10/30 17:14:57 DEBUG : /: >Releasedir: errc=0
2022/10/30 17:14:58 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/10/30 17:14:58 DEBUG : /: >Getattr: errc=0
2022/10/30 17:14:58 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/10/30 17:14:58 DEBUG : /: >Getattr: errc=0
2022/10/30 17:14:58 DEBUG : /: Opendir:
2022/10/30 17:14:58 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/10/30 17:14:58 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2022/10/30 17:14:58 DEBUG : /: >Opendir: errc=0, fh=0x0
2022/10/30 17:14:58 DEBUG : /: Releasedir: fh=0x0
2022/10/30 17:14:58 DEBUG : /: >Releasedir: errc=0
2022/10/30 17:14:58 DEBUG : rc: "vfs/refresh": with parameters map[recursive:true]
2022/10/30 17:14:58 DEBUG : : Reading directory tree
2022/10/30 17:15:00 DEBUG : : Reading directory tree done in 2.076883s
2022/10/30 17:15:00 DEBUG : rc: "vfs/refresh": reply map[result:map[:OK]]: <nil>
2022/10/30 17:15:00 DEBUG : /autorun.inf: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/10/30 17:15:00 DEBUG : /autorun.inf: >Getattr: errc=-2
2022/10/30 17:15:00 DEBUG : /autorun.inf: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/10/30 17:15:00 DEBUG : /autorun.inf: >Getattr: errc=-2
2022/10/30 17:15:00 DEBUG : /autorun.inf: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/10/30 17:15:00 DEBUG : /autorun.inf: >Getattr: errc=-2
2022/10/30 17:15:00 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/10/30 17:15:00 DEBUG : /: >Getattr: errc=0
2022/10/30 17:15:00 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/10/30 17:15:00 DEBUG : /: >Getattr: errc=0
2022/10/30 17:15:00 DEBUG : /: Opendir:
2022/10/30 17:15:00 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/10/30 17:15:00 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2022/10/30 17:15:00 DEBUG : /: >Opendir: errc=0, fh=0x0
2022/10/30 17:15:00 DEBUG : /: Releasedir: fh=0x0
2022/10/30 17:15:00 DEBUG : /: >Releasedir: errc=0
2022/10/30 17:15:00 DEBUG : /AutoRun.inf: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/10/30 17:15:00 DEBUG : /AutoRun.inf: >Getattr: errc=-2
2022/10/30 17:15:00 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/10/30 17:15:00 DEBUG : /: >Getattr: errc=0
2022/10/30 17:15:00 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/10/30 17:15:00 DEBUG : /: >Getattr: errc=0
2022/10/30 17:15:00 DEBUG : /: Opendir:
2022/10/30 17:15:00 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/10/30 17:15:00 DEBUG : /: >Getattr: errc=0
2022/10/30 17:15:00 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/10/30 17:15:00 DEBUG : /: >Getattr: errc=0
2022/10/30 17:15:00 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/10/30 17:15:00 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2022/10/30 17:15:00 DEBUG : /: >Opendir: errc=0, fh=0x0
2022/10/30 17:15:00 DEBUG : /: Opendir:
2022/10/30 17:15:00 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/10/30 17:15:00 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2022/10/30 17:15:00 DEBUG : /: >Opendir: errc=0, fh=0x1
2022/10/30 17:15:00 DEBUG : /: Statfs:
2022/10/30 17:15:00 DEBUG : /: Releasedir: fh=0x1
2022/10/30 17:15:00 DEBUG : /: >Releasedir: errc=0
2022/10/30 17:15:00 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2022/10/30 17:15:00 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/10/30 17:15:00 DEBUG : /: >Getattr: errc=0
2022/10/30 17:15:00 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/10/30 17:15:00 DEBUG : /: >Getattr: errc=0
2022/10/30 17:15:00 DEBUG : /: Releasedir: fh=0x0
2022/10/30 17:15:00 DEBUG : /: >Releasedir: errc=0
2022/10/30 17:15:00 DEBUG : /: Opendir:
2022/10/30 17:15:00 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/10/30 17:15:00 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2022/10/30 17:15:00 DEBUG : /: >Opendir: errc=0, fh=0x0
2022/10/30 17:15:00 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/10/30 17:15:00 DEBUG : /: >Getattr: errc=0
2022/10/30 17:15:00 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/10/30 17:15:00 DEBUG : /: >Getattr: errc=0
2022/10/30 17:15:00 DEBUG : /: Opendir:
2022/10/30 17:15:00 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/10/30 17:15:00 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2022/10/30 17:15:00 DEBUG : /: >Opendir: errc=0, fh=0x1
2022/10/30 17:15:57 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/10/30 17:16:57 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/10/30 17:17:57 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/10/30 17:18:57 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/10/30 17:19:57 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/10/30 17:20:57 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/10/30 17:21:57 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/10/30 17:22:57 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/10/30 17:23:57 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/10/30 17:24:57 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/10/30 17:25:57 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/10/30 17:26:57 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/10/30 17:27:57 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/10/30 17:28:57 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/10/30 17:29:57 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2022/10/30 17:30:57 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2

that is for the vfs directory cache. caches the directory names, file names and some metadata such as modtime.
that should speed up navigating but often does not really matter too much.

to better understand that two caches, vfs file cache and vfs dir cache.might check out
https://forum.rclone.org/t/status-about-using-rclone-for-music-storage-playback-in-2021-access-times-improved/27648/34

wasabi is as fast as it gets, your mount should perform well, as does my wasabi mounts.

any reason to use that?

as for using windows explorer, it is terrible to use.
need to disable previews, content, tiles, etc - as that forces rclone to download the contents of files.
so try --network-mode and --vfs-cache-mode=full

what is the result of a speed test including ping?
it could be that high latency is slowing rclone down.

1 Like
3

Very helpful, thanks!

Leftover bits from thestigma's script. Still trying to figure out what I can safely delete! :melting_face:

100%. I saw you mention this elsewhere. Maybe I underestimate just how much it affects slowness.

60 Mbps down / 18 Mbps up. Ping is reliably 15ms or less.

4

not sure if tried any of my suggestions, did none of them help?

5

Just tried, it works! Thank you so much! :heart_eyes:

--network-mode made directories open faster!
--vfs-cache-mode=full made files open faster!

If you don't mind, please could you help me understand why --vfs-cache-mode=full makes opening remote files quicker? For exmaple, I tried to preview a small 50KB audio file, but with flags like --read-only or --vfs-cache-mode=writes it takes at least 5–10 seconds to play. However, with --vfs-cache-mode=full it only takes about 1 second to play.

Will --vfs-cache-mode=full wear out my local disks quickly if I am re-mounting every day?

1 Like
6

well, i would not worry about it for at least two reasons.
--- when using full, the cache uses chunked reading and sparse files.
let's say you have a 60GiB video file of length 60 minutes and you want to stream with the mount.
if you watch just one minute of video, then rclone will only download 1GiB`

---- might say that rclone mount is lazy about removing files or chunks of files from the cache.
as controlled by

--vfs-cache-max-age 8760h
--vfs-cache-max-size 1024G

and this could be tested.
--- upload a media file to wasabi
--- stream the media file for a short period of time, 30 secs
--- kill the mount
--- figure out the size of the vfs file cache
--- restart the mount
--- notice that the size of the vfs file cache did not change.

and notice here the size of the file.
to windows explorer, it might appear as 1.03GB, tho on local storage, using sparse file, just 4.37MB as size on disk.
image

1 Like
7

Hi. It looks as if the precaching is working allright, but a common issue with windows explorer is that you may be using a view-mode that shows other info than the basic attributes. the precache only fetches those basic attributes, so if you have a viewmode that tries to show image previews, show media length ect. then it will need to open and check every file anyway - which can defeat the whole purpose.

I would recommend using "details" mode. If you go into properties on the mounted drive and customize it to "optimize" for documents then it will use this view-mode for the whole drive instead of having to set it in every folder:

image
image717×967 17.9 KB

When the precache works and is used with a correct view mode then it should respond like an SSD (actually faster, since you will be browsing from RAM)

cache-mode-full is a much more advanced (and recent) cache system. But the MAIN difference from "writes" is that it also caches some of the downloaded data. I believe this includes some listing metadata also, so that probably masks a lot of these issues. I would generally recommend to use "full" mode now because Nick made it really great.
full cache won't cache anything just from mounting. It will only cache when accessing data (listing metadata, or downloading files).
Wear on a spinning HDD is negligible. The don't generally wear out from writes but from other things. Wear on a modern SSD is usually not an issue, but I guess it depends on how heavy your usage pattern is.

Hope that helps :slight_smile:

PS: I wrote the precache for use with Gdrive. It should work fine with wasabi too, but I think Wasabi is inherently much more responsive than Gdrive in many ways, so it may not need it as badly as Gdrive does :slight_smile:

1 Like
8

hey, i like that, has the makings of a rclone slogan :wink:

1 Like
9

I forgot about this. This effectively does the same as what I described and is a simpler way to achieve that effect.
Although I suppose doing it my way does let you have selective control over a handful of folders that can use other view-modes set individually if needed. In those few folders where you can afford to be slow to see previews or something like that.

11

Very much. Thank you (and @asdffdsa) for your advice.

Now I have a network drive that feels just like a local drive. Cool! :metal:

Here's my working startup script (inspired by you @thestigma) in case it can help anyone else:

@echo off

:: Variables.
set driveletter=W
set remotename=wasabi
set port=5572

:: Window title.
title %remotename%-boot

:: Location of rclone binary.
cd D:\Prizi\Downloads\media\applications\rclone-v1.60.0-windows-amd64

:: Mount network drive.
echo Mounting %driveletter%:
start /min "%remotename%" rclone mount %remotename%: %driveletter%: ^
-vv ^
--rc ^
--rc-addr localhost:%port% ^
--rc-user test ^
--rc-pass test ^
--network-mode ^
--vfs-cache-mode full ^
--cache-dir "%cd%\cache"

:: Check if mount successful.
echo Checking if %driveletter%: is ready...
:LOOP1
vol %driveletter%: >nul 2>nul
if errorlevel 1 (
  echo " "| set /p dummyName=.
  timeout /t 1 > nul
  goto LOOP1
) else (
  echo .
  echo %driveletter%: is ready to use!
)

:: Pre-cache the full directory tree.
echo Pre-caching. This may take a few minutes.
rclone rc vfs/refresh -vv recursive=true --rc-addr localhost:%port% --rc-user test --rc-pass test > nul

:: If exit code is non-zero (error) pause for diagnosis.
if not %ERRORLEVEL% equ 0 (
  echo Pre-cache failed. Is the mount still running? Press any key to continue.
  pause
) else (
  echo Pre-cache for %driveletter%: OK!
)

:: Keep the window open.
cmd /k
13

i rely on wasabi as a destination for backups, tho never using rclone mount

keep in mind that wasabi has a 180 day retention period on files.
depending on how you use the rclone mount, that can get expensive.
tho that can be reduced to 90 days.

14

Like cold storage that is scheduled to backup at a set interval?

I could be wrong but I think Wasabi's default retention period is now 90 days for pay-as-you-go customers, see here. But it's an important point regardless.

I plan to use Wasabi as an everyday working drive. I'll need to browse/preview files on the remote to find what I'm looking for, but then I'll copy those files to my local SSD while I work. I'm currently paying $5.99 / month for 1TB and hope not to exceed that. I don't expect my Wasabi bucket to grow beyond 250GB / 200K files / 20K folders, so that should leave me some breathing room. What do you think?

15

i keep recent backups, veeam, 7zip, etc in wasabi.
move older backup to aws s3 deep glacier.

looks that way now.
in the past, it was 180 days. if i uploaded veeam backup files, make a request to wasabi, then wasabi would change the retention period to 90 days for all files.

well, wasabi has very fast api calls, that would be helpful with rclone mounts.

as an experiment some time ago,
i did a veeam instant recovery where the backup files, using a rclone mount, were in wasabi.
with a fiber optic 1Gbps connection, i was surprised about how well that worked.

on the other hand,
if you are using the mount to manually download/edit/upload files,
then could use most any provider, perhaps, cheaper provider.
hetzner storage box, $3.77/GiB/month, is a popular choice in the forum, as well as myself.

16

Oh yes that sounds like it would be fast. Just out of interest, what do you mainly use Wasabi for, backing up production environments or backing up personal files? From my limited knowledge, it seems like it's great for both.

Found a pic of your internet cable being installed :grin:

1 Like
17

both, run the same basic setup, run veeam on a windows server 2019 hyper-v edition using REFS.

i only use s3 storage, as it has SSE-C, MFA, session tokens, iam/bucket polices.

  1. use veeam to backup servers/workstation to local backup server
  2. rclone copy --immutable

and can create a VSS snaphost and use that as the source for rclone copy
https://forum.rclone.org/t/i-want-to-share-how-to-enable-vss-for-rclone/11644

1 Like
18

After more experimentation, I'm very happy to say I can now browse my network drive faster than my local SSD, even in Windows File Explorer... in fact it's so good I'm finding it hard to believe :face_with_raised_eyebrow:

The final piece of the puzzle was restricting the fetching of modtime metadata using --use-server-modtime, or alternatively --no-modtime. I found that fetching true modtime for every file significantly slowed the opening of big folders, even in --network-mode. For example, a folder (1.2k files) that took 9s to open when fetching true modtime was practically instant with --use-server-modtime.

So I've decided I don't care anymore about true modtime, I only care which file is newer (local vs remote), and --use-server-modtime is enough for me to make this determination. If I ever want true modtime I'll just remove the --use-server-modtime flag temporarily.

I've also not found a benefit to pre-caching the directory tree in addition to using --network-mode (as discussed here), so I've removed the pre-caching code from my startup script.

Here's my updated (and I think final) .bat file that works like a dream:

@echo off

:: Set variables, window title, and rclone location.
set remotename=wasabi
set driveletter=W
title %remotename%-boot
cd D:\Prizi\Downloads\media\applications\rclone-v1.60.0-windows-amd64

:: Mount remote drive with some flags.
echo Mounting %remotename% to %driveletter%:
rclone mount %remotename%: %driveletter%: ^
--vfs-cache-mode full ^
--cache-dir "%cd%\cache" ^
--use-server-modtime ^
--network-mode

Huge thanks to @asdffdsa and @thestigma (and @ncw) :metal:

1 Like
19

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.