Optimal mount parameters for live editing small files on remote SFTP server

[Chris_Dempsey]

Can you help me optimise the mount parameters such that it is optimal for live editing small files on remote SFTP server?

• Rclone version 1.62.2 on Windows 10 Pro

Use case

Using rclone I create a mount to edit files on remote web servers. After saving changes to a file I want to be able to refresh the browser and see the change immediately.

I also need to be able to search recursively through directories on the web server for text strings within files. This is done with the Find function in VSCode.

Related information

• mounts are created as and when I need them from the terminal, usually to edit a couple of files, disconnect and move on to another server. Drive letter is never allocated explicitly, I let rclone grab the first available
• often the mount type used is type = combine which combines several connections defined in rclone.conf as type = sftp
• the files being edited are small, typically 10-50kb
• the servers used are fast, usually with dedicated resource, not cheap shared hosting with high contention
• the internet connection used is a fairly low latency business fibre, 1000Mbps down, 100Mbps up

Mount command

The command I'm using is

rclone mount SERVER_COMBINED: * --vfs-cache-mode writes

Early issue

First few times using rclone I experienced

ERROR : file.txt: WriteFileHandle: Can't open for write without O_TRUNC on existing file without --vfs-cache-mode >= writes

This was resolved by adding --vfs-cache-mode writes.

Current issue

There is no issue per se, this is more of a sanity check because I'm not convinced the mount command above is optimal.

That said

• I sometimes resort to pushing edited files to the server using FileZilla (I'll make a point of documenting the circumstances in future) and
• I have noticed that searching directories sometimes fails to return any results (I should have checked for error output in the terminal).

The same query Best practice for mounting with sftp was raised on the link below. Discussion covers --polling and --dir-cache-time but I'm unclear if this is relevant to my use case.

Best practice for mounting with sftp

Hi just had a few questions on how some of these options work. I'm working with a sftp mount and am trying to figure out whats the best way to get updates as close to real time as possible. Im not limited by google or some other cloud storage provider, so whats the best way getting around that difference in mount and my remote file listings? I read that --polling isn't supported for sftp mounts, am I limited to --dir-cache-time? What happens after --dir-cache-time runs out? Does it poll for ch…

I'm interested to learn if the default --dir-cache-time 5m0s is optimal for my use case. Should this be reduced to 20s or lower? Or would that adversely affect searching through directories?

The documentation for mount has a bewildering array of options, I can't figure if any would refine what I'm looking to achieve.

rclone mount

Any advice would be appreciated.

Thanks

[kapitainsky]

I think for your use case it can be kept simple and easy.

I would for sure change --vfs-cache-mode writes to --vfs-cache-mode full
You search files by content - first search is always slow as data has to be pulled out of the server but next searches with full mode will be "local". With writes it is always from the server.

Then couple of flags to control --vfs-cache-mode:

--cache-dir string                   Directory rclone will use for caching.
--vfs-cache-max-size SizeSuffix      Max total size of objects in the cache (default off)

Everything else I would leave to default.

@Chris_Dempsey

I'm interested to learn if the default --dir-cache-time 5m0s is optimal for my use case. Should this be reduced to 20s or lower? Or would that adversely affect searching through directories?

Setting it too low completely invalidates cache benefits - I would rather go opposite way and increase it value.

[Chris_Dempsey]

@kapitainsky

I would for sure change --vfs-cache-mode writes to --vfs-cache-mode full

Noted, a persistent cache makes sense to facilitate search.

@kapitainsky

Then couple of flags to control --vfs-cache-mode

--cache-dir - the documentation notes

You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir. You don't need to worry about this if the remotes in use don't overlap.

Next question - I have many SFTP remotes for single websites, and often one of those remotes is defined in a second remote of type = combine. In this case, are the remotes considered to be overlapping and each must have a unique --cache-dir specified?

@kapitainsky

Setting --dir-cache-time too low completely invalidates cache benefits - I would rather go opposite way and increase it value

Understood in relation to the note above.

Final question, am I correct to think that on saving a file rclone will attempt to send it to the remote immediately? If so would this action be delayed if there was another operation running in the background eg. search?

For context a typical workflow follows and this might explain why I sometimes need to resort to using FileZilla to push updated files to the server? ie. mount a remote from the terminal > immediately search for text in a directory with hundreds of files > update a file > save > refresh the browser immediately - and the change is not apparent on site.

Looking further I should also need to investigate the --exclude-from flag. This will be helpful to ignore cache directories and similar on the remote server.

Rclone Filtering

Thanks @kapitainsky

[kapitainsky]

@Chris_Dempsey

--cache-dir - the documentation notes

Yes every remote mount should use its own cache

if you have:

remote1
remote2

use

rclone mount remote1 --cache-dir /path/to/remote1cache
rclone mount remote2 --cache-dir /path/to/remote2cache

Does not matter if remote is combine or not. One remote - one cache location.

The easiest would be to write script wrapper and only run my_mount.script remote_name. And script would take care of not using the same cache e.g. --cache-dir /path/to/$remote_name. As remotes' names are unique.

@Chris_Dempsey

Final question, am I correct to think that on saving a file rclone will attempt to send it to the remote immediately? If so would this action be delayed if there was another operation running in the background eg. search?

It is not immediate. First file has to be closed (if was opened by any app) then:

--vfs-write-back duration            Time to writeback files after last use when using cache (default 5s)

If you are changing multiple files you can also add --transfers N - by default it is 4 files uploaded in parallel.

[Chris_Dempsey]

Perfect, thank you.

I tested --cache-dir as working with the command below and saw files appearing in the cache as I browsed the mounted disk.

rclone mount mysite:/var/www/vhosts/mysite.co.uk/httpdocs * --vfs-cache-mode full --cache-dir C:\cache\rclone\mysite_co_uk --vfs-cache-max-size 50M --vfs-cache-max-age 168h

However for this particular site I got the remote path incorrect. Instead of :/var/www/vhosts/mysite.co.uk/httpdocs it should have been one level higher :/var/www/vhosts/mysite.co.uk.

I disconnected the mount, deleted the files from the cache directory, updated the mount command and reconnected it.

Now when browsing the mounted disk files show as normal but nothing is saved to the cache directory. So two new questions are

• if I had just disconnected, updated the mount path and reconnected would the cache still be working?
• is there a way I can make the existing cache path C:\cache\rclone\mysite_co_uk work again? do I need to rename the config to mysite2 and the cache path to C:\cache\rclone\mysite2_co_uk?

I tried restarting my workstation but the behaviour above persists.

Any further help appreciated.

[kapitainsky]

If you deleted the files from the cache directory then it should work. There is absolutely no reason not to.

[Chris_Dempsey]

@kapitainsky

If you deleted the files from the cache directory then it should work. There is absolutely no reason not to.

I found if deleting files from within the cache directory they were not recreated once the remote was mounted again eg.

C:\cache\rclone\mysite_co_uk\vfs\mysite_co_uk\var\www\vhosts\mysite.co.uk\subdomains\stage\*.*

However if the cache directory itself is deleted ie. C:\cache\rclone\mysite_co_uk files are recreated once the remote was mounted again.

@kapitainsky

It is not immediate. First file has to be closed (if was opened by any app)

Adding the --vfs-write-back flag to the mount command with a value of 1s seems to work well under a brief test.

Initially I took your comment to mean the file had to be closed in the software used to edit it ie. close the edit window. However tests show this isn't required, it seems to be as long as anything using the file has released any locks rclone is then free to upload it according to the --vfs-write-back schedule.

Thanks again, this has been very helpful.