Max Pages on Linux with rClone/MergerFS

@trapexit - I saw you had a pull request for this:

which moves the 128K up to 1M, but I wasn't sure how to take advantage of that. Is that something that both mergerfs and rclone can take advantage of a kernel newer than 4.20?

That would seem to greatly speed up sequential reads as it would remove a number of calls.

It needs support in the underlying communication to the kernel. mergerfs will inform the kernel to use 1MB if the kernel says it's supported. It's configurable to allow comparison and testing.

rclone would need to explicitly support it. Not sure what it uses to talk to the kernel but that would need to set the correct flags at the FUSE init stage (and allocate enough memory for incoming buffers).

@ncw - is that something to look at for rclone?

Rclone uses

as its fuse library on Linux. I have to say it is looking a bit unmaintained with only 2 commits since 2016 :frowning:

It does not appear to allow setting of max_pages.

The alternative library is this one and there is an issue by our very own @darthShadow about this

I have an experimental branch making rclone work with go-fuse which almost works!

Yeah, I had a bit of an experiment about this earlier.

Noticed the max_pages commit when it was pushed to mergerfs and was searching about how to use it in rclone. Figured that it was something that needed to be supported in the fuse backend used by rclone so went looking for it in the corresponding library.

Saw that bazil/fuse was not being maintained so again went looking through the alternative libraries available for rclone. Found the go-fuse library and created this issue there. Unfortunately, haven't been able to make progress there since I am not sure what flag to pass and the author seems stumped too.

Perhaps @trapexit could provide some help here?

Just like any of the other negotiated features you'd need to check in INIT if FUSE_MAX_PAGES is available and if so you would then set it in the flags of the init_out struct and set the max_pages field.

FYI I revamped the go-fuse interface to rclone. You can try it here with rclone mount2.

Any testing much appreciated! This will only work on linux and macOS (not on freebsd).

https://beta.rclone.org/branch/v1.50.1-023-gd0ba9379-mount2-v2-beta/ (uploaded in 15-30 mins)

go-fuse seemed quite a bit quicker in my testing as it does async reads and writes.

I can probably add that to rclone mount too...

1 Like

Did the builds fail for Linux?

Yes it looks like they did! That will take a little while to fixup! The tests all pass locally so not sure why they didn't on the CI! https://github.com/rclone/rclone/runs/287962101

Looks like the tests finally succeeded? https://github.com/rclone/rclone/commit/48de5eb1bd4dd20becaf76fcd0464bd7efff0c47/checks?check_suite_id=305489468

Excellent work fixing the issues. Will try the new mount out soon. Any suggestions regarding the config flags that you think may require changes with mount2?

Current flags:

   --allow-other \
   --dir-cache-time=96h \
   --drive-chunk-size=64M \
   --vfs-read-chunk-size=128M \
   --vfs-read-chunk-size-limit=2G \
   --vfs-cache-mode=writes \
   --vfs-cache-max-age=30m \
   --vfs-cache-poll-interval=5m \
   --poll-interval=30s \
   --cache-dir=/home/darthshadow/rclone/vfs/TD/ \
   --buffer-size=512M \
   --attr-timeout=1s \
   --umask=002 \
   --log-file=/home/darthshadow/rclone/rclone-td.log \
   --log-level=INFO \
   --stats=30s \
   --stats-log-level=NOTICE \
   --rc \
   --rc-addr=127.0.0.1:5573 \

Yes, here is rclone with a mount2 command for testing

https://beta.rclone.org/branch/v1.50.1-036-g9aa70582-mount2-v2-beta/

Most of those flags are for the VFS layer which is unchanged. Potentially --attr-timeout might need tweaking, but 1s is the default value so probably not. I haven't tried --allow-other so that may be broken :wink: I think it should be pretty much 100% compatible flags wise, just change mount to mount2!

1 Like