I'm trying it right now with cache mode full. Cache mode writes didn't help.
I've tried the cache backend on and off several times, but it hasn't helped. I enjoy the simplicity of not having the extra cache layer since it doesn't seem to do anything (for my use case).
I can try again and check on my dir-cache-time and info-age numbers, make sure that stuff is right.
I chose 1G buffer size based on a post by @Animosity022 over here.
I may try buffer size 0, as that has also been recommended for performance.
What do you think?