I am attempting (again) to use Rclone to create a SSD read/write cache for specific folders on my unraid HDD array.
For those who may not know unraid is basically just drive pooling, so the r/w speeds are only that of a single drive.
The SSDs I am using are 2 Intel enterprise nvme in RAID0.
So far im running into two issues;
The two things I am currently trying to cache are my media, using Plex & Jellyfin. As well as an app called Seafile for file storage/sync.
First issue is that working through the mount is unbearably slow. Im talking so slow that seafile thinks theres something wrong and starts throwing sync errors on the clients. Jellyfin took days to finish scanning my library, and plex just thinks a bunch of my media is now missing.
The second issue is that the mount will just vanish. I can see my script is still running, and htop is showing that there are still rclone processes running, but the mount point just shows up emtpy after some random amount of time.
Run the command 'rclone version' and share the full output of the command.
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
[Local]
type = local
A log from the command that you were trying to run with the -vv flag
Will add logs
I will do my best to try and log what’s happening, so far it’s been a bit difficult as sometimes it will happen within an hour, sometimes it takes a day or more. Ill try running the commands with tmux instead of in background scripts so I can monitor them better.
I feel like on the other discussions, ZFS doesn't do sparse files so that's a bad choice for a rclone cache disk.
Oooh Ill have to look into that then. There is no reason for me using ZFS other than Im not a fan of BTRFS.
That does sound a lot more elegant, ill do that. For each directory I have a separate mount, and separate cache. Right now it’s only two so ill provide them both as they crash.
Ok so I added --vv as well as the line to write the log to a file. So far I don't see anything odd with the Media mount which is for Plex and Jellyfin.
However the mount for Seafile threw all kinds of things right off the bat.
2024/02/12 14:56:28 ERROR : logs/var-log/nginx/notification.access.log: vfs cache: failed to reload item: vfs cache: failed to download missing parts of cache file: failed to start downloader: failed to open downloader: vfs reader: failed to open source file: invalid seek position
2024/02/12 14:56:28 ERROR : logs/var-log/nginx/notification.error.log: vfs cache: failed to reload item: vfs cache: failed to download missing parts of cache file: failed to start downloader: failed to open downloader: vfs reader: failed to open source file: invalid seek position
2024/02/12 14:56:28 ERROR : logs/var-log/nginx/seafhttp.access.log: vfs cache: failed to reload item: vfs cache: failed to download missing parts of cache file: failed to start downloader: failed to open downloader: vfs reader: failed to open source file: invalid seek position
2024/02/12 14:56:28 ERROR : logs/var-log/nginx/seahub.access.log: vfs cache: failed to reload item: vfs cache: failed to download missing parts of cache file: failed to start downloader: failed to open downloader: vfs reader: failed to open source file: invalid seek position
2024/02/12 14:56:28 ERROR : seafile/logs/controller.log: vfs cache: restart download failed: failed to start downloader: failed to open downloader: vfs reader: failed to open source file: invalid seek position
2024/02/12 14:56:33 ERROR : seafile/logs/controller.log: vfs cache: restart download failed: failed to start downloader: failed to open downloader: vfs reader: failed to open source file: invalid seek position
2024/02/12 15:01:26 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "mount" "-vv" "Local:/mnt/user/seafile" "/mnt/read-cache/ZFS-R-Seafile" "--uid" "0" "--gid" "0" "--copy-links" "--vfs-cache-mode" "full" "--vfs-cache-max-age" "9999h" "--vfs-cache-max-size" "1000G" "--cache-dir=/mnt/read-cache/ZFS-R-VFS_Cache/Seafile" "--log-file" "/mnt/user/Public/rclone-logs/RCM-Seaf.log"]
2024/02/12 15:01:26 DEBUG : Creating backend with remote "Local:/mnt/user/seafile"
2024/02/12 15:01:26 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2024/02/12 15:01:26 DEBUG : Local: detected overridden config - adding "{12rtk}" suffix to name
2024/02/12 15:01:26 DEBUG : fs cache: renaming cache item "Local:/mnt/user/seafile" to be canonical "Local{12rtk}:/mnt/user/seafile"
2024/02/12 15:01:26 INFO : Local file system at /mnt/user/seafile: poll-interval is not supported by this remote
2024/02/12 15:01:26 DEBUG : vfs cache: root is "/mnt/read-cache/ZFS-R-VFS_Cache/Seafile"
2024/02/12 15:01:26 DEBUG : vfs cache: data root is "/mnt/read-cache/ZFS-R-VFS_Cache/Seafile/vfs/Local{12rtk}/mnt/user/seafile"
2024/02/12 15:01:26 DEBUG : vfs cache: metadata root is "/mnt/read-cache/ZFS-R-VFS_Cache/Seafile/vfsMeta/Local{12rtk}/mnt/user/seafile"
2024/02/12 15:01:26 DEBUG : Creating backend with remote "/mnt/read-cache/ZFS-R-VFS_Cache/Seafile/vfs/Local{12rtk}//mnt/user/seafile"
2024/02/12 15:01:26 DEBUG : local: detected overridden config - adding "{12rtk}" suffix to name
2024/02/12 15:01:26 DEBUG : fs cache: renaming cache item "/mnt/read-cache/ZFS-R-VFS_Cache/Seafile/vfs/Local{12rtk}//mnt/user/seafile" to be canonical "local{12rtk}:/mnt/read-cache/ZFS-R-VFS_Cache/Seafile/vfs/Local{12rtk}/mnt/user/seafile"
2024/02/12 15:01:26 DEBUG : Creating backend with remote "/mnt/read-cache/ZFS-R-VFS_Cache/Seafile/vfsMeta/Local{12rtk}//mnt/user/seafile"
2024/02/12 15:01:26 DEBUG : local: detected overridden config - adding "{12rtk}" suffix to name
2024/02/12 15:01:26 DEBUG : fs cache: renaming cache item "/mnt/read-cache/ZFS-R-VFS_Cache/Seafile/vfsMeta/Local{12rtk}//mnt/user/seafile" to be canonical "local{12rtk}:/mnt/read-cache/ZFS-R-VFS_Cache/Seafile/vfsMeta/Local{12rtk}/mnt/user/seafile"
2024/02/12 15:01:26 DEBUG : logs/var-log/nginx/notification.access.log: vfs cache: checking remote fingerprint "0,2024-02-12 04:54:01.898637926 +0000 UTC,d41d8cd98f00b204e9800998ecf8427e" against cached fingerprint "0,2024-02-12 04:54:01.898637926 +0000 UTC,d41d8cd98f00b204e9800998ecf8427e"
2024/02/12 15:01:26 DEBUG : logs/var-log/nginx/notification.access.log: vfs cache: truncate to size=1016 (not needed as size correct)
2024/02/12 15:01:26 DEBUG : vfs cache: looking for range={Pos:0 Size:1016} in [{Pos:0 Size:127}] - present false
2024/02/12 15:01:26 DEBUG : logs/var-log/nginx/notification.access.log: ChunkedReader.RangeSeek from -1 to 127 length -1
2024/02/12 15:01:26 INFO : logs/var-log/nginx/notification.access.log: vfs cache: downloader: error count now 1: failed to open downloader: vfs reader: failed to open source file: invalid seek position
2024/02/12 15:01:26 ERROR : logs/var-log/nginx/notification.access.log: vfs cache: failed to reload item: vfs cache: failed to download missing parts of cache file: failed to start downloader: failed to open downloader: vfs reader: failed to open source file: invalid seek position
2024/02/12 15:01:26 DEBUG : logs/var-log/nginx/notification.error.log: vfs cache: checking remote fingerprint "0,2024-02-12 04:54:01.901995293 +0000 UTC,d41d8cd98f00b204e9800998ecf8427e" against cached fingerprint "0,2024-02-12 04:54:01.901995293 +0000 UTC,d41d8cd98f00b204e9800998ecf8427e"
2024/02/12 15:01:26 DEBUG : logs/var-log/nginx/notification.error.log: vfs cache: truncate to size=2195 (not needed as size correct)
2024/02/12 15:01:26 DEBUG : vfs cache: looking for range={Pos:0 Size:2195} in [{Pos:0 Size:272}] - present false
2024/02/12 15:01:26 DEBUG : logs/var-log/nginx/notification.error.log: ChunkedReader.RangeSeek from -1 to 272 length -1
2024/02/12 15:01:26 INFO : logs/var-log/nginx/notification.error.log: vfs cache: downloader: error count now 1: failed to open downloader: vfs reader: failed to open source file: invalid seek position
2024/02/12 15:01:26 ERROR : logs/var-log/nginx/notification.error.log: vfs cache: failed to reload item: vfs cache: failed to download missing parts of cache file: failed to start downloader: failed to open downloader: vfs reader: failed to open source file: invalid seek position
2024/02/12 15:01:26 DEBUG : logs/var-log/nginx/seafhttp.access.log: vfs cache: checking remote fingerprint "0,2024-02-12 04:54:01.905247845 +0000 UTC,d41d8cd98f00b204e9800998ecf8427e" against cached fingerprint "0,2024-02-12 04:54:01.905247845 +0000 UTC,d41d8cd98f00b204e9800998ecf8427e"
2024/02/12 15:01:26 DEBUG : logs/var-log/nginx/seafhttp.access.log: vfs cache: truncate to size=248038 (not needed as size correct)
2024/02/12 15:01:26 DEBUG : vfs cache: looking for range={Pos:0 Size:248038} in [{Pos:0 Size:142}] - present false
2024/02/12 15:01:26 DEBUG : logs/var-log/nginx/seafhttp.access.log: ChunkedReader.RangeSeek from -1 to 142 length -1
2024/02/12 15:01:26 INFO : logs/var-log/nginx/seafhttp.access.log: vfs cache: downloader: error count now 1: failed to open downloader: vfs reader: failed to open source file: invalid seek position
2024/02/12 15:01:26 ERROR : logs/var-log/nginx/seafhttp.access.log: vfs cache: failed to reload item: vfs cache: failed to download missing parts of cache file: failed to start downloader: failed to open downloader: vfs reader: failed to open source file: invalid seek position
2024/02/12 15:01:26 DEBUG : logs/var-log/nginx/seahub.access.log: vfs cache: checking remote fingerprint "0,2024-02-12 04:54:01.908254742 +0000 UTC,d41d8cd98f00b204e9800998ecf8427e" against cached fingerprint "0,2024-02-12 04:54:01.908254742 +0000 UTC,d41d8cd98f00b204e9800998ecf8427e"
2024/02/12 15:01:26 DEBUG : logs/var-log/nginx/seahub.access.log: vfs cache: truncate to size=98030 (not needed as size correct)
2024/02/12 15:01:26 DEBUG : vfs cache: looking for range={Pos:0 Size:98030} in [{Pos:0 Size:122}] - present false
2024/02/12 15:01:26 DEBUG : logs/var-log/nginx/seahub.access.log: ChunkedReader.RangeSeek from -1 to 122 length -1
2024/02/12 15:01:26 INFO : logs/var-log/nginx/seahub.access.log: vfs cache: downloader: error count now 1: failed to open downloader: vfs reader: failed to open source file: invalid seek position
2024/02/12 15:01:26 ERROR : logs/var-log/nginx/seahub.access.log: vfs cache: failed to reload item: vfs cache: failed to download missing parts of cache file: failed to start downloader: failed to open downloader: vfs reader: failed to open source file: invalid seek position
2024/02/12 15:01:26 DEBUG : seafile/logs/controller.log: vfs cache: checking remote fingerprint "12502,2024-02-10 05:41:34.870894248 +0000 UTC,b33059d76c70f9dd98723cf41d7003a4" against cached fingerprint "12502,2024-02-10 05:41:34.870894248 +0000 UTC,b33059d76c70f9dd98723cf41d7003a4"
2024/02/12 15:01:26 DEBUG : seafile/logs/controller.log: vfs cache: truncate to size=14049 (not needed as size correct)
2024/02/12 15:01:26 DEBUG : vfs cache: looking for range={Pos:0 Size:14049} in [{Pos:12502 Size:113}] - present false
2024/02/12 15:01:26 DEBUG : seafile/logs/controller.log: ChunkedReader.RangeSeek from -1 to 0 length -1
2024/02/12 15:01:26 DEBUG : seafile/logs/controller.log: ChunkedReader.Read at -1 length 32768 chunkOffset 0 chunkSize 134217728
2024/02/12 15:01:26 DEBUG : seafile/logs/controller.log: ChunkedReader.openRange at 0 length 134217728
2024/02/12 15:01:26 DEBUG : seafile/logs/controller.log: ChunkedReader.RangeSeek from -1 to 12615 length -1
2024/02/12 15:01:26 INFO : seafile/logs/controller.log: vfs cache: downloader: error count now 1: failed to open downloader: vfs reader: failed to open source file: invalid seek position
2024/02/12 15:01:26 ERROR : seafile/logs/controller.log: vfs cache: restart download failed: failed to start downloader: failed to open downloader: vfs reader: failed to open source file: invalid seek position
2024/02/12 15:01:31 DEBUG : seafile/logs/controller.log: ChunkedReader.RangeSeek from -1 to 12615 length -1
2024/02/12 15:01:31 INFO : seafile/logs/controller.log: vfs cache: downloader: error count now 2: failed to open downloader: vfs reader: failed to open source file: invalid seek position
2024/02/12 15:01:31 ERROR : seafile/logs/controller.log: vfs cache: restart download failed: failed to start downloader: failed to open downloader: vfs reader: failed to open source file: invalid seek position
Ok so based the some of the reading I did on ZFS it seems there may actually be some kind of...maybe not incompatibility, but inefficiency in using something like Rclone to cache data onto a ZFS volume, especially directly from a disk of a totally different file system.
I decided to wipe the cache drive since my vfs cache became corrupted anyway, and I reformatted it as RAID0 BTRFS.
So far it's still been painfully slow, so much so that Jellyfin has only progressed about 15% on its library scan in over 12hours. and Seafile is basically unusable.
I did continue the logging, but in that time span the logs grew to 180MB and 2.7GB so I'm sure what to do with those, seems a bit large to try and attach.
You can paste them to any file share service that you use. Google Drive, One Drive, anything.
Rclone cache is really only going to help once you get something in it. Prior to that, if Seafile or whatever you are using is slow, it's going to be slow with a cache too until the files are in the cache.
Sounds like you are scanning a new library or something, which would (in theory) be about the same with or without the cache (some slight overhead on cache, but neglible imo).
I will absolutely do that, but I am going to have to start the logging over. The files grew so large I was struggling to even open them.
Of course. The issue I was having is that when accessing files through the mount that were not yet cached, it was so slow that things were crashing, or throwing error messages.
I didn't do this, mostly because I have nowhere else to put it. However it did give me an idea.
Instead of using "local" as the remote, I used SMB as the remote with an address of 127.0.0.1 (local host) but still pointing to the same directory as before.
I completely re-formatted the disk I'm using as the vfs cache and So far I have ran one test and it's looking pretty good. I took my kids two favorite shows, and made a playlist per season, so thats 7 playlists with a total of around 400 videos. I shared the rclone mount over SMB so I could access it from my desktop. Then I simultaneously played through the entirety of all 7 playlists at 16x playback speed, I watched the cache fill up as they played and the whole time I didn't notice a single hiccup or stutter during playback.
I still need to get Jellyfin, Plex, and Seafile set back up and that will be the "true" test. But since I messed them all up anyway Im taking the opportunity to do a little housekeeping lol. Luckily Seafile had nothing in it other than test data anyway.