I was impatient to wait until the next freeze so I set the following:
--bwlimit off:40M
--vfs-refresh
--onedrive-delta
--tpslimit 3
I haven't watched anything large for an extended period of time, but anything I did watch took noticeably longer to start or resume; where before things would start playing within 3-4 seconds, now they take 8-10 seconds at least. Is this because of the tpslimit?
My understanding is that rclone will allow itself to exceed this on a per-file basis. So if you have a 10gb file to stream, rclone won't do anything to it because it is over the cache.
That is correct, yet he is not selecting a cache directory, so cache will be on RAM, which can easily run out with big files, specially if you got slow internet and also process uploads via it (in particular with OneDrive that requires file to be completely in cache before upload can begin) so again, might not be a problem, but can easily cause problems and hence my recommendation.
That most likely is the reason, you are basically limiting the amount of transactions per minute to 3, so things will take overall longer to be performed. Also the bwlimit might also slow things in some scenarios. In reality this flags will slow things in hopes of avoiding Microsoft complain and throttle you.
I am 99% sure that is not correct. If you don't set a cache directory, rclone uses the default from
rclone config paths
That's correct.
It usees the .config path if you don't set it as it's a file base cache and not memory.
I've read that too, but I don't think rclone is caching the entire file. If I am watching a file that's way over the server disk free space, that can't (and doesn't) happen.
I was under the impression rclone only caches parts of the file that you need, i.e. the parts you are watching right now in the case of video. Is that not the case?
So, with 5GB vfs cache, it will cache a 5GB recently watched portion, no?
I have watched a video file that is over 140GB, so either it cached it in parts, or it didn't cache it at all.
If rclone really doesn't cache files bigger than the vfs-cache-max-size, then with most of my files being larger than that, is vfs-cache even making a difference?
I can live with that if it means no more freezing.
You are correct. But the reason it doesn't happen is that rclone makes a sparse file for the cache. Every file system I have tested with won't allow you to make a sparse file larger than its free space.
It certainly would have made a sparse file at 140GB. I do not know how it expunges the previously cached portions of the same file. That is a good question. It is likely not even documented. You'd need to look at source
I do not belive it does make a difference. Except if you have smaller files and a larger cache, it lets more stick around
I have streamed from OneDrive but I haven't tried 140GB. That is massive! And requires an average about 20Mb/sec for a 2 hr video (if my math is correct). Honestly, I wonder if that is beyond what OneDrive will give you.
It has worked for the most part for 3 such files, with the exception of a random freeze on one of those files, and that was 3+ hours into playback. So for the majority of time OneDrive does seem able to deliver it. Let's hope that with these settings we'll eliminate any incidental throttling that may occur after such long playback times.
btw. as we're talking about Onedrive:
In case you want to span open a crypt drive on onedrive, consider to use base32768 filename encoding.
This will compress the path-length as onedrive and dropbox are heavy limited with this.
Hmm, thanks for the heads-up. I don't have particularly long paths, and I don't believe I've had any path-too-long errors (everything I upload seems to get there successfully).
If I wanted to change the filename encoding, could I do it now that all onedrive crypts are already in place? Or can it only be for a new crypt?
BTW I found that tpslimit 3 makes scanning media way too slow, in the order of many hours for just one library, so I changed it to tpslimit 5 and tpslimit-burst 10. Now scanning is done within a reasonable time. I have yet to play something very large to test for the freezing issue.
Something happened today, while watching a large media file.
At some random point, it started glitching, freezing for half a second, then resuming, then glitching again. When I exited Jellyfin, opened it again and resumed, the problem went away.
This is what the log shows:
2024/02/01 21:30:56 DEBUG : vfs cache: looking for range={Pos:46949662720 Size:131072} in [{Pos:43661799424 Size:3288330240}] - present true
2024/02/01 21:30:56 DEBUG : Movies/2000 - 2009/movie.mkv(0xc00aef6
380): _readAt: size=131072, off=46949793792
2024/02/01 21:30:56 DEBUG : Movies/2000 - 2009/movie.mkv(0xc00aef6
380): >_readAt: n=131072, err=<nil>
2024/02/01 21:30:56 DEBUG : vfs cache: looking for range={Pos:46949793792 Size:131072} in [{Pos:43661799424 Size:3288330240}] - present true
2024/02/01 21:30:56 DEBUG : &{Movies/2000 - 2009/movie.mkv (rw)}:
>Read: read=131072, err=<nil>
2024/02/01 21:30:56 DEBUG : Movies/2000 - 2009/movie.mkv(0xc00aef6
380): >_readAt: n=131072, err=<nil>
2024/02/01 21:30:56 DEBUG : &{Movies/2000 - 2009/movie.mkv (rw)}:
>Read: read=131072, err=<nil>
2024/02/01 21:30:56 DEBUG : &{Movies/2000 - 2009/movie.mkv (rw)}:
Read: len=131072, offset=46949924864
2024/02/01 21:30:56 DEBUG : Movies/2000 - 2009/movie.mkv(0xc00aef6
380): _readAt: size=131072, off=46949924864
2024/02/01 21:30:56 DEBUG : vfs cache: looking for range={Pos:46949924864 Size:131072} in [{Pos:43661799424 Size:3288330240}] - present true
2024/02/01 21:30:56 DEBUG : Movies/2000 - 2009/movie.mkv(0xc00aef6
380): >_readAt: n=131072, err=<nil>
2024/02/01 21:30:56 DEBUG : &{Movies/2000 - 2009/movie.mkv (rw)}: >Read: read=131072, err=<nil>
2024/02/01 21:30:56 DEBUG : &{Movies/2000 - 2009/movie.mkv (rw)}: Read: len=131072, offset=46950055936
2024/02/01 21:30:56 DEBUG : Movies/2000 - 2009/movie.mkv(0xc00aef6380): _readAt: size=131072, off=46950055936
2024/02/01 21:30:56 DEBUG : vfs cache: looking for range={Pos:46950055936 Size:131072} in [{Pos:43661799424 Size:3288330240}] - present false
2024/02/01 21:30:56 DEBUG : &{Movies/2000 - 2009/movie.mkv (rw)}: Read: len=131072, offset=46950187008
2024/02/01 21:30:56 DEBUG : Movies/2000 - 2009/movie.mkv(0xc00aef6380): _readAt: size=131072, off=46950187008
Suddenly, there is a 'present false'.
Does this mean that at that point, it is failing to find the next segment of the file in the vfs cache?
And if so, what could be the reason?
I don't see any of "increasing sleep|Reducing sleep|throttledRequest" anywhere in the entire log file.
EDIT: I see this 'present false' elsewhere in the log, when the file was playing fine, so probably not it.
EDIT2: At the final point of glitching where it was doing that so much it was unwatchable, I can see 3 'present false' in quick succession, maybe it is related after all?
However, I found something else:
2024/02/01 21:31:00 DEBUG : Movies/2000 - 2009/movie.mkv(0xc00aef6380): _r
eadAt: size=131072, off=46974435328
2024/02/01 21:31:00 DEBUG : vfs cache: looking for range={Pos:46974435328 Size:131072} in [{Pos:43661799424 Size:3312447488}] - present false
2024/02/01 21:31:00 INFO : OneDrive root 'dirname': Change notify listener failure: invalidRequest: [token.String] A value is required but was not present in the request.
2024/02/01 21:31:00 INFO : OneDrive root 'dirname': Change notify listener failure: invalidRequest: [token.String] A value is required but was not present in the request.
2024/02/01 21:31:01 DEBUG : Movies/2000 - 2009/movie.mkv: ChunkedReader.Re
ad at 46975295488 length 1048576 chunkOffset 45675065344 chunkSize 2147483648
2024/02/01 21:31:01 DEBUG : Movies/2000 - 2009/movie.mkv(0xc00aef6380): >_
readAt: n=131072, err=<nil>
All this was at the time the glitch was happening. I am really not sure what that log entry might mean.
Have a look here
Ok, that person had the same error message, but it appears he fixed it by including a client id & secret in the remote; I already have client IDs and the corresponding secrets for all remotes. I just checked the the cliend ID/secret for the remote involved today is in the rclone config file and is correct.
He started from scratch with a new remote. Maybe it helps...
I would completely delete all the cache folder, remount and use the delta flag and see if it helps.
I have already added the delta flag. I will delete the contents of the vfscache folder and see how it goes.
EDIT: As I am not seeing anything at all in the rclone logs (set to DEBUG mode) to hint that there is throttling going on (as per previous suggestions in this thread to grep for certain keywords), is this getting to the point where we can assume there is no throttling and this is a different problem?
Yes, I guess so.
It could also be a network/bandwidth problem. If OneDrive is not delivering data fast enough for you to stream. You can check it with Azure speedtest which gives you a pretty close estimate of of OneDrive speed.
Be sure to select a couple of regions close to your server to test.
You could also make a cache big enough to hold the whole file, and fetch the whole file to the cache and see how fast you can download the whole file. This would also show any issues in the log. Once downloaded, try playing it and see if it buffers, it might shed some further light into the problem.
That's the conclusion/assumption I ended up with as well. And I got to thinking this network issue could be anywhere, e.g. between:
For OneDrive --> server, I can rclone copy at 80MB/s sustained. Of course that doesn't mean that the link doesn't go down at random points in time.
For server --> LAN, I am getting 25MB/s average file transfer rate, so that's good, although it could be a lot higher (both server and LAN have gigabit links).
LAN internally runs on gigabit ethernet. It would be a possibility to try playing media over WiFi (6) instead of ethernet to rule out any badly wired ethernet ports.
However, last night I was again thinking about this after a freeze and I remember this little detail:
What kind of issues were those?
I just thought I'd remove http2 from my jellyfin nginx config to try it out... and two and a half movies later, no freezing!
I don't know if it will last, more testing to be done.
Just disable it with --disable-http2 since my mount runs better with