Recommended OneDrive mount options for streaming 4K

[dinosm]

Hi everyone!
I searched for old topics about this but could only find one or two broken links to recommended scripts/templates, so...

What is the recommended rclone mount configuration for OneDrive to be used as a media mount for Plex/Jellyfin?
The use case is a remote VPS that needs to mount the OneDrive remote and use crypt on top. In fact, it's a union of OneDrives but I am assuming the union doesn't affect the recommended configuration. I am mostly looking to be able to play high-bitrate 4K movies without problems.

I've had some problems with media either freezing or, in a recent case posted here, something gone very wrong and preventing the movie file from playing again until I re-downloaded it, so I am wondering what mount options etc have others got that work for you?

Thank you in advance for any suggestions.

My current mount command is as follows:

ExecStart=/usr/bin/rclone mount \
        --config=/home/dinosm/.config/rclone/rclone.conf \
        --vfs-cache-mode full \
        --vfs-cache-max-age 4h \
        --vfs-cache-max-size 5G \
        --vfs-fast-fingerprint \
        --vfs-read-ahead 2G \
        --union-min-free-space 15G \
        --allow-other \
        --user-agent "ISV|rclone.org|rclone/v1.65.1" \
        --log-file=/home/dinosm/log/rclone_mediaf.log \
        --log-level DEBUG \
        mediaf: /home/dinosm/media/MediaF

I have tried it without the --vfs-read-ahead option, in fact that's how I used to have it, but the same problems occurred.

[Dalo (google account)]

Are you sure not hitting Microsofts API Limit ? You didn't set an --tpslimit which is highly recommended.

I noticed my thoughts here

Best practice Jellyfin with Onedrive (business)

Hey guys, I'm running a Jellyfin Server with 50 TB+ of storage (a few sharepoint shares connected) and watched the rclone debug log for a long time now, hoping not to trigger the pacer...here are my conclusions: Setting up jellyfin at first needs to scan the whole library, surfing through each folder and file and build up an index, while transferring a LOT of small files. (metadata) means a lot of queries in a short period. -> Consider using your local storage OR a remote share with FTP by 10…

[dinosm]

Thanks for answering my question!

@Dalo

Are you sure not hitting Microsofts API Limit ?

I am not sure, however I can't see anything like that in rclone logs. I have seen others post logs where rate limiting happened and it's very clear in the logs, however mine don't contain anything like that when the freezing happens (or anywhere else).

@Dalo

You didn't set an --tpslimit which is highly recommended.

What exactly is the recommended settings for --tpslimit for OneDrive?
And would that help you think? Now my library is all set up, rescans happen very quickly, I don't think I've ever had a problem/slowdown when rescanning. Would Jellyfin/rclone hit API limits when just playing a single file?

EDIT: Since you mentioned throttling (even though I am not seeing this in the logs), I found this thread below, where the conclusion seems to be that the rclone defaults are best for transfers of up to 50-100GB or over a few hours, but then more throttling might be applied. I believe some of my freezing issues have been on very large files and after the middle of a movie, or after having watched at least one movie and going on to another one. But I have not kept detailed notes; I seem to remember once or twice it happened after a few minutes of watching something.

What do you think --tpslimit should be to avoid any extra throttling while still allowing large files to stream without issue?

Rclone v1.55.1 throttling and worse on OneDrive copy to GD

Thanks for your help @Ole! Decided to laeave things alone well past the 1.26 days. Started running this command rclone --ignore-existing --checkers=16 copy -P with no other flags as you suggested and it's going great now Many mixed file sizes but the current rate is showing ~33 Mbyte/s with no warnings as yet. Will add the cloud-specific backend config entries added afterwards, really appreciate the info!

[Dalo (google account)]

Blockquote
I am not sure, however I can't see anything like that in rclone logs. I have seen others post logs where rate limiting happened and it's very clear in the logs, however mine don't contain anything like that when the freezing happens (or anywhere else).

You can check this with
cat /yourpath/rclone.log | grep -E "increasing sleep|Reducing sleep|throttledRequest"

or observe live with

tail -f /yourpaths/rclone.log | grep -E "increasing sleep|Reducing sleep|directory tree done|list files|throttledRequest|too long"

Blockquote
What exactly is the recommended settings for --tpslimit for OneDrive?
And would that help you think? Now my library is all set up, rescans happen very quickly, I don't think I've ever had a problem/slowdown when rescanning.

There is no general recommendation. My Observations show, that 3 is a good value.

You can have my present Mount options. I use it with onedrive business

  /usr/bin/rclone mount \
    --config=%h/.config/rclone/rclone.conf \
    --log-level DEBUG \
    --log-file /root/logs/rclone.log \
    --umask 022 \
--vfs-cache-mode full \
--allow-other \
--no-modtime \
--buffer-size 32M \
--cache-dir /root/rclone/cache \
--no-checksum \
--disable-http2 \
--vfs-fast-fingerprint \
--ignore-checksum \
--no-check-certificate \
--checkers 1 \
--tpslimit 3 \
--transfers 1 \
--bwlimit-file 40M:40M \
--low-level-retries 1 \
--onedrive-no-versions \
--onedrive-hash-type none \
--onedrive-chunk-size 250M \
--dir-cache-time 9999h \
--vfs-cache-max-age 1m \
--vfs-read-chunk-size 256M \
--poll-interval 1m \
--ignore-size \
--user-agent "ISV|sadasdasda|1fdsffdsfdsfsdc" \
--vfs-cache-min-free-space 40G

Blockquote
Would Jellyfin/rclone hit API limits when just playing a single file?

This should not happen, even if you have some streams in parallel

[jcastill (Jose)]

I highly recommend using disk cache and a bigger size one. As OneDrive does a terrible job streaming files, you will have problems with only 5GB of cache.

You can also grep your log for pacer and see if you are getting throttled, OneDrive is very sensitive to multiple requests. My sweet spot to minimize issues on mount has been: --tpslimit 3 --tpslimit-burst 0

You might also try --vfs-refresh and --onedrive-delta flag to optimize your mount drive.

I've also experienced weird issues with http2 from time to time, --disable-http2 flag also gives me a better experience.

[dinosm]

Thank you both for all the great advice!

@Dalo

You can check this with
cat /yourpath/rclone.log | grep -E "increasing sleep|Reducing sleep|throttledRequest"

So, my next step will be to monitor it (I haven't been able to watch anything 'large' lately) and if/when it happens again, I'll grep for the strings you mentioned.

@Dalo

My Observations show, that 3 is a good value.

@jcastill

My sweet spot to minimize issues on mount has been: --tpslimit 3 --tpslimit-burst 0

I am not sure requests are going above API limits, I mean, only one stream being watched at the time this happens. But this is definitely something I will change next time it happens.

@jcastill

--vfs-refresh and --onedrive-delta flag

From the docs I understand this is to list the files quicker, and only/mostly works if the files are in the root directory, which for me they are not, so not sure this would help? I never have issues rescanning libraries.

@jcastill

--disable-http2 flag also gives me a better experience.

That's why I am asking here . See, such a weird find, how did you figure that out? This is another one for my list in case the problem persists.

@jcastill

you will have problems with only 5GB of cache.

Hmmm maybe I don't understand the cache correctly. I assumed that the cache helps a) because it fetches the next few seconds from the remote so they play from the server directly (which is faster/more reliable) and b) I can rewind much faster. But why would I need to set a bigger vfs cache size?

Overall, I have managed to watch one movie since I increased cache time to 4h and removed --vfs-read-ahead. I will follow these suggestions if/when the next movie freezes.

[Dalo (google account)]

I highly recommend using disk cache and a bigger size one. As OneDrive does a terrible job streaming files, you will have problems with only 5GB of cache.

You can also grep your log for pacer and see if you are getting throttled, OneDrive is very sensitive to multiple requests. My sweet spot to minimize issues on mount has been: --tpslimit 3 --tpslimit-burst 0

You might also try --vfs-refresh and --onedrive-delta flag to optimize your mount drive.

I've also experienced weird issues with http2 from time to time, --disable-http2 flag also gives me a better experience.

Exact the same obersavtions here.

Good mention!

I recommend caching for metadata on HDD. Here I have a Rclone-instance with my external FTP for that by 10 simultanious connections.

So, my next step will be to monitor it (I haven't been able to watch anything 'large' lately) and if/when it happens again, I'll grep for the strings you mentioned.

The one i gave you is my "multi-string-Universal" observation command

I am not sure requests are going above API limits, I mean, only one stream being watched at the time this happens. But this is definitely something I will change next time it happens.

If you see "pacer" "increasing sleep to XX Minutes" wordings in the log, you know that you have been throttled. All you can do is cooldown and wait...

From the docs I understand this is to list the files quicker, and only/mostly works if the files are in the root directory, which for me they are not, so not sure this would help? I never have issues rescanning libraries.

by toggle this, the directory listing is cached through the whole directory tree. This makes listing quicker, Scanning for Jellyfin even quicker....

[Dalo (google account)]

Keep in mind that you still can be throttled with TPS Limit "3". I observed Onedrive since a year now and still can't make out a magic limit of X Commands, X Amount of Gigs, X Amout of multiple requests. Throttling appears out the blue and when it's there you need to calm. What I can say is, that the Limit is not hitting by multiple streams with large chunks. The speed you are uploading and downloading is more relevant (I know this sounds odd but it definitely were my observation results). Limiting speed to 40M in both ways was a good choice either. You can see this in my mount options. I'm running pretty fine with this setup

[jcastill (Jose)]

@dinosm

I am not sure requests are going above API limits, I mean, only one stream being watched at the time this happens. But this is definitely something I will change next time it happens.

Problem is OneDrive has some very weird unpredictable throttling events. They also depends on the "mood" they are, so if it's too busy they will throttle way more than if their infrastructure is idle.

@dinosm

From the docs I understand this is to list the files quicker, and only/mostly works if the files are in the root directory, which for me they are not, so not sure this would help? I never have issues rescanning libraries.

--vfs-refresh will warm the cache with all the folder content. --onedrive-delta will use the delta API that is not as easily throttled as the regular list API command. it basically returns the whole drive contents vs having rclone scan folder by folder. The problem is that if you mount a subfolder, it might be very inefficient to pull the whole drive folder for a couple of files, but if you are mounting the whole drive, it will be faster and it's OneDrive recommended way of loading the drive structure.

@dinosm

That's why I am asking here . See, such a weird find, how did you figure that out? This is another one for my list in case the problem persists.

Some time ago, I noticed multiple issues with OneDrive, troubleshooting I noticed some streams were timing out with an http2 error, disabling it helped a lot. Also I was getting a terrible route using IPv6, while IPv4 route was way better, so I also binded to IPv4 for better performance (not related to throttling, but speeds increased significantly)

@dinosm

Hmmm maybe I don't understand the cache correctly. I assumed that the cache helps a) because it fetches the next few seconds from the remote so they play from the server directly (which is faster/more reliable) and b) I can rewind much faster. But why would I need to set a bigger vfs cache size?

You got the idea of cache correctly, the problem is how OneDrive works. OneDrive remote cannot stream, so multiple things, first OneDrive is very inefficient resuming downloads, so if you try to resume a video halfway and it's not in the cache, it will reload the whole file to the cache before resuming play. If you add this to potential throttling events you can be in for a long wait.
Also, OneDrive does not support stream uploads, that means that if you use your rclone mount to also upload files, it will need to cache the whole file to know the complete size and details before upload can begin. So you will have problems trying to upload any file bigger than 5gb. So ideally you want to at least be able to cache a couple dozen files (based on the size of files you manage) so you know you have enough cache for everything and not run into problems or errors.

[jcastill (Jose)]

I have experienced the same as Dalo mentions. TPS limit of 3 helps most of the time and gives a doable experience. But overall Microsoft loves throttling and increases time penalty exponentially. This is a huge headache when you are organizing or working on your data. Rclone does a terrific job, but it's honestly impossible to keep up with Microsoft erratic policies, as it might work great for years, and one day decide to throttle you with no clear understanding or any detail on what triggered the throttle.

[Dalo (google account)]

@jcastill

--vfs-refresh will warm the cache with all the folder content. --onedrive-delta will use the delta API that is not as easily throttled as the regular list API command.

I personally only know the way with --vfs-refresh.

As far as I know --onedrive-delta is a mount flag and it must be enabled alongsinde with --fast-list?

Also I was getting a terrible route using IPv6, while IPv4 route was way better, so I also binded to IPv4 for better performance (not related to throttling, but speeds increased significantly)

exact the same behaviour here. I thought it was my ISPs fault. That's why I removed --bind 0.0.0.0 from the Mount-Code above. Thanks for the confirmation.

[jcastill (Jose)]

@Dalo

I personally only know the way with --vfs-refresh.

As far as I know --onedrive-delta is a mount flag and it must be enabled alongsinde with --fast-list?

--vfs-refresh is only a mount flag. - -onedrive-delta enables - -fast-list support in OneDrive but has some disadvantages as it pulls the whole drive structure and not just the path you are working with. So depending on your data distribution and what you are trying to accomplish it might actually slow things down. But if you are mounting the root folder of your OneDrive and have a big amount of folders and files it can speed things significantly
For example a vfs-refresh withould delta takes 15 minutes in my drive, while it takes 2 minutes with delta. Yet if I just want to work with a folder with a couple of files disabling delta gives me a 10s speed vs the 2 minutes from delta (as it needs to pull the whole drive content)

Hope this makes sense and helps out!

[Dalo (google account)]

Thank you for the quick info.

Do i need --fast-list in the mount options anyhow, when I add --onedrive-delta

[kapitainsky]

@Dalo

Do i need --fast-list in the mount options anyhow, when I add --onedrive-delta

When you add you will see in the log:

NOTICE: --fast-list does nothing on a mount

[jcastill (Jose)]

@Dalo

Do i need --fast-list in the mount options anyhow, when I add --onedrive-delta

As mentioned by kapitainsky, --fast-list does nothing on a mount. Rclone will try to use the best method and implies --fast-list in some commands (don't remember which) but for sure vfs-refresh is one of them that will always (if possible) use fast-list, so no need to supply the flag. Now, if you do not use onedrive-delta flag, it will not use fast-list in OneDrive, while supplying the flag will allow rclone to use it, with OneDrive implementation limitation.

Hope this makes sense

[Dalo (google account)]

I'm using the whole root directory, so --vfs-refresh and --onedrive-delta makes sense. Browsing through the directories is fast as hell

[dinosm]

@jcastill

if you try to resume a video halfway and it's not in the cache, it will reload the whole file to the cache before resuming play

Is this the same as starting a random video from, say 4/5 in (so near the end of it)? I just did that to test with an 8GB video. It resumed (or FFed) to the point I set within 8 seconds. I don't think it could have downloaded the roughly 6GB it had to (in order to get to the part I was resuming it on) within 8s so I am not sure it works as you describe, unless I'm missing something.

Thanks for the other information. Your mentions of 'weird unpredictable throttling' makes sense with what I have experienced. The single common factor is freezing after a while of streaming high-bitrate files, other than that it's pretty random.

@jcastill

depending on your data distribution and what you are trying to accomplish it might actually slow things down. But if you are mounting the root folder of your OneDrive and have a big amount of folders and files it can speed things significantly

Ok, as one of my favourite YouTubers says, I am a bearer of very little brain, so let me try to massively simplify this so I can get it right.
When you (and the docs) say "mounting the root folder of OneDrive" (and I think the docs mention something like "or a folder close to the root"), do you mean that I am mounting say onedrive: as is (which I believe the docs or rclone config itself discourages), or e.g. onedrive:Media/ ?
As opposed to mounting e.g. onedrive:Media/Movies/GoodOnes/.

So, if my mount is onedrive:Media/, and there are no other folders in the OneDrive root, would --vfs-refresh and --onedrive-delta mount options do any good to me when streaming? I currently have zero issues with listing mount contents, and library rescans are done quite quickly (although I guess could be quicker).

@Dalo

Limiting speed to 40M in both ways

Another great find. I guess this makes good sense. I see that if I download, OneDrive maxes out at 80MB/s. I am not sure how rclone handles streaming - does it burst at max speed, then stop, then burst again etc (in which case bwlimit would help), or does it do a steady download at a speed matching the stream bitrate (which, in my head, means bwlimit wouldn't make much difference as I would have to set it to be slightly higher than my highest-bitrare video)?

[Dalo (google account)]

I'm running fine now with the following Mount as a service:

# User service for Rclone mounting
#
# Place in ~/.config/systemd/user/
# File must include the '@' (ex rclone@.service)
# As your normal user, run 
#   systemctl --user daemon-reload
# You can now start/enable each remote by using rclone@<remote>..
#   systemctl --user enable rclone@dropbox
#   systemctl --user start rclone@dropbox

[Unit]
Description=rclone: Remote FUSE filesystem for cloud storage config %i
Documentation=man:rclone(1)
After=network-online.target
Wants=network-online.target
AssertPathIsDirectory=%h/mnt/%i
StartLimitInterval=200
StartLimitBurst=5

[Service]
Type=notify
Environment="RCLONE_CONFIG_PASS=passwordofconfig"
ExecStart= \
  /usr/bin/rclone mount \
    --config=%h/.config/rclone/rclone.conf \
#    --log-level DEBUG \
#    --log-file /root/logs/rclone.log \
    --umask 022 \
--vfs-cache-mode full \
--allow-other \
--bind 0.0.0.0 \
--no-modtime \
--buffer-size 32M \
--cache-dir /root/rclone/cache \
--no-checksum \
--disable-http2 \
--vfs-fast-fingerprint \
--ignore-checksum \
--no-check-certificate \
--checkers 1 \
--tpslimit 3 \
--transfers 1 \
--bwlimit-file 40M:40M \
--low-level-retries 1 \
--onedrive-no-versions \
--onedrive-hash-type none \
--onedrive-chunk-size 250M \
--onedrive-delta \
--dir-cache-time 9999h \
--vfs-refresh \
--vfs-cache-max-age 1m \
--vfs-read-chunk-size 256M \
#--vfs-read-chunk-size-limit 1G \
--poll-interval 1m \
--ignore-size \
--user-agent "ISV|dsadsadas|1d8bdsdsadas9eebec" \
--vfs-cache-min-free-space 40G \
    %i: %h/mnt/%i
ExecStop=/bin/fusermount -u %h/mnt/%i
Restart=always
RestartSec=30

[Install]
WantedBy=default.target

I commented out the log.

[kapitainsky]

--checkers has no effect on the VFS - so it is ignored

@Dalo

--vfs-cache-max-age 1m \

This one is odd. You invalidate whatever managed to be cached after 1 min. Effectively you are almost not using cache at all:) Default is 1h but I would recommend to set it to 9999h. If something is in cache then let it be - rclone will prune old stuff when there is not enough space as you use -vfs-cache-min-free-space 40G. You can also add --vfs-cache-max-size so rclone will try to maintain cache size within set limits.

[Dalo (google account)]

This one is odd. You invalidate whatever managed to be cached after 1 min. Effectively you are almost not using cache at all:) Default is 1h but I would recommend to set it to 9999h. If something is in cache then let it be - rclone will prune old stuff when there is not enough space as you use -vfs-cache-min-free-space 40G . You can also add --vfs-cache-max-size so rclone will try to maintain cache size within set limits.

Thanks for your input, still experimenting with cache options.

The Idea behind this was:
Local Server HDD-Space is low as 240GB, means: free up the cache as soon as possible (when streaming Videofiles ends or transferring ends). I managed this with --vfs-cache-min-free-space 40G so yes, maybe --vfs-cache-max-age can be obsolete or given a longer value.

I have 7 Mounts sheduling this service.... To set --vfs-cache-min-free-space 40G was mandatory.