Clairifcation about how multi-threaded read works

What is the problem you are having with rclone?

Not really a problem :slight_smile:

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0
- os/version: Microsoft Windows 10 Pro 1903 (64 bit)
- os/kernel: 10.0.18362.239 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.17.2
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Microsoft OneDrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount Onedrive_remote: X: \
    --vfs-cache-mode full \
    --vfs-read-chunk-size 10M \
    --vfs-read-chunk-size-limit 20M \
    --multi-thread-cutoff 9M  \
    --multi-thread-streams 4

The rclone config contents with secrets removed.

type = onedrive
token = ***
drive_id = ***
drive_type = business


I've just discovered rclone and I must say that it had solved so many problems for me.

However, there is one point that I couldn't fully understand, which is how the "--vfs-read-chunk-size 10M" and "--multi-thread-cutoff" flags interact with each other. Understand that the first flag, allows rclone to only request a small part of the file that I'm accessing, the second flag controls the number of simultinous download/upload streams. Does it mean that each "vfs-read-chunk" is downloaded using multiple 4 streams?

I've been playing with the values of these flags without any noticeable effect on the read speed.

your help is greatly appreciated

Best Regards

hello and welcome to the forum,

pretty sure that --multi-thread-streams does not apply to mounts.

but this says that it does if you use --vfs-cache-mode full which I'm using

take a read of this

well, that's unfortunate. I was really happy when I found rclone and it is almost perfect for what I need.

anyhow, thank you very much for your quick reply, I'll keep an eye on that issue and see what happens

might test using --vfs-cache-mode=off/minimal/writes,
as rclone will download direct from remote, and not buffer partial reads to the vfs file cache.

perhaps if you define that, we can offer suggestions.

already played with that flag and not much changed.

basically, I want to use OneDrive as a media vault (with Plex/Kodi) and stream from it directly. the problem is that if I use a single download stream, it will be slow as heck due to the poor peering between my ISP and OneDrive

understood about peering,

fwiw, i use onedrive business as a media vault for a local emby server.
runs great.

and based that link i shared,
if you use an older version of rclone,
then can use --multi-thread-streams

you make me jealous :smiling_face_with_tear:

can you provide me with a specific version?

based on the rclone changelog, something older than v1.53.0
"Implement partial reads for --vfs-cache-mode full"

and keep in mind, can run different versions of rclone at the same time.
old rusty rclone for the mount for onedrive
new shiny rclone for other rclone mounts and everything else.

Sounds like a plan, I'll try it tomorrow, it's 3:00 am here. thanks again for your valuable information

So after a lot of testing, here are my findings regarding this issue. I mainly tested two versions:
1.48.0 and 1.52.0.

using 1.52.0:
when using the mount command with the --multi-thread-streams, the flag is completely ignored and the file is slowly streamed into the local storage in chunks using a single thread. Using the --vfs-cache-mode didn't change anything.

using 1.48.0:
when using the mount command with the --multi-thread-streams, the flag is completely ignored and the file is slowly streamed into the local storage in chunks using a single thread. however, when using the --vfs-cache-mode full flage, the file is in fact downloaded using multiple streams but the requested chunks are ignored and the entire file is downloaded to the cache instead.

what I ended up doing, is that I deployed a vps on aws with rclone 1.48.0 using this command rclone serve http Onedrive: --addr :8000 --config rclone.conf --multi-thread-cutoff 100M --multi-thread-streams 8 --vfs-cache-mode full. when the file is first requested, it will be downloaded to the cache of the vps, then I can stream it however I want. the download to the vps uses a Gigabit connection so I just need for a few seconds before the file is served to me.
I also wrote a simple local HTTP server that will download the file sequentially from the vps using multiple threads so I can stream it without a hitch :smiley:

Yeah, you are comparing very difference versions with cache mode full.

Those old versions download the entire file before streaming anything as the cache mode was completely rewritten.

If you have a front end that is requesting a file sequentially, you can leverage current versions and just use vfs-read-ahead to grab the stream.

When you say poor peering, do you mean packet loss? high latency with no packet loss?

Multi threaded streams I'd imagine has to download the whole file on those versions before streaming it.

If you can read sequentially, you can buffer way ahead on the local disk and that'll prevent any packet loss/latency assuming you can have enough bandwidth to stay ahead and your packet loss ratio is not poor enough to keep ahead.

Albeit, if you got something working now, I may leave it well alone depending on how it works for you.

By poor peering, I mainly mean high latency (around 100ms). I can only download sequentially with a single thread at around 1 MB/s so even with --vfs-read-ahead my internet can't keep up with the stream.

using the HTTP server I talked about, I can almost saturate my 100mbps connection. I wished rclone would do this seamlessly for me, but what I got also works although not very cleanly.

Do you see a lot of packet loss though as well? Latency is somewhat high but should be able to do more than 1MB/s.

No, not really, i have a pretty stable fiber connection, but i guess it just because of where i live (Iraq btw)

By 1MB i mean 1 Mega Byte (just to clarify).

This speed matches what i get when i download from many other sources not just Onedrive

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.