Run the command 'rclone version' and share the full output of the command.
rclone v1.57.0
- os/version: Microsoft Windows 10 Pro 1903 (64 bit)
- os/kernel: 10.0.18362.239 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.17.2
- go/linking: dynamic
- go/tags: cmount
Which cloud storage system are you using? (eg Google Drive)
Microsoft OneDrive
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone mount Onedrive_remote: X: \
--vfs-cache-mode full \
--vfs-read-chunk-size 10M \
--vfs-read-chunk-size-limit 20M \
--multi-thread-cutoff 9M \
--multi-thread-streams 4
The rclone config contents with secrets removed.
[Onedrive_remote]
type = onedrive
token = ***
drive_id = ***
drive_type = business
Hi.
I've just discovered rclone and I must say that it had solved so many problems for me.
However, there is one point that I couldn't fully understand, which is how the "--vfs-read-chunk-size 10M" and "--multi-thread-cutoff" flags interact with each other. Understand that the first flag, allows rclone to only request a small part of the file that I'm accessing, the second flag controls the number of simultinous download/upload streams. Does it mean that each "vfs-read-chunk" is downloaded using multiple 4 streams?
I've been playing with the values of these flags without any noticeable effect on the read speed.
already played with that flag and not much changed.
basically, I want to use OneDrive as a media vault (with Plex/Kodi) and stream from it directly. the problem is that if I use a single download stream, it will be slow as heck due to the poor peering between my ISP and OneDrive
based on the rclone changelog, something older than v1.53.0
"Implement partial reads for --vfs-cache-mode full"
and keep in mind, can run different versions of rclone at the same time.
old rusty rclone for the mount for onedrive
new shiny rclone for other rclone mounts and everything else.
So after a lot of testing, here are my findings regarding this issue. I mainly tested two versions: 1.48.0 and 1.52.0.
using 1.52.0:
when using the mount command with the --multi-thread-streams, the flag is completely ignored and the file is slowly streamed into the local storage in chunks using a single thread. Using the --vfs-cache-mode didn't change anything.
using 1.48.0:
when using the mount command with the --multi-thread-streams, the flag is completely ignored and the file is slowly streamed into the local storage in chunks using a single thread. however, when using the --vfs-cache-mode full flage, the file is in fact downloaded using multiple streams but the requested chunks are ignored and the entire file is downloaded to the cache instead.
what I ended up doing, is that I deployed a vps on aws with rclone 1.48.0 using this command rclone serve http Onedrive: --addr :8000 --config rclone.conf --multi-thread-cutoff 100M --multi-thread-streams 8 --vfs-cache-mode full. when the file is first requested, it will be downloaded to the cache of the vps, then I can stream it however I want. the download to the vps uses a Gigabit connection so I just need for a few seconds before the file is served to me.
I also wrote a simple local HTTP server that will download the file sequentially from the vps using multiple threads so I can stream it without a hitch
Yeah, you are comparing very difference versions with cache mode full.
Those old versions download the entire file before streaming anything as the cache mode was completely rewritten.
If you have a front end that is requesting a file sequentially, you can leverage current versions and just use vfs-read-ahead to grab the stream.
When you say poor peering, do you mean packet loss? high latency with no packet loss?
Multi threaded streams I'd imagine has to download the whole file on those versions before streaming it.
If you can read sequentially, you can buffer way ahead on the local disk and that'll prevent any packet loss/latency assuming you can have enough bandwidth to stay ahead and your packet loss ratio is not poor enough to keep ahead.
Albeit, if you got something working now, I may leave it well alone depending on how it works for you.
By poor peering, I mainly mean high latency (around 100ms). I can only download sequentially with a single thread at around 1 MB/s so even with --vfs-read-ahead my internet can't keep up with the stream.
using the HTTP server I talked about, I can almost saturate my 100mbps connection. I wished rclone would do this seamlessly for me, but what I got also works although not very cleanly.
