Union on Windows Slow start times

Help and Support
1

What is the problem you are having with rclone?

Slow start times for films through a Union.

Movies are kept on Local NAS and Drive. Local NAS is mounted as V:\
Drive is mounted as G:
Union (of G: and V:) is mounted as H:

If I navigate and open a movie on Union (H:)
-Movies on Local (or both locations) start in seconds.
-Movies on Drive open in circa 20 seconds

By comparison movies opened in Drive/mount (G:) open in circa 8 seconds (quicker if in VFS-cache i.e. on subsequent plays).

I can't work out what's causing the discrepancy in load times between direct mount and Union Mount

Note: I'm using VLC to test the load times.

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0
- os/version: Microsoft Windows 10 Pro 2009 (64 bit)
- os/kernel: 10.0.19042.1466 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.17.2
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

G: is mounted with BAT file:

Title "Media_VFS G: Mount"
@echo on
"C:\Program Files\rclone\rclone.exe" ^
mount ^
Media_vfs: ^
G: ^
--allow-other ^
--dir-cache-time 5000h ^
--drive-chunk-size 64M ^
--log-level INFO ^
--log-file="C:\Users\robev\.config\rclone\Rclone_Log_Files\Mount_Media_vfs.txt" ^
--vfs-read-chunk-size 32M ^
--vfs-read-chunk-size-limit off ^
--config "C:\Users\robev\.config\rclone\rclone.conf" ^
--poll-interval 10s ^
--cache-dir "C:\zzzRclone Cache\Media_VFS" ^
--vfs-cache-mode full ^
--vfs-cache-max-size 25G ^
--vfs-cache-max-age 5000h ^
--vfs-cache-poll-interval 5m ^
--vfs-read-ahead 2G ^
--log-level=DEBUG ^
--drive-pacer-min-sleep 10ms ^
--drive-pacer-burst 200

Union (H:) is mounted with BAT file:

@echo off
Title "TEST_TEST H: Mount"
@echo on
"C:\Program Files\rclone\rclone.exe" ^
mount ^
TEST_TEST: ^
H: ^
--allow-other ^
--log-file="C:\Users\robev\.config\rclone\Rclone_Log_Files\Media_UNION.txt" ^
--config "C:\Users\robev\.config\rclone\rclone.conf" ^
--log-level DEBUG ^
--cache-dir "C:\zzzRclone Cache\UNION" ^
--dir-cache-time 30s

The rclone config contents with secrets removed.

[Gdrive]
type = drive
client_id = **REDACTED**
client_secret = **REDACTED**
scope = drive
token = {"access_token":"**REDACTED**","token_type":"Bearer","refresh_token":"**REDACTED**","expiry":"2022-02-07T14:27:18.2103586Z"}

[Media_vfs]
type = crypt
remote = Gdrive:Media_Storage_crypt
filename_encryption = standard
directory_name_encryption = true
password = **REDACTED**

[TEST_TEST]
type = union
upstreams = V:\ G:\:ro
cache_time = 10

A log from the command with the -vv flag

2

hi,
--allow-other does nothing on windows, as per the docs and debug log

for the H: mount,
--cache-dir does nothing as vfs-cache-mode=off

3

Been meaning to take that out!

Added it when I added the "--dir-cache-time 30s" as I wasn't sure if that went to Memory or disk. I'm assuming Memory?

I've removed both and still get the same result.

4

yeah, can be confusing, take a read of my post
https://forum.rclone.org/t/status-about-using-rclone-for-music-storage-playback-in-2021-access-times-improved/27648/34

5

I think I've found a culprit!

--vfs-read-ahead 2G

I just left a video playing (made a brew) came back and reopened it and boom instant playback. Used Wiztree to see actual data size in the sparse files and 2GB was the magic number for instant playback.

When I take the whole line out I'm down to load times that are on PAR with the drive mounted directly. (8'ish seconds first play. And near instant start for subsequent plays)

6

not sure that should be the case,
rclone uses chunked reading and playback should start quickly.

here a media file starts after five seconds.
the mount command is using --vfs-read-ahead=2G

image

7

It's starts in that time if it's I'm in the VFS mount (G: in my case), but from the UNION it is looking like it's that line.
-I've tested it by looking at the size on disk of the file (in the cache folder) if I set the read-ahead greater than that size it opens slowly, if I set it under it open's instantly. Just tested on 10 files.
-If I remove the line completely it functions as normal.

Have to be honest I copied that line (--vfs-read-ahead 2G) from @Animosity022 . Do I understand it correctly that by taking it out, my vfs cache is now just writing what it's using? so I'm a little more susceptible to network issues?

8

Got to do the school run, but I'll recreate the error and post some clean log files later.

9

good questions and just now i was looking at @Animosity022 systemd service files

noticed that for --vfs-read-ahead
rclone-drive.service uses it.
rclone-movies.service does not use it

10

That would be analogous to my setup though:

  • Rclone-movie.service, loads mergerfs = (my) "TEST_TEST H: Mount", loads Union

  • rclone-movies.service loads, mount (Drive/dropbox now) = (my) "Media_VFS G: Mount", loads Drive mount

In both cases --vfs-read-ahead isn't part of the Union/ mergerfs. Just in my case it seems to be messing with the Union.

11

thanks but i was/am hoping that @Animosity022 would see that and reply to my question
and might know why, in your case, --vfs-read-ahead is causing an issue.

12

The only thing vfs-read-ahead does is if you are sequentially reading a file, it'll continue to read further along until the file is closed. It doesn't impact streaming start time or not. I think I was testing it before I moved away from GDrive but I will probably just remove it as it's not really needed and with cache mode full, doesn't add a whole lot of value imo.

13

Thanks both for weighing in. I'll mark the comment to remove "--vfs-read-ahead" as the answer for now then.

If you think its a bug do you want me to recreate and do some fresh logs? Happy to if it helps, let me know.

14

I don't think vfs-read-ahead should/would impact anything.

I don't use Windows myself so not sure offhand.

15

--vfs-read-ahead works fine on windows

16

It's working fine for me on the mount. It's only when I access that mount though a Union it's causing an issue with slow video start.

17

The logs I posted originally were for the Union Mount. Logs from the Drive Mount when I a) play the file through the mount (no issue) b) through the Union (Slow Start)

In both cases I'm navigating via Windows Explorer and opening with VLC.

Open file through Mount pt1
Open file through Mount pt2
Open file through Mount pt3
Open file through Mount pt4

Open file through Mount pt1
Open file through Mount pt2

Only if you're curious.. It's working without the flag, so I'm happy enough :slight_smile:

18

fwiw, in general, for mounts that i stream from,
--- do not use the vfs flie cache.
--- do not use any vfs flags

19

One question related to the set-up, I don't think related to the particular problem, but ain't worth it's own post...

The Cache option when you create the Union in Rclone Config:

cache_time

What is being cached? Is it separate to the vfs cache and --dir-cache-time?

20

never used the union remote
and not sure it matters based on your union remote.

best to start a new topic, fill out the template and ask your question

1 Like