Excuse me,i hava a question
For example the following video example，No problem with right clicking on other types of files，but the exe is so slow
My reclone version:1.60
mount command: .\rclone.exe mount tx:/ Z: --network-mode --file-perms 0777
Hi bruce,
I guess it is because the exe file is being downloaded. This can be due to one of your Explorer addins, your antivirus or Windows finding digital signatures or similar in the file. I would expect the Explorer to unfreeze once the file has been downloaded.
You could add -vv to you mount command to see more details (if manually started in a terminal).
You may see better performance if adding --vfs-cache-mode=full.
More information here:
https://rclone.org/commands/rclone_mount/#vfs-file-caching
Thank you Ole,I think I'm going to give up accessing mounted exe files on windows
I installed a blank windows virtual machine and turned off windows defender, it is still slow until it freezes
The --vfs-cache-mode=full parameter might work, but it also takes time to load the cache
Hi Ole,
I think you are right, the same running environment to access the exe is fast on linux but slow on windows, but strangely, when I use raidrive software to mount it doesn't have this problem, do you know my windows needs what settings to make to skip freeze?
Hi Bruce,
I have found a way to reproduce your example in a way that seems independent from backend and network speed:
rclone mount --network-mode --bwlimit=1M -vv myOneDrive:testfolder/ Z:
The test folder has this well-known content:
rclone.exe lsl OneDrive:testfolder/
2064016 2022-11-02 17:24:07.000000000 rclone-v1.60.0-windows-amd64/README.html
1698677 2022-11-02 17:24:07.000000000 rclone-v1.60.0-windows-amd64/README.txt
50587 2022-11-02 17:24:07.000000000 rclone-v1.60.0-windows-amd64/git-log.txt
1918740 2022-11-02 17:24:07.000000000 rclone-v1.60.0-windows-amd64/rclone.1
47669760 2022-11-02 17:24:07.000000000 rclone-v1.60.0-windows-amd64/rclone.exe
To reproduce, first open Explorer and select the "rclone-v1.60.0-windows-amd64" folder and press Enter. Then select an ordinary file in the listing while being careful not to hover the mouse over the line with rclone.exe. Then use the arrow keys to select rclone.exe and wait a few seconds, then I see a period of approximately 30 seconds where the cursor becomes busy and Explorer shows "Not Responding" on and off.
I am using Windows 11 without any fancy addons and can confirm that it doesn't help to turn off thumbnails or built-in antivirus. It does improve the experience to add --vfs-cachemode=full, but it still takes 30 seconds from selection before you can right-click without freezing Explorer. Seems like Windows insists on reading app. 15MB which causes rclone to toggle between reading the head and tail of the exe file (if I am reading the log correctly).
It puzzles me that Windows does this in --network-mode, so there may be an issue with rclone. I have not been able to find a way to verify that Windows indeed sees the mount as networked (that is possibly slow with high latency and drop outs).
Perhaps somebody with better insight can reproduce and shed some light on this.
2 Likes
yes, that is strange.
did a test, not using rclone.
-
net useto an azure files storage account, which supports smb.
net use z: \\redacted.file.core.windows.net\zork /user:zork redacted - right-mouse-click a .exe
and i got the same result/behavior as rclone, there is a long delay as windows downloads chunks from network.
odds are simply clicking to disable anti-vrius does not stop windows from certain actions, such as scanning .exe
and i did a bunch of rclone mount variations, did not make a real difference.
so perhaps the questions is why the .exe delay behavior occurs with rclone and drives from net use
but NOT raidrive.
1 Like
@asdffdsa
Hi Ole, jojothehumanmonkey
I think it is really reading my exe file information that is causing it to become slow.
Maybe it has something to do with the file manager, so I tried another file manager: files, its right-click menu doesn't read extra file info, it's fast as I thought.
Maybe this is also related to the way rclone is mounted, that as ole guess, although it uses the --network-mode parameter, I think Windows does not really regard the mount as networking, like in the picture below, the left is the windows self The dav connection with the belt, right-click it is fast, the right is mounted through rclone, it is slow, and through the properties view, you can see that it has a lot more information than the way windows mounts the network disk.
@asdffdsa @Ole
When I used windows to build smb network disks, I found that unlike windows mounting webdav, it read more properties, and unlike rclone cacheless mounting smb, the right click was only delayed for a moment. I wondered why windows mounting was faster than rclone. Then I tried to monitor their traffic, and found that when windows mounted smb in a native way, the exe file was still being downloaded by right-clicking, but the traffic was much smaller than that in rclone mode. Considering smb is a native protocol of windows, is there a separate space for storing file information and separate from the local file for windows to download and read?
That is interesting.
You could try mounting rclone via webdav using rclone serve webdav. Use all the same parameters you've been using with rclone mount (eg --vfs-cache-mode). I wonder how that will perform.
I have tried to rclone mount jianguoyun:/ Z: --network-mode --vfs-cache-mode=full by mount, but it is still slow when there is no cache and more information than by windows mount,rclone's mount method can't seem to stop explorer from downloading file and getting info
I have tried rclone serve webdav --bwlimit=1M and see somewhat similar behavior, except it seems like Windows sometimes start reading the exe file as soon as the folder is opened and therefore may be ready if checking a few other files before right-clicking the exe (depending on bandwidth).
I can make it fast by enabling "Always show icons, never thumbnails" in the Explorer View options. I guess this eliminates the need to extract the icon from the exe.
This trick doesn't help (much) when using rclone mount --network-mode --vfs-cache-mode=full --bwlimit=1M
Good tip.
Why not do you think?
Thanks, but seems like my conclusion was too hasty.
A more thorough test shows that both webdav and mount will read the executable to extract the icon. This will happen with or without "Show thumbnails instead of icons".
Apparently Windows has some kind of (timed) icon cache that blurred my tests unless each action was performed with minimum 10 minutes delay.
Good question, it made me discover the above and investigate why Explorer behaves so badly when using rclone mount.
This is what Windows requests from rclone each time you hover, select or right-click on an executable like rclone.exe:
DEBUG : rclone-v1.60.0-windows-amd64/rclone.exe(0xc0005921c0): _readAt: size=4096, off=0
DEBUG : rclone-v1.60.0-windows-amd64/rclone.exe(0xc0005921c0): _readAt: size=16384, off=47643648
DEBUG : rclone-v1.60.0-windows-amd64/rclone.exe(0xc0005921c0): _readAt: size=1536, off=47668224
That is 4K from the start and 17K near the end of the file (probably the icon resource). This amount of data would typically be transferred quicker than the human reaction/detection time.
rclone however has a default --buffer-size of 16Mi, so it reads 16MB from the start of the file and another 16 MB from the end. That will roughly take the above mentioned 30 seconds to complete on my simulated 10 mbps connection. That is an effective transfer speed of 5,600 bit per second seen from Windows. This speed is comparable to some of the very first phone modems I used 30+ years ago, so I guess this would cripple any modern (Linux or Windows) program performing small random reads.
With this knowledge I tried setting --buffer-size=0 which effectively reduced the buffer to 32KB (on my machine). This made Explorer nicely responsive, even with thumbnails enabled. I was even able to do reasonable pleasant Explorer navigation on a simulated 1 mbps connection (starting with empty cache):
rclone mount --network-mode --cache-mode=full --buffer-size=0 --bwlimit=100K myOneDrive:testfolder/ Z:
I also tried a few other things (downloading rclone.exe and browsing a folder with photos). It was slow as expected, but I didn't see any negative consequences of the much lower --buffer-size. Performance also seem fine when using the mount with --buffer-size=0 on my full 600mbps bandwidth.
I guess this explains the much better performance seen with the default installation/usage of RaiDrive. RaiDrive probably doesn't try to read (as much) ahead as rclone does with default settings, and it also seems to enable a small read cache by default.
So these are the rclone settings that seems to match the RaiDrive default settings:
rclone mount --network-mode --cache-mode=full --buffer-size=0 ...
@wslrj Does these settings improve/solve your issue?
A closing remark for the technically interested: I also briefly tried the same trick with rclone serve webdav, and that wasn't a success. Seems like the whole file is being downloaded irrespective of --buffer-size, I didn't investigate why or any details. That will be a challenge for somebody else.
More info on --buffer-size here:
https://rclone.org/docs/#buffer-size-size
https://rclone.org/commands/rclone_mount/#vfs-file-buffering
https://rclone.org/commands/rclone_mount/#vfs-cache-mode-full
3 Likes
both webdav and mount will read the executable to extract the icon. This will happen with or without "Show thumbnails instead of icons".
Hmm didn't know that.
For me a 100MB exe responds instantly to both right-clicking and displaying file properties. But perhaps, as you've mentioned, the data is being cached too fast for me to notice. A 700MB exe on the other hand takes about ~1s to respond to a right click, then ~8s to display properties.
Handling that 700MB exe is the first time I've felt any lag while using these mount flags:
--vfs-cache-mode full --use-server-modtime --network-mode
After clearing the cache and adding an additional flag of --buffer-size=0 the 700MB exe responds instantly to a right-click (reduced from ~1s), and takes ~4s to display file properties (reduced from ~8s). That's a really good improvement.
Are there any downsides to reducing the buffer?
Thanks, appreciate the test and confirmation!
I am a bit puzzled by the 4 seconds to display file properties, is it a publicly available executable that I can test?
Not that I am aware of.
A typical pc can easily move data with the responsiveness needed to efficiently utilize networked storage, so I see no need for the additional rclone buffering on mounts.
Please let me know if you experience anything different.
I'm puzzled too. Wish I had other large executables to test.
The exe I used was an old version of ZBrush 4R4 — probably difficult to find now. Below is a GIF of me interacting with it via my Wasabi mount using these flags:
--vfs-cache-mode full --use-server-modtime --network-mode --buffer-size=0
Here's the log from one Right Click > Properties attempt.
For what it's worth I also tested this exe on a local HDD (WD Black 1TB). It responded instantly to a right click and instantly displayed file properties.
The delay on the mounted drive isn't a big problem for me personally, since I don't often interact with executables. But I'd be interested to learn why it happens.
Very useful log - thank you.
Looking at your log, probably the main slowdown is the fact the OS opens and closes the file 8 times!
Each time the OS opens the file rclone checks it is the same on the remote end - this takes an HTTP roundtrip. This could potentially be optimised.
The OS reads these chunks out of the file
File Handle Operation Size Read Offset in file
0xc00070e8c0: _readAt: size=4096, off=0
0xc00070e8c0: _readAt: size=16384, off=840192
0xc00070e8c0: _readAt: size=15360, off=872960
0xc000b323c0: _readAt: size=4096, off=0
0xc0004101c0: _readAt: size=32768, off=0
0xc0004101c0: _readAt: size=32768, off=839680
0xc0004101c0: _readAt: size=32768, off=880640
0xc000410280: _readAt: size=32768, off=0
0xc000a4a0c0: _readAt: size=32768, off=0
0xc00070e100: _readAt: size=32768, off=0
0xc000b320c0: _readAt: size=4096, off=0
0xc000b320c0: _readAt: size=16384, off=840192
0xc000b320c0: _readAt: size=15360, off=872960
0xc000b32200: _readAt: size=4096, off=0
0xc000b32200: _readAt: size=16384, off=840192
0xc000b32200: _readAt: size=15360, off=872960
You can see all the different file handles corresponding to the different opens by the OS.
The OS seems to read bits out of the first 32k 8 times! This isn't a problem it should be cached after the first read. It reads another 32k 3 times also.
As the reads are relatively close together you can see rclone just streams the first 900k of the file.
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.RangeSeek from -1 to 0 length -1
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at -1 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.openRange at 0 length 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 32768 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 65536 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 98304 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 131072 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 163840 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 196608 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 229376 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 262144 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 294912 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 327680 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 360448 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 393216 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 425984 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 458752 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 491520 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 524288 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 557056 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 589824 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 622592 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 655360 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 688128 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 720896 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 753664 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 786432 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 819200 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 851968 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 884736 length 32768 chunkOffset 0 chunkSize 134217728
2022/11/10 02:48:08 DEBUG : wasabi/media/apps/3d/zbrush/install/ZBrush_4R4_Installer_WIN.exe: ChunkedReader.Read at 917504 length 32768 chunkOffset 0 chunkSize 134217728
This means that the file bits are in cache very quickly - see the present = true here. Everything is in cache at 02:48:08 (when the read above finished) so <1 second after starting but the OS wasn't done with the file until 02:48:16 so there are 8 seconds of overhead.
2022/11/10 02:48:08 DEBUG : vfs cache: looking for range={Pos:0 Size:4096} in [] - present false
2022/11/10 02:48:08 DEBUG : vfs cache: looking for range={Pos:840192 Size:16384} in [{Pos:0 Size:32768}] - present false
2022/11/10 02:48:08 DEBUG : vfs cache: looking for range={Pos:872960 Size:15360} in [{Pos:0 Size:884736}] - present false
2022/11/10 02:48:12 DEBUG : vfs cache: looking for range={Pos:0 Size:4096} in [{Pos:0 Size:950272}] - present true
2022/11/10 02:48:12 DEBUG : vfs cache: looking for range={Pos:0 Size:32768} in [{Pos:0 Size:950272}] - present true
2022/11/10 02:48:12 DEBUG : vfs cache: looking for range={Pos:839680 Size:32768} in [{Pos:0 Size:950272}] - present true
2022/11/10 02:48:12 DEBUG : vfs cache: looking for range={Pos:880640 Size:32768} in [{Pos:0 Size:950272}] - present true
2022/11/10 02:48:13 DEBUG : vfs cache: looking for range={Pos:0 Size:32768} in [{Pos:0 Size:950272}] - present true
2022/11/10 02:48:14 DEBUG : vfs cache: looking for range={Pos:0 Size:32768} in [{Pos:0 Size:950272}] - present true
2022/11/10 02:48:15 DEBUG : vfs cache: looking for range={Pos:0 Size:32768} in [{Pos:0 Size:950272}] - present true
2022/11/10 02:48:15 DEBUG : vfs cache: looking for range={Pos:0 Size:4096} in [{Pos:0 Size:950272}] - present true
2022/11/10 02:48:15 DEBUG : vfs cache: looking for range={Pos:840192 Size:16384} in [{Pos:0 Size:950272}] - present true
2022/11/10 02:48:15 DEBUG : vfs cache: looking for range={Pos:872960 Size:15360} in [{Pos:0 Size:950272}] - present true
2022/11/10 02:48:16 DEBUG : vfs cache: looking for range={Pos:0 Size:4096} in [{Pos:0 Size:950272}] - present true
2022/11/10 02:48:16 DEBUG : vfs cache: looking for range={Pos:840192 Size:16384} in [{Pos:0 Size:950272}] - present true
2022/11/10 02:48:16 DEBUG : vfs cache: looking for range={Pos:872960 Size:15360} in [{Pos:0 Size:950272}] - present true
Could you do this test again with more accurate timing? You can enable that with --log-format time,microseconds - then I can check which part of the open is causing the problem. I suspect it is the checking remote fingerprint and a more accurate timing will show that.
If it is, I don't think it would be unreasonable to cache these fingerprints. Exactly for how long I don't know but somewhere around --dir-cache-time maybe. Or maybe with its own parameter.
1 Like
Great example, illustration and debug log! 
I have made the same observations as Nick, who probably thinks you are using SFTP (from the thread heading).
I do however know that you are using S3 (Wasabi), so I feel pretty sure you will see much better responsiveness by adding --vfs-fast-fingerprint:
If you are running a vfs cache over
local,s3orswiftbackends then using this flag is recommended.
Source: https://rclone.org/commands/rclone_mount/#fingerprinting
Perhaps @ncw can give a quick rundown of the drawbacks/risks.
Agree, Windows reads the same app. 35K several times. And the 35K comes from different parts of the file so 2-3 reads of the minimum buffer size (32768) from the remote is to be expected.
It does however puzzle me why rclone decides to stream the first 900k of the file, can you briefly give me a hint?
You see two actions in the log: The file is selected at 02:48:08 and Windows checks something (icon) behind the scenes. The right-click is happening at 02:48:12 and may be a combined select and right-click, driving the number of requests up. It takes 4 seconds with all content loaded from cache, so really only checking the fingerprint 13 times - 300ms per fingerprint.
Might be a very good idea to better support applications with high appetite on rapidly sniffing small parts of (executable) files.
However, not sure I understand the fingerprinting method and usage in relation to mounts well enough to comment on the best approach. Do we have a description of rclone's finger printing and especially the method used by --vfs-fast-fingerprint? something between the above link and the code (docs, forum posts, ...)
1 Like
For mount sftp,why right click on exe file is so slow until it freezes - Help and Support - rclone forum.Right clicking on REMOVEDURL .exe files* or folders causes file explorer to freeze , on desktop as well as in the start menu and in a File Explorer window. If an app's added shell extension isn't working properly , it can cause a context menu to be extremely slow or freeze when you right click to ... file which is responsible for the context menu is actually still loaded, enabled and causing the right click to be very slow!
Nice breakdown of the log!
Sure. Performed the same test (right click > properties) a couple more times:
The other flags were kept the same:
--vfs-cache-mode full --use-server-modtime --network-mode --buffer-size=0
I didn't notice that fast fingerprint was recommended for S3 (Wasabi), thanks for pointing that out! There's not much difference for me with it enabled, and I wonder if it's because I'm already using --use-server-modtime. They seem to do similar things, so maybe their effects don't stack very well 
1 Like