SMB vs rclone mount file operation compatibility

What is the problem you are having with rclone?

I was wondering why SMB seems to have much better (perfect?) file operation support compared to rclone mount without requiring a cache.

For example, JDownloader is a program that downloads files in chunks and stitches them together after downloading. On Windows, mounting an SMB share allows this program to work perfectly without any caching on disk. Using rclone mount, this would require a VFS cache that is set to writes/full. The problem with this is that when downloading hundreds of GB or TB of data using rclone mount increases the wear (TBW) on the SSD while using SMB doesn't cause any wear at all.

Can anyone briefly explain the technical reason for this?

I'd imagine trying to compare SMB which is a specific use case for sharing via Windows vs a rclone mount, which is fuse based, supports a plethora of backends in terms of numerous cloud storage based providers and comparing that to a local shared protocol on a LAN doesn't really seem like comparing apples.

I'm not sure what JDownloaded does as generally, if you do something, you'd share a command, a log, etc and we can see what's going on via the log an answer the question.

Most people use rclone without using any cache mode at all for many things.

If you are download, not sure why you'd 'have' to use a cache mode. The cache modes exist because of what the mounted file system must emulate through the fuse layer.

All the specifics are documented here: rclone mount

They are based on the calls the applications / programs sitting on top of the mount are calling so the cache is only "needed" if a certain program is calling a specific ask for the mount and it requires the cache for emulating a normal mounted disk.

As for wear/tear on SSDs, I've used the same SSD for years for my OS that my box runs on and about 14-16 months for my rclone cache disk. By the time this drive wears out, I'll happily chip in another $75 or so later as for power consumption/speed/etc, it meets my needs.

doesn't really seem like comparing apples.

true, this question was more about getting a brief explanation of the technical reasons

local shared protocol on a LAN

The latest versions of SMB support encryption and can be used over the internet, you can also make an SSH tunnel and use it that way.

I'm not sure what JDownloaded does as generally, if you do something, you'd share a command, a log, etc and we can see what's going on via the log an answer the question.

It uses one or more of the file operations that requires a vfs cache. I don't really care about jdownloader specifically, my question is more about the technical reasons why SMB supports all of these operations without a need for a cache and rclone mount doesn't.

Files can't be opened for both read AND write
Files opened for write can't be seeked
Existing files opened for write must have O_TRUNC set
Files open for read with O_TRUNC will be opened write only
Files open for write only will behave as if O_TRUNC was supplied
Open modes O_APPEND, O_TRUNC are ignored
If an upload fails it can't be retried

If you are download, not sure why you'd 'have' to use a cache mode. The cache modes exist because of what the mounted file system must emulate through the fuse layer.

a straightforward sequential copy wouldn't need a cache, but advanced tools which support multi connection downloads like JDownloader, Internet Download Manager, aria2 (haven't tested) will need a write cache when using rclone mount but not when using an SMB mount

They are based on the calls the applications / programs sitting on top of the mount are calling so the cache is only "needed" if a certain program is calling a specific ask for the mount and it requires the cache for emulating a normal mounted disk.

Yes, you're right. I'm wondering why rclone mount needs the cache to fully emulate a disk and support all operations but SMB doesn't.

As for wear/tear on SSDs, I've used the same SSD for years for my OS that my box runs on and about 14-16 months for my rclone cache disk. By the time this drive wears out, I'll happily chip in another $75 or so later as for power consumption/speed/etc, it meets my needs.

Yes this is true, but downloading many TB with JDownloader or doing some other advanced file operation using SMB causes no wear on my PC's SSD which is preferred compared to rclone mount with a cache

Godspeed if you want to use SMB that way. It's "generally" used in a LAN environment.

I don't know as you've not shown a log or an example of it so that's why I like to see a log and we can see specifically why.

Feel free to share a log with any questions as it makes answering much easier.

It's a fuse mount and any fuse mount has these things as they are used based file systems which is a quite a different pattern than SMB. More technical details on fuse here -> Filesystem in Userspace - Wikipedia

Use the tool that works the best for your need. If SMB meets my needs, I'd use it. I really just find the best tool for the job.

SMB isn't going to mount my Dropbox remote on my Linux box.

Sorry as I'm not fully following this as if you are downloading and writing, you are causing wear on the SSD.

hi,

the rclone vfs file cache is not needed to download files.
DEBUG : file.ext: Open: flags=O_RDONLY

and there are four cache modes, might try
--vfs-cache-mode=writes as it has direct downloads, does not use cache

for that, i use tailscale. it encrypts all data.

I don't know as you've not shown a log or an example of it so that's why I like to see a log and we can see specifically why.

I'm not sure why you keep asking for a log, I'm not trying to make the application work. I'm trying to briefly understand the technical reason why vfs cache writes/full is required for some file operations but SMB supports it natively.

It's a fuse mount and any fuse mount has these things as they are used based file systems which is a quite a different pattern than SMB. More technical details on fuse here

Yes, I took a quick look at some Winfsp docs NTFS Compatibility · WinFsp but don't see an obvious reason why a cache is needed

Use the tool that works the best for your need. If SMB meets my needs, I'd use it. I really just find the best tool for the job.
SMB isn't going to mount my Dropbox remote on my Linux box.

I agree

Sorry as I'm not fully following this as if you are downloading and writing, you are causing wear on the SSD.

I have a PC and a server, I am downloading/writing from the PC to the server over SMB. PC only has a boot SSD, server has HDDs so using SMB for writing files to the server won't cause any wear on the PC's drive.

the rclone vfs file cache is not needed to download files.

for simple downloading this is true, but with advanced tools that use multiple connections (jdownloader, IDM, etc) a cache is needed

--vfs-cache-mode=writes as it has direct downloads, does not use cache

write caching will write files to my drive first before transferring them to the server which is what I want to avoid

for that, i use tailscale. it encrypts all data.

I've tried zerotier, not tailscale, unfortunately my networks are locked down heavily (don't have control) so what happens is that all traffic goes through zerotier bridge servers causing terrible performance and packet loss

port forwarding + SSH tunneling works really well though

I think this back and forth is a bit pointless, maybe NCW can comment on this later this week

I don't know why I keep asking either as a log actually has the data in it and we can explain the log file with real information and share back why it's doing stuff rather than speaking without it.

If you share the log, you can specifically see why and I can explain it by looking at the log file you've created and going through it, but alas, I'll stop asking.

Right as you fight to not give the information I'm asking for so not sure as it goes both ways. I'm trying to help you but you generally refuse any basic requests for more information to help further explain/show/describe your question with data.

This is the correct answer

I'll expand it though.

JDownloader requires the ability to write blocks in the middle of a file. This is something which isn't supported by most cloud providers. You can't upload in the middle of an s3 object or a drive file - you have to upload the whole file again. This is the problem --vfs-cache-mode writes or --vfs-cache-mode full solves. Rclone keeps a local copy on disk so that it can write to the middle of it and upload it when done.

However network file systems like SMB (also SFTP) do support writing in the middle of files. Rclone is not yet clever enough to know this

So it would be possible to add another Backend method WriteAt to the SMB backend and expand the VFS layer to use that to write to the middle of files without --vfs-cache-mode writes.

This is a moderately complicated thing to do though (anything to do with the VFS layer is complicated!) and would only benefit a small number of backends: local, smb, sftp, hdfs so we haven't done it yet.

This would also allow these backends to be destination for --multi-thread-streams which would also be useful.

Thanks for the info, that makes sense

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.