Mount help, caching specific file formats?

What is the problem you are having with rclone?

For some reason my ebook reader doesn't work well with files in a mounted remote if the caching is set to writes. However, setting the cache mode to full makes it work.

I don't want to set the whole mounted remote's cache mode to full though. I just want the caching to affect *.epub files

What is your rclone version (output from rclone version)

rclone v1.49.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10 64-bit

Which cloud storage system are you using? (eg Google Drive)

Google Drive

Hmm, write-cache should normally give you full OS compatibility, but troubleshooting that is something I will consider a different topic.

For your question:
This is a little difficult to do super cleanly, as cache settings will apply to the entire remote.
However, I can suggest some ideas.

The simple way:
Use two remotes.
One write-cached for general use.
One spesifically for .epub with full cache. If you want this remote to only show .epub files you can use --include .epub . If you have a single fodler-path you keep your library it would make sense to mount that directly maybe.
If you wanted to exclude these files from the general remote, you could do that too with a similar --exclude .epub

Complicated, but seamless way:
Do the same as above (using both include and exclude respectively on each), but use rclone Union (or alternatively mergerFS on Linux) to merge the two together. This will show them as one merged disk with the original intact file-structure, but anytime you try to access a .epub the request will go to the underlying full-cache mount - thus providing the best of both worlds. Unfortunately the only way to attach separate filters or other general flags in this scenario currently is to have mount at each endpoint. This may be possible to do just with VFS layers later on (but that is probably a while away still).

This setup currently has some caveats though due to current limitations in Union. Only one of the two unioned remotes (the last one in the union remote definition) can write. In practice this would make you not able to delete .epub files on the unioned remote. Moving them would similarly be problematic. You'd have to do those operations on a non-union remote, so you'd effectively need 2 remotes anyway. It works, but if you often reorganize it's a hassle.

Thankfully - the Union system is due for a rewamp soon that will get rid of most of the significant limitations, so with a little patience (maybe in 1.51 or 1.52 with some luck) this will be a much more practical setup. Alternatively, mergerFS on Linux can do the same job and does not face the same limitations that Union has right now. I would probably suggest you wait for these improvements before trying a solution like that.

Hey thanks for the response!

I weighed the pros and cons of both approach and decided I'll just go the easy route of creating a second remote. However, do I still need to create a new client ID and key for the new remote? Or can I reuse the one I have on the first one?

The main reason I don't want to use full cache is because Windows keeps trying to read meta info on media files making it redownload all the time

It mgiht work (not sure really) but... I would highly suggest you authorize again on the new remote. Just go through the rclone config for it again.

Hehe, yea this is an issue. Doesn't really pertain to full-cache mode though. On many types of media files windows will read most or even all the file to make thumbnails. The best way to avoid this is either:

  • on the mounted disk -> go to properties -> customize. Optimize for "documents" and check the box for apply to all subfolders. This will make windows only try to fetch standard metadata and show lists rather than thumbnails. This should make it not do unnecessary reads - at least unless you specifically change the folder view yourself.
    or
  • Use a third-party file-explorer when you browse your mounted disk. Double commander, Freecommander XE, Explorer++ ect.

The real reason full cache sucks (and I rarely recommend it's use) is that it forces files to fully download before use. It makes it impossible to do partial reads, and impossible to stream files (especially media obviously).

I will haven't figured out how I could make windows scan and save thumbnails to a thumb.db so I could avoid this problem WHILE getting thumbnails too. There seems to be an actual windows bug in effect for thumbnail creation on networked drives. Maybe some day :wink:

I already did this but it still kept downloading them so I switched back to writes but then I have this issue with SumatraPDF and *.epub files. Actually it does open up but after like 5 minutes or so of stalling. And even then I can only read 1-2 pages off te ebook. It won't move to the next pages.

Some applications read files in ways that are really really badly optimized for clouds-storage (as that was rarely a consideration for the devs).
The #1 problem is applications that open and close the file a gazillion times. This is fast on a HDD or SSD, but on a cloud it takes like a second each time. Sometimes this can be solved by using an alternative software application. Unfortunately there is not much rclone can do to fix this directly.

Using a cache-backend sometimes works for these problem but... it's pretty hit or miss to be honest.

It sounds like you probably just want to use 2 mounts for this as I suggested.
Or of course - you can just drag&drop copy the book to local storage when you want to read it. Since books are small this probably only takes 2 seconds. That will not suffer any of the same problems as the reader application trying to access it directly - because the copy will a bulk operation rather than trying to read it one tiny piece at a time.

Regarding windows behaviour:
Windows should not make thumbnails or read the files for extended attributes (like for example "length" for a video) unless you have those rows activated in your view and/or have thumbnail view on. Optimizing doe "documents" means you only ask for the standard attributes that rclone already knows from it's listing data. Using detailed or list view will not ask for thumbnails.

If for reason it STILL won't stop fetching this sort of data then you can either

  • Disable thumbnail generation entirely, either via registry or settings (search google for guide)
    or
  • As I mentioned, use a third-party file-explorer when looking around on your mount. I mentioned a few good ones already. I like Freecommander XE and Double Commander personally, but there are many good ones.

SumatraPDF uses the absolute file path to restore the section where you were last reading the book though. FWIW, I posted on their forums too awhile ago to bring it to the attention of the developer.

Yeah it still does that but I didn't want to go the drastic measure of completely disabling thumbnails altogether and I like the simplicity of Windows Explorer too so my only option right now is to make another remote and hope the developer of SumatraPDF does something about the read issue.

Well, now you know most of the options you have for the problems I think. Let me know if you need more details on any of them :slight_smile:

But yes, the best solution in terms of the problem you face with Sumatra is if the software got fixed to have better (more network friendly) ways of reading and writing data. Any other solution will be a sub-optimimal workaround, or else you have to use another software for the job that works for this scenario.

We've had to dig into such issues quite a few times when it comes to video and audio playback - and the solution is almost always to use software that behaves more network friendly with it's file-access :slight_smile:

Problem is, I don't know any other lightweight reader that supports:

  • pdf
  • mobi
  • epub
  • cbz
  • cbr
  • chm

All in one package

Sorry man, I'm not an authority on ebook readers. Can't really help you much there.
That is probably a question best directed to a forum for ebook entusiasts or something :slight_smile:

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.