Using rclone as local cache to reduce HDD spin-up time and power consumption

Help and Support
1

What is the problem you are having with rclone?

I am trying to utilize rclone to cache HDD video files to an SSD. The goal of it is to bypass the spin up time of an HDD and save energy and noise compared to running them 24/7. I am using a local mount to mount the video folder from the HDD to an SSD Folder. This at first seems to work because I can confirm with 'tree' in the SSD folder that is picks up the files from the HDD folder. But I am unable to access these files reliable.

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.1-DEV

  • os/version: ubuntu 22.04 (64 bit)
  • os/kernel: 6.7.0-060700-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.21.1
  • go/linking: dynamic
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

local only

The command you were trying to run (eg rclone copy /tmp remote:tmp)

'rclone -vv mount /mnt/ext/lvm/Movies /mnt/rclone/local/Movies --allow-other --dir-cache-time 30s --vfs-cache-mode full  --cache-dir=/var/cache/rclone'

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

not used in this setup

A log from the command that you were trying to run with the -vv flag

Test1: Reading some bytes from the files with head -c "5M" "/mnt/rclone/local/Movies/******/******" > /dev/null

2024/01/18 13:36:18 DEBUG : : >Statfs: stat={Blocks:5836122872 Bfree:2423962580 Bavail:2423962580 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2024/01/18 13:36:24 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item ***** not removed, freed 0 bytes
2024/01/18 13:36:24 DEBUG : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item 1***** not removed, freed 0 bytes
2024/01/18 13:36:24 INFO  : vfs cache: cleaned: objects 2 (was 2) in use 0, to upload 0, uploading 0, total size 191.992Mi (was 191.992Mi)
2024/01/18 13:36:33 DEBUG : : Statfs:
2024/01/18 13:36:33 DEBUG : : >Statfs: stat={Blocks:5836122872 Bfree:2423962580 Bavail:2423962580 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2024/01/18 13:36:37 DEBUG : /: Lookup: name="*****"
2024/01/18 13:36:37 DEBUG : /: >Lookup: node=*****/, err=<nil>
2024/01/18 13:36:37 DEBUG : *****/: Attr:
2024/01/18 13:36:37 DEBUG : *****/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2024/01/18 13:36:37 DEBUG : *****/: Lookup: name="*****"
2024/01/18 13:36:37 DEBUG : *****/: >Lookup: node=*****/*****, err=<nil>
2024/01/18 13:36:37 DEBUG : *****/*****: Attr:
2024/01/18 13:36:37 DEBUG : *****/*****: >Attr: a=valid=1s ino=0 size=80476221971 mode=-rw-r--r--, err=<nil>
2024/01/18 13:36:37 DEBUG : *****/*****: Open: flags=OpenReadOnly
2024/01/18 13:36:37 DEBUG : *****/*****: Open: flags=O_RDONLY
2024/01/18 13:36:37 DEBUG : *****/*****: newRWFileHandle:
2024/01/18 13:36:37 DEBUG : *****/*****: >newRWFileHandle: err=<nil>
2024/01/18 13:36:37 DEBUG : *****/*****: >Open: fd=*****/***** (rw), err=<nil>
2024/01/18 13:36:37 DEBUG : *****/*****: >Open: fh=&{*****/***** (rw)}, err=<nil>
2024/01/18 13:36:37 DEBUG : &{*****/***** (rw)}: Read: len=16384, offset=0
2024/01/18 13:36:37 DEBUG : *****/*****(0xc0005a6440): _readAt: size=16384, off=0
2024/01/18 13:36:37 DEBUG : *****/*****(0xc0005a6440): openPending:
2024/01/18 13:36:48 DEBUG : : Statfs:
2024/01/18 13:36:48 DEBUG : : >Statfs: stat={Blocks:5836122872 Bfree:2423962580 Bavail:2423962580 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2024/01/18 13:37:18 DEBUG : : Statfs:
2024/01/18 13:37:18 DEBUG : : >Statfs: stat={Blocks:5836122872 Bfree:2423962580 Bavail:2423962580 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2024/01/18 13:37:33 DEBUG : : Statfs:
2024/01/18 13:37:33 DEBUG : : >Statfs: stat={Blocks:5836122872 Bfree:2423962580 Bavail:2423962580 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2024/01/18 13:37:48 DEBUG : : Statfs:
2024/01/18 13:37:48 DEBUG : : >Statfs: stat={Blocks:5836122872 Bfree:2423962580 Bavail:2423962580 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2024/01/18 13:38:03 DEBUG : : Statfs:
2024/01/18 13:38:03 DEBUG : : >Statfs: stat={Blocks:5836122872 Bfree:2423962580 Bavail:2423962580 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>

It keeps in this state forever and never reads the file.
Edit: Without using vfs-cache everything works fine

Test2: I tried to add the library to Jellyfin to see if I can testplay a file

2024/01/18 13:49:59 DEBUG : &{*****(rw)}: Read: len=65536, offset=0
2024/01/18 13:49:59 DEBUG : I*****(0xc000730080): _readAt: size=65536, off=0
2024/01/18 13:49:59 DEBUG : I*****(0xc000730080): openPending:
2024/01/18 13:49:59 DEBUG : /: Lookup: name="*****"
2024/01/18 13:49:59 DEBUG : /: >Lookup: node=*****/, err=<nil>
2024/01/18 13:49:59 DEBUG : *****/: Attr:
2024/01/18 13:49:59 DEBUG : *****/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2024/01/18 13:49:59 DEBUG : *****/: Lookup: name="***** {imdb-tt0446724} [WEBDL-1080p][AAC 2.0]-Oergel.nfo"
2024/01/18 13:49:59 DEBUG : *****/: >Lookup: node=<nil>, err=no such file or directory
2024/01/18 13:49:59 DEBUG : *****/: Lookup: name="movie.nfo"
2024/01/18 13:49:59 DEBUG : *****/: >Lookup: node=<nil>, err=no such file or directory
2024/01/18 13:49:59 DEBUG : /: Lookup: name="*****"
2024/01/18 13:49:59 DEBUG : /: >Lookup: node=*****/, err=<nil>
2024/01/18 13:49:59 DEBUG : *****/: Attr:
2024/01/18 13:49:59 DEBUG : *****/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2024/01/18 13:49:59 DEBUG : *****/: Lookup: name="***** ***** [Remux-2160p][DV HDR10][TrueHD Atmos 7.1]-pmHD.nfo"
2024/01/18 13:49:59 DEBUG : *****/: >Lookup: node=<nil>, err=no such file or directory
2024/01/18 13:49:59 DEBUG : *****/: Lookup: name="movie.nfo"
2024/01/18 13:49:59 DEBUG : *****/: >Lookup: node=<nil>, err=no such file or directory
2024/01/18 13:49:59 DEBUG : /: Lookup: name="*****"
2024/01/18 13:49:59 DEBUG : /: >Lookup: node=*****/, err=<nil>
2024/01/18 13:49:59 DEBUG : *****/: Attr:
2024/01/18 13:49:59 DEBUG : *****/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2024/01/18 13:49:59 DEBUG : I*****: Open: flags=OpenReadOnly
2024/01/18 13:49:59 DEBUG : I*****: Open: flags=O_RDONLY
2024/01/18 13:49:59 DEBUG : I*****: newRWFileHandle:
2024/01/18 13:49:59 DEBUG : I*****: >newRWFileHandle: err=<nil>
2024/01/18 13:49:59 DEBUG : I*****: >Open: fd=I***** (rw), err=<nil>
2024/01/18 13:49:59 DEBUG : I*****: >Open: fh=&{I***** (rw)}, err=<nil>
2024/01/18 13:49:59 DEBUG : &{I***** (rw)}: Read: len=65536, offset=0
2024/01/18 13:49:59 DEBUG : I*****(0xc000a521c0): _readAt: size=65536, off=0
2024/01/18 13:49:59 DEBUG : I*****(0xc000a521c0): openPending:
2024/01/18 13:49:59 DEBUG : It*****: Open: flags=OpenReadOnly
2024/01/18 13:49:59 DEBUG : It*****: Open: flags=O_RDONLY
2024/01/18 13:49:59 DEBUG : It*****: newRWFileHandle:
2024/01/18 13:49:59 DEBUG : It*****: >newRWFileHandle: err=<nil>
2024/01/18 13:49:59 DEBUG : It*****: >Open: fd=It***** (rw), err=<nil>
2024/01/18 13:49:59 DEBUG : It*****: >Open: fh=&{It***** (rw)}, err=<nil>
2024/01/18 13:49:59 DEBUG : &{It***** (rw)}: Read: len=65536, offset=0
2024/01/18 13:49:59 DEBUG : It*****(0xc000730100): _readAt: size=65536, off=0
2024/01/18 13:49:59 DEBUG : It*****(0xc000730100): openPending:
2024/01/18 13:49:59 DEBUG : *****: Open: flags=OpenReadOnly
2024/01/18 13:49:59 DEBUG : *****: Open: flags=O_RDONLY
2024/01/18 13:49:59 DEBUG : *****: newRWFileHandle:
2024/01/18 13:49:59 DEBUG : *****: >newRWFileHandle: err=<nil>
2024/01/18 13:49:59 DEBUG : *****: >Open: fd=***** (rw), err=<nil>
2024/01/18 13:49:59 DEBUG : *****: >Open: fh=&{***** (rw)}, err=<nil>
2024/01/18 13:49:59 DEBUG : &{***** (rw)}: Read: len=65536, offset=0
2024/01/18 13:49:59 DEBUG : *****(0xc000730180): _readAt: size=65536, off=0
2024/01/18 13:49:59 DEBUG : *****(0xc000730180): openPending:
2024/01/18 13:49:59 DEBUG : *****: Open: flags=OpenReadOnly
2024/01/18 13:49:59 DEBUG : *****: Open: flags=O_RDONLY
2024/01/18 13:49:59 DEBUG : *****: newRWFileHandle:
2024/01/18 13:49:59 DEBUG : *****: >newRWFileHandle: err=<nil>
2024/01/18 13:49:59 DEBUG : *****: >Open: fd=***** (rw), err=<nil>
2024/01/18 13:49:59 DEBUG : *****: >Open: fh=&{***** (rw)}, err=<nil>
2024/01/18 13:49:59 DEBUG : &{***** (rw)}: Read: len=65536, offset=0
2024/01/18 13:49:59 DEBUG : *****(0xc000da8040): _readAt: size=65536, off=0
2024/01/18 13:49:59 DEBUG : *****(0xc000da8040): openPending:
2024/01/18 13:49:59 DEBUG : *****/***** ***** [Remux-2160p][DV HDR10][TrueHD Atmos 7.1]-pmHD.mkv: Open: flags=OpenReadOnly
2024/01/18 13:49:59 DEBUG : *****/***** ***** [Remux-2160p][DV HDR10][TrueHD Atmos 7.1]-pmHD.mkv: Open: flags=O_RDONLY
2024/01/18 13:49:59 DEBUG : *****/***** ***** [Remux-2160p][DV HDR10][TrueHD Atmos 7.1]-pmHD.mkv: newRWFileHandle:
2024/01/18 13:49:59 DEBUG : *****/***** ***** [Remux-2160p][DV HDR10][TrueHD Atmos 7.1]-pmHD.mkv: >newRWFileHandle: err=<nil>
2024/01/18 13:49:59 DEBUG : *****/***** ***** [Remux-2160p][DV HDR10][TrueHD Atmos 7.1]-pmHD.mkv: >Open: fd=*****/***** ***** [Remux-2160p][DV HDR10][TrueHD Atmos 7.1]-pmHD.mkv (rw), err=<nil>
2024/01/18 13:49:59 DEBUG : *****/***** ***** [Remux-2160p][DV HDR10][TrueHD Atmos 7.1]-pmHD.mkv: >Open: fh=&{*****/***** ***** [Remux-2160p][DV HDR10][TrueHD Atmos 7.1]-pmHD.mkv (rw)}, err=<nil>
2024/01/18 13:49:59 DEBUG : *****/*****: Open: flags=OpenReadOnly
2024/01/18 13:49:59 DEBUG : *****/*****: Open: flags=O_RDONLY
2024/01/18 13:49:59 DEBUG : *****/*****: newRWFileHandle:
2024/01/18 13:49:59 DEBUG : *****/*****: >newRWFileHandle: err=<nil>
2024/01/18 13:49:59 DEBUG : *****/*****: >Open: fd=*****/***** (rw), err=<nil>
2024/01/18 13:49:59 DEBUG : *****/*****: >Open: fh=&{*****/***** (rw)}, err=<nil>
2024/01/18 13:49:59 DEBUG : &{*****/***** ***** [Remux-2160p][DV HDR10][TrueHD Atmos 7.1]-pmHD.mkv (rw)}: Read: len=65536, offset=0
2024/01/18 13:49:59 DEBUG : *****/***** ***** [Remux-2160p][DV HDR10][TrueHD Atmos 7.1]-pmHD.mkv(0xc001296040): _readAt: size=65536, off=0
2024/01/18 13:49:59 DEBUG : *****/***** ***** [Remux-2160p][DV HDR10][TrueHD Atmos 7.1]-pmHD.mkv(0xc001296040): openPending:
2024/01/18 13:49:59 DEBUG : *****/***** {imdb-tt0446724} [WEBDL-1080p][AAC 2.0]-Oergel.mkv: Open: flags=OpenReadOnly
2024/01/18 13:49:59 DEBUG : *****/***** {imdb-tt0446724} [WEBDL-1080p][AAC 2.0]-Oergel.mkv: Open: flags=O_RDONLY
2024/01/18 13:49:59 DEBUG : *****/***** {imdb-tt0446724} [WEBDL-1080p][AAC 2.0]-Oergel.mkv: newRWFileHandle:
2024/01/18 13:49:59 DEBUG : *****/***** {imdb-tt0446724} [WEBDL-1080p][AAC 2.0]-Oergel.mkv: >newRWFileHandle: err=<nil>
2024/01/18 13:49:59 DEBUG : *****/***** {imdb-tt0446724} [WEBDL-1080p][AAC 2.0]-Oergel.mkv: >Open: fd=*****/***** {imdb-tt0446724} [WEBDL-1080p][AAC 2.0]-Oergel.mkv (rw), err=<nil>
2024/01/18 13:49:59 DEBUG : *****/***** {imdb-tt0446724} [WEBDL-1080p][AAC 2.0]-Oergel.mkv: >Open: fh=&{*****/***** {imdb-tt0446724} [WEBDL-1080p][AAC 2.0]-Oergel.mkv (rw)}, err=<nil>
2024/01/18 13:49:59 DEBUG : /: Lookup: name="I*****"
2024/01/18 13:49:59 DEBUG : /: >Lookup: node=I*****/, err=<nil>
2024/01/18 13:49:59 DEBUG : I*****/: Attr:
2024/01/18 13:49:59 DEBUG : I*****/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2024/01/18 13:49:59 DEBUG : I*****/I***** [Bluray-2160p][HDR10Plus][DTS-HD MA 5.1]-ENDSTATiON.mkv: Open: flags=OpenReadOnly
2024/01/18 13:49:59 DEBUG : I*****/I***** [Bluray-2160p][HDR10Plus][DTS-HD MA 5.1]-ENDSTATiON.mkv: Open: flags=O_RDONLY
2024/01/18 13:49:59 DEBUG : I*****/I***** [Bluray-2160p][HDR10Plus][DTS-HD MA 5.1]-ENDSTATiON.mkv: newRWFileHandle:
2024/01/18 13:49:59 DEBUG : I*****/I***** [Bluray-2160p][HDR10Plus][DTS-HD MA 5.1]-ENDSTATiON.mkv: >newRWFileHandle: err=<nil>
2024/01/18 13:49:59 DEBUG : I*****/I***** [Bluray-2160p][HDR10Plus][DTS-HD MA 5.1]-ENDSTATiON.mkv: >Open: fd=I*****/I***** [Bluray-2160p][HDR10Plus][DTS-HD MA 5.1]-ENDSTATiON.mkv (rw), err=<nil>
2024/01/18 13:49:59 DEBUG : I*****/I***** [Bluray-2160p][HDR10Plus][DTS-HD MA 5.1]-ENDSTATiON.mkv: >Open: fh=&{I*****/I***** [Bluray-2160p][HDR10Plus][DTS-HD MA 5.1]-ENDSTATiON.mkv (rw)}, err=<nil>
2024/01/18 13:49:59 DEBUG : /: Lookup: name="*****"
2024/01/18 13:49:59 DEBUG : /: >Lookup: node=*****/, err=<nil>
2024/01/18 13:49:59 DEBUG : *****/: Attr:
2024/01/18 13:49:59 DEBUG : *****/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2024/01/18 13:49:59 DEBUG : *****/***** *****: Open: flags=OpenReadOnly
2024/01/18 13:49:59 DEBUG : *****/***** *****: Open: flags=O_RDONLY
2024/01/18 13:49:59 DEBUG : *****/***** *****: newRWFileHandle:
2024/01/18 13:49:59 DEBUG : *****/***** *****: >newRWFileHandle: err=<nil>
2024/01/18 13:49:59 DEBUG : *****/***** *****: >Open: fd=*****/***** ***** (rw), err=<nil>
2024/01/18 13:49:59 DEBUG : *****/***** *****: >Open: fh=&{*****/***** ***** (rw)}, err=<nil>
2024/01/18 13:50:03 DEBUG : : Statfs:
2024/01/18 13:50:03 DEBUG : : >Statfs: stat={Blocks:5836122872 Bfree:2423962580 Bavail:2423962580 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2024/01/18 13:50:18 DEBUG : : Statfs:
2024/01/18 13:50:18 DEBUG : : >Statfs: stat={Blocks:5836122872 Bfree:2423962580 Bavail:2423962580 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2024/01/18 13:50:33 DEBUG : : Statfs:
2024/01/18 13:50:33 DEBUG : : >Statfs: stat={Blocks:5836122872 Bfree:2423962580 Bavail:2423962580 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2024/01/18 13:50:40 DEBUG : I*****: vfs cache: checking remote fingerprint "482222101,2023-12-01 03:45:34.252941693 +0000 UTC,94f5b3f9ca38a3b5cdd95b8fb65aaf4e" against cached fingerprint ""
2024/01/18 13:50:40 DEBUG : I*****: vfs cache: truncate to size=482222101
2024/01/18 13:50:48 DEBUG : : Statfs:
2024/01/18 13:50:48 DEBUG : : >Statfs: stat={Blocks:5836122872 Bfree:2423962580 Bavail:2423962580 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>

it is the same as before that suddenly on one file it stops reading and never continues

2

Can you explain what you mean?

Also this is not clear at all... your SSD cache is mounted in /var/cache/rclone? Otherwise not sure what you are doing.

Your mount command looks ok but if you want to avoid constantly reading from your HDD you should set much longer cache validity, e.g.:

--dir-cache-time 9999h --vfs-cache-max-age 9999h

With --dir-cache-time 30s your dir cache expires after 30s and rclone will check your HDD for changes.

Also you should set --vfs-cache-max-size. Otherwise all free space in your cache location will be used at some stage.

Add --vfs-refresh to read all directories data upfront after mount.

all together it should be something like this:

rclone mount /mnt/HDD/Movies /path/to/mountpoint --cache-dir=/path/to/SSD/cache --vfs-cache-mode full --dir-cache-time 9999h --vfs-cache-max-age 9999h --vfs-cache-max-size 100G --vfs-refresh

You can also adjust chunk reading to make sure that all film is cached on SSD when you attempt to access it. Otherwise reads are in small chunks.

For example:

--vfs-read-chunk-size 5G

3

First of all. I am sorry if it was unclear what I meant. I am not a native english speaker but I try to clarify what I want to achieve.

It means that some files are not read from the mount.
E.g. if I do head -c "5M" "/path/to/HDD/file" I will get the first 5MB of the file. But if I do the same with head -c "5M" "/path/to/rclonemount/file" I sometimes get the same result (as expected) and sometimes nothing happens at all. It will unsucesfully try to read the file until I manually end the command in the terminal (ctrl + c). That is what I meant in my section about Test1.
It basically stays 2024/01/18 13:36:37 DEBUG : *****/*****(0xc0005a6440): openPending: here forever.

Additional Information: After further testing I can confirm that it only happens when using vfs-cache. Without cache everything works fine.

My SSD is mounted in '/' so the path '/mnt/rclone/local' is on the SSD but my path '/mnt/ext/lvm/Movie' is on the HDD. I will then point my Mediaserver to '/mnt/rclone/local' where I always have the first MBs of the movie cached. This means that the movie is directly played because the beginning is read from the cache. You only need enough cache to skip the time until the HDD is fully spinned up. I already have this setup from an old cloud mount I had. I am happy to share and discuss it in a tutorial if I can get it running with an HDD.

True like mentioned I already have the correct mount command for my usecase. I will share and explain it seperately after some testing if every thing works.

I tested this and for me it wasn't the case. I thought that maybe Linux caches the directory structure somewhere, so that it is not read directly from the HDD. My HDD were spinned down even when I used the 30s cache expiry.

4

rclone has two vfs caches. the vfs dir cache is stored in memory.
so it would make sense that the HDD would spin down.

can check out my overview of the two vfs caches at
to be clear, there are two caches.

1 Like
5

Solved the problem with adding --vfs-fast-fingerprint. The normal fingerprinting was too slow because it needs to read the whole file as mentioned in the docs. Will add a tutorial soon on how to use a cache for HDD spindown with rclone.

6

Nice. Yes please share your findings

7

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.