Rclone Cache to Ram

I currently have chunk_total_size = 32GB set in my config but want to move away from using the same drive that is reading/writing/encrypting the files for rclone. Is there anyway to use the RAM as cache instead of hard drive space. Would something like having the cache in /dev/shm work better than relying on the ram in rclone itself, which may erase cache if it closes?
Would a smaller cache limit playback in anyway? If not, then I could have 2-3GB of cache on an SSD, although it wouldn’t perform as well as the 32GB cache.

I should have read the wiki, I see that this happens by default. Does the cache size affect the performance if the network is fast enough to keep up with the file? I am using 10MB chunks but have 10gbps port speed. If the cache is in the ram, can the location on the HDD that it stores itself in (.cache/rclone) cause a bottleneck if the drive is slow or under i/o load?

The cache chunks can get evicted from RAM in which case they will be read back off the disk when required, so yes the drive speed can make a difference. Whether it does or not will depend on your usage I think - you might need to do a bit of testing.

What are you using the cache for?

For Plex primarily. I upped the chunk size to 55MB from 10MB and the overall transfer speed from GDrive>Local went up from 7 MB/s to 100 MB/s but then after its halfway done, it starts to drop down to 15 MB/s. Might be the server location or it could possibly be the cache drive i/o being used and it’s slowing down the transfer.

Have you looked at some sort of system graph with disk IO on it? Even non SSD disks are capable of 100 MB/s so I think the cache drive IO probably isn’t the bottleneck, but measuring what is going on on the disk should give an idea.

I had a different idea after playing around and thought of a way to cache to RAM.

If you use the chunk dir as /dev/shm and than configure chunk-no-memory, you are effectively using RAM only and giving rclone a limit to use so it doesn’t break the system.

My system has 32GB so my /dev/shm is 16GB so I configured rclone to use a max chunk storage of 10GB.


chunk_total_size = 10G

My startup:

ExecStart=/usr/bin/rclone mount gmedia: /gmedia \
   --allow-other \
   --dir-cache-time=160h \
   --cache-chunk-path /dev/shm \
   --cache-chunk-no-memory \
   --cache-chunk-size=10M \
   --cache-info-age=168h \
   --cache-workers=6 \
   --cache-tmp-upload-path /data/rclone_upload \
   --cache-tmp-wait-time 60m \
   --buffer-size 0M \
   --syslog \
   --umask 002 \
   --rc \
   --log-level INFO

Give it a whirl!

1 Like

Any significant improvements with this config as compared to the older one?

It just removes the double use of memory and performs pretty well from what i can tell so far.

It would work well for someone with a lot of memory on a system and slower storage.

What kind of difference did you get from having it inside of your /dev/shm folder. I was syncing a file with rsync and saw a potential bottleneck with having it strictly on the ram.

If I ran it without the chunk path and chunk no memory, then it would stay consistently around 30 MB/s, but with those two on and the db path on my ssd, I get 2 MB/s and it stays evenly at that. Something is causing it to slow down, not sure what.