Rclone Cache to Ram

Help and Support
1

I currently have chunk_total_size = 32GB set in my config but want to move away from using the same drive that is reading/writing/encrypting the files for rclone. Is there anyway to use the RAM as cache instead of hard drive space. Would something like having the cache in /dev/shm work better than relying on the ram in rclone itself, which may erase cache if it closes?
Would a smaller cache limit playback in anyway? If not, then I could have 2-3GB of cache on an SSD, although it wouldn’t perform as well as the 32GB cache.

2

I should have read the wiki, I see that this happens by default. Does the cache size affect the performance if the network is fast enough to keep up with the file? I am using 10MB chunks but have 10gbps port speed. If the cache is in the ram, can the location on the HDD that it stores itself in (.cache/rclone) cause a bottleneck if the drive is slow or under i/o load?

3

The cache chunks can get evicted from RAM in which case they will be read back off the disk when required, so yes the drive speed can make a difference. Whether it does or not will depend on your usage I think - you might need to do a bit of testing.

What are you using the cache for?

4

For Plex primarily. I upped the chunk size to 55MB from 10MB and the overall transfer speed from GDrive>Local went up from 7 MB/s to 100 MB/s but then after its halfway done, it starts to drop down to 15 MB/s. Might be the server location or it could possibly be the cache drive i/o being used and it’s slowing down the transfer.

5

Have you looked at some sort of system graph with disk IO on it? Even non SSD disks are capable of 100 MB/s so I think the cache drive IO probably isn’t the bottleneck, but measuring what is going on on the disk should give an idea.

6

I had a different idea after playing around and thought of a way to cache to RAM.

If you use the chunk dir as /dev/shm and than configure chunk-no-memory, you are effectively using RAM only and giving rclone a limit to use so it doesn’t break the system.

My system has 32GB so my /dev/shm is 16GB so I configured rclone to use a max chunk storage of 10GB.

rclone.conf:

chunk_total_size = 10G

My startup:

ExecStart=/usr/bin/rclone mount gmedia: /gmedia \
   --allow-other \
   --dir-cache-time=160h \
   --cache-chunk-path /dev/shm \
   --cache-chunk-no-memory \
   --cache-chunk-size=10M \
   --cache-info-age=168h \
   --cache-workers=6 \
   --cache-tmp-upload-path /data/rclone_upload \
   --cache-tmp-wait-time 60m \
   --buffer-size 0M \
   --syslog \
   --umask 002 \
   --rc \
   --log-level INFO

Give it a whirl!

1 Like
7

Any significant improvements with this config as compared to the older one?

8

It just removes the double use of memory and performs pretty well from what i can tell so far.

It would work well for someone with a lot of memory on a system and slower storage.

9

What kind of difference did you get from having it inside of your /dev/shm folder. I was syncing a file with rsync and saw a potential bottleneck with having it strictly on the ram.

If I ran it without the chunk path and chunk no memory, then it would stay consistently around 30 MB/s, but with those two on and the db path on my ssd, I get 2 MB/s and it stays evenly at that. Something is causing it to slow down, not sure what.