Cap for Buffer Size


#1

Hi All,

Is there any setting to set a cap for the buffer-size ?
I have a 2 GB memory in my VPS and after a while even if i set buffer size as small as1M rclone crashes because the memory is not enough.

Thanks.


#2

You can set buffer size to 0M.


#3

i know but i need the buffer.


#4

I’m not sure what you mean as

--buffer-size 0M

is your flag to set the buffer.

https://rclone.org/commands/rclone_mount/#file-buffering


#5

if you set a buffer size let’s say 10MB, for every open file rclone allocates a memory of 10 MB.
if rclone opens 100 files , it will use 100 * 10 = 1000 MB memory and so on. What i need is a cap for the total memory it uses. When the memory reaches the cap, it should release the memory based on LIFO maybe. i need the buffer for the open files so setting it 0 is not a solution for me.


#6

There is no current way to cap the buffer size.


#7

It is the application using the mount that opens the files, not rclone. So rclone mount would have to throw errors at that point or stop buffering files.

A total buffer pool allocation might be nice.

Note that you can use --use-mmap which makes rclone better at returning memory to the OS.


#8

Yes the application i use try to open as many files as it can. So after while it causes rclone to crash if the buffer-size is greater than 0. It would be great to have a total pool allocation. I also tried --use-mmap but result is the same.


#9

Can you set a ulimit on your application?

ulimit -n will limit number of open files.


#10

i will check it.