Cap for Buffer Size

Hi All,

Is there any setting to set a cap for the buffer-size ?
I have a 2 GB memory in my VPS and after a while even if i set buffer size as small as1M rclone crashes because the memory is not enough.

Thanks.

You can set buffer size to 0M.

i know but i need the buffer.

I’m not sure what you mean as

--buffer-size 0M

is your flag to set the buffer.

https://rclone.org/commands/rclone_mount/#file-buffering

if you set a buffer size let’s say 10MB, for every open file rclone allocates a memory of 10 MB.
if rclone opens 100 files , it will use 100 * 10 = 1000 MB memory and so on. What i need is a cap for the total memory it uses. When the memory reaches the cap, it should release the memory based on LIFO maybe. i need the buffer for the open files so setting it 0 is not a solution for me.

1 Like

There is no current way to cap the buffer size.

It is the application using the mount that opens the files, not rclone. So rclone mount would have to throw errors at that point or stop buffering files.

A total buffer pool allocation might be nice.

Note that you can use --use-mmap which makes rclone better at returning memory to the OS.

Yes the application i use try to open as many files as it can. So after while it causes rclone to crash if the buffer-size is greater than 0. It would be great to have a total pool allocation. I also tried --use-mmap but result is the same.

Can you set a ulimit on your application?

ulimit -n will limit number of open files.

i will check it.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.