Handling open files

Hi guys,

first things first: Once again a big thanks for the new release!

From time to time I am running out of memory on my server because rclone has opened one file too many times. I was trying to figure out if it is a rclone or an Emby problem and I think it is an issue with Emby.

Anyways, there might be a solution for this on the rclone side… My thought:
Is there a way to limit rclone to open a file only 1 time? My logs sometimes show that a file is opened dozens of times although there is only one Emby client accessing the file.

I am not using rclone cache (might that help?) and the --chache-chunk-no-memory flag did not help here, still ran out of memory…

I hope I could explain myself, in case you have questions please let me know. :slight_smile:

Big thanks once again!

Why is rclone running out of memory? Have you set --buffer-size very large?

Can you use lsof on Emby to see if it does open the file many times? If it does then there isn’t a lot rclone can do about it.

–buffer-size is default, I didn’t change anything

Just did a quick test and while streaming only one file on one client using lsof I could see that the file was open 16 times and that eventually leads to rclone crashing with no memory left.
So, if I got you right there is no way to limit the times a single file is opened, for instance only 2-3 times?

I will contact the people over at Emby, maybe they have a solution. Will report back!

€dit: Do you think rclone cache could solve the problem given that it probably would read from cache?

Old client or something?

Was the same thing.

So you saw Emby had the file open 16 times with lsof. That sounds like a file descriptor leak in Emby…

Probably not - if you are running out of memory with too many files open then I think cache probably won’t help.

The developer over at Emby pointed out that it actually is a known issue with Kodi and not Emby’s fault. Using their official Android App does not cause the problem, only Kodi + Emby for Kodi PlugIn.

Anyways, there doesn’t seem to be anything rclone can do about it.

@Animosity022, yes I had this issue once but it was with a different client (Emby for SamsungTV). I didn’t this combination (Kodi + Emby) would do the same thing.

Nevertheless, big thanks once again for the support. This can be considered “solved”!

1 Like

Great :slight_smile: Glad we have got to the bottom of it!

1 Like