Handling open files

[neik]

Hi guys,

first things first: Once again a big thanks for the new release!

From time to time I am running out of memory on my server because rclone has opened one file too many times. I was trying to figure out if it is a rclone or an Emby problem and I think it is an issue with Emby.

Anyways, there might be a solution for this on the rclone side… My thought:
Is there a way to limit rclone to open a file only 1 time? My logs sometimes show that a file is opened dozens of times although there is only one Emby client accessing the file.

I am not using rclone cache (might that help?) and the --chache-chunk-no-memory flag did not help here, still ran out of memory…

I hope I could explain myself, in case you have questions please let me know.

Big thanks once again!

[ncw (Nick Craig-Wood)]

@neik

From time to time I am running out of memory on my server because rclone has opened one file too many times. I was trying to figure out if it is a rclone or an Emby problem and I think it is an issue with Emby.

Why is rclone running out of memory? Have you set --buffer-size very large?

@neik

Anyways, there might be a solution for this on the rclone side… My thought:

Is there a way to limit rclone to open a file only 1 time? My logs sometimes show that a file is opened dozens of times although there is only one Emby client accessing the file.

Can you use lsof on Emby to see if it does open the file many times? If it does then there isn't a lot rclone can do about it.

[neik]

–buffer-size is default, I didn’t change anything

Just did a quick test and while streaming only one file on one client using lsof I could see that the file was open 16 times and that eventually leads to rclone crashing with no memory left.
So, if I got you right there is no way to limit the times a single file is opened, for instance only 2-3 times?

I will contact the people over at Emby, maybe they have a solution. Will report back!

€dit: Do you think rclone cache could solve the problem given that it probably would read from cache?

[Animosity022]

Old client or something?

Insanely high memory usage

IIRC then I must have quit rclone right before it crashed but apparently, it might have been a tiny bit to late and that caused the mem profile being so small (I was also wondering about that). I will try to create a proper one.

Was the same thing.

[ncw (Nick Craig-Wood)]

@neik

–buffer-size is default, I didn’t change anything

Just did a quick test and while streaming only one file on one client using lsof I could see that the file was open 16 times and that eventually leads to rclone crashing with no memory left.

So you saw Emby had the file open 16 times with lsof. That sounds like a file descriptor leak in Emby...

@neik

€dit: Do you think rclone cache could solve the problem given that it probably would read from cache?

Probably not - if you are running out of memory with too many files open then I think cache probably won't help.

[neik]

@ncw

So you saw Emby had the file open 16 times with lsof. That sounds like a file descriptor leak in Emby…

The developer over at Emby pointed out that it actually is a known issue with Kodi and not Emby's fault. Using their official Android App does not cause the problem, only Kodi + Emby for Kodi PlugIn.

Anyways, there doesn't seem to be anything rclone can do about it.

@Animosity022, yes I had this issue once but it was with a different client (Emby for SamsungTV). I didn't this combination (Kodi + Emby) would do the same thing.

Nevertheless, big thanks once again for the support. This can be considered "solved"!

[ncw (Nick Craig-Wood)]

@neik

Nevertheless, big thanks once again for the support. This can be considered “solved”!

Great Glad we have got to the bottom of it!