So I encounter a regular crash of rclone on OSX that is pretty bad.
Bad as in my mount crashes (and rclone), without freeing up the mount folder, so it is impossible to re mount again at all short of force ejecting the disk and re mounting the disk where the mount folder is, or more simply restarting the computer.
If trying to remount, I get this message in terminal : Device not configured.
Indeed the mount folder does not appear in finder anymore (but you can't create a new folder of the same name). It reappears when restarting.
Would there be a rclone command to finsih the unmount of that folder ?
My setup is :
gdrive(rclone) --> gcrypt(rclone) --> union(rclone) --> mount to folder "mountpoint"
My mount command is :
rclone mount --allow-other --buffer-size=1G --dir-cache-time 96h --vfs-cache-mode writes runion: /Volumes/Raid/mountpoint
I tried to setup a log file but nothing was written on it when rclone crashes. Tried removing the buffer size as i have limited amount of RAM but crash happened even further.
Funnily enough, when I setup a DEBUG level log file, it seems the system would not crash, though I need to further test that.
Crashes usually happen when rclone handles too much data.
It happens when updating the plex library (on radarr/sonarr push) when networks spikes at 500/800 Mbit/s, and the memory used by rclone goes through the roof (6G on 8G available of physical memory). But then again it did not happen 2 days ago when updating the whole library and generating previews when i moved the library to the remote drive.
Or the crash most usually happens when radarr / sonarr copy files to the union mount.
I import a whole lot of missing data this last day so for now resolved to rename some folders so radarr and sonarr can work locally and turned off plex until the terabyte of missing data is downloaded.
In the end, I am simply thinking of installing Linux on my late 2014 i7 mac mini to avoid this instability.
Would that be the simplest thing to do to get a robust system working ?
Thanks for your help !
Have a good sunday everybody.
Well you can see how to manually dismount of the mount page of the docs so that's probably the solution to that but...
rclone sshould not be crashing crashing in the first place so this is what you need to fix. From your description this is almost certainly an out of memory issue - and a 1GB buffer (which is crazy overkill imo) is not helping. This is generally the only reason I hear reports of crashes in rclone.
The actual cause may be in Plex however. Manyor one of the other accessing software. For example with 1g buffer radar might move several files that each eat a full GB of memory. For Plex it has some auto-San and analysis functions were designed to work on local data, and not only will it go nuts opening tons of files at once (all of which get to use the full max size of the buffer - this is Not a shared amount). Some things like deep analysis will also literally read the entire file to do its job, which you almost certainly don't want. You should refer to animosity thread on "recommended settings" for suggestions of what you probably want to disable or at least have manual control over. I'm sure he can help more than I can as I don't use Plex myself.
5-6gb ram usage is not normal and suggests a misconfiguration and memory crash for sure. Sorry for poor spelling. Written from pad
Thanks thestigma, I indeed was being ambitious with the 1G buffer and now i understand the per transfer thing, ahah I damn the day I thought the unupgradeable memory of the mac mini would not be a good deal, ah well I was waiting for it is all ...
I setup everything with a simpler mount to test things out and removing the buffer for 3 hours, no crash for now just copying files with radarr / sonarr, have to test if plex is working correctly.
Looks like I forgot to add one important thing last night.
The other thing you should be aware of for memory purposes is --drive-chunk-size (or chunk_size = 256M if set directly in the remote config).
This also works on a pr-transfer (upload only) basis, so extreme settings can balloon out of control if you aren't clear on what you are doing
This is a trivial 8MB by default, but it is actually well worth to go a little higher than default (much better to use some memory here than going with a huge buffer-size IMO). The reason is that this will reduce TCP saw-toothing on uploads (only) and increase your bandwidth utilization closer to the optimal - on files of a non-trivial size at least. This is a "diminishing returns" sort of setting - so going up to 32 or 64MB provides very significant and noticeable improvements over the default that most systems can easily handle in terms of memory. Going beyond 128MB starts to have little benefit, for the memory cost is large. Not really recommended unless you just happen to have nothing better to use your memory on and you are sure it will never max out.
TLDR: Make sure that (buffer-size + drive-chunk-size) x (maximum # of transfers) remains well within your memory limits and you should be good. These aren't the only things that use memory, but they are the big ones.
For reference the defaults (if my memory serves) are:
buffer-size 16MB
drive-chunk-size 8MB
transfers 4
Thanks again, will test some more in the coming week.
Crash last night emptied alllibraries from sonarr radarr and Plex ahah si day of refreshing everything but removing the buffer size already seems to have stabilized the thing as no crash from this evening, I had read your answer i think in the pi 4 post, but now you give me all the settings I need on a silver platter.
Well, whatever command I mount with with the latest release systematically results in a crash.
The same command is stable with the latest beta ... except i do not seem to get any accentuated filename recognized by Plex, even when they are working properly on the mounted filesystem.
does your memory use still balloon before it crashes? if it's repeatable, please check that because that's usually the cause, and in that case we need to ffigure out why that happens. a debugog -vv of a crash would also be greatly helpful
