Could you explain what this command would be doing?
I'm a little confused about why this is happening in the first place, so I'm being cautious about any further changes I make
that would unmount that folder before executing ExecStart
1 Like
Ah I see. Thanks, I'll give it a try. Not that this helps explain what the cause of the issue is.
The problem seems to be the mount is being called multiple times, but I don't understand why 
- are you still using the beta or did you start using latest stable v1.53.1?
- for what it is worth, you have the most complex setting i have seen.
No, I jumped back to the main branch as it has the VFS cache update now
Complex? ha ha, it doesn't feel that way for me. It's actually proved helpful in troubleshooting this as I was able to turn docker off completely, confirming none of those 12 pieces of software are related to this issue.
So before I run the updated service file, am I implementing it correctly here?
[Unit]
Description=Plex-Mount Service
[Service]
Type=notify
Environment=RCLONE_CONFIG=//home/craftyclown/.config/rclone/rclone.conf
KillMode=none
RestartSec=5
ExecStartPre=-/bin/fusermount -uz /home/craftyclown/Skull \
ExecStart=/usr/bin/rclone mount "RichFlixCrypt:/Encrypted/" /home/craftyclown/Skull \
--allow-other \
--buffer-size 256M \
--dir-cache-time 72h \
--log-level INFO \
--log-file /home/craftyclown/logs/PlexMount.log \
--poll-interval 15s \
--timeout 1h \
--umask 002 \
--cache-dir=/home/craftyclown/rclone_cache \
--vfs-cache-mode full \
--vfs-cache-max-size 100G \
--vfs-cache-max-age 24h \
--tpslimit 2
ExecStop=/bin/fusermount -uz /home/craftyclown/Skull
Restart=on-failure
User=craftyclown
Group=craftyclown
[Install]
WantedBy=multi-user.targeti am not a linux expert and perhaps it does not matter.
- there is a leading
-at the start of the command but i am sure @random404 knows what he is doing. - there is a trailing slash at the end of the one line command
as for the systemd file, test it and see what happens.
Oooh, good eyes! I didn't spot that pesky -
Sadly that didn't fix things
You must keep the - at the beginning of the command... Just delete the \ at the end
Thanks, yes I did it exactly as you posted it. Unfortunately it didn't fix the problem.
Also although the script runs and the service starts, there is an error and an exit code. Is that a sign that fuse is failing to unmount the folder in the ExecStartPre command?
● Plex-Mount.service - Plex-Mount Service
Loaded: loaded (/etc/systemd/system/Plex-Mount.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2020-09-28 20:16:33 BST; 3h 24min ago
Process: 5929 ExecStartPre=/bin/fusermount -uz /home/craftyclown/Skull (code=exited, status=1/FAILURE)
Main PID: 5931 (rclone)
Tasks: 16 (limit: 38303)
Memory: 9.7G
CGroup: /system.slice/Plex-Mount.service
└─5931 /usr/bin/rclone mount RichFlixCrypt:/Encrypted/ /home/craftyclown/Skull --allow-other --buffer-size 256M --dir-cache-time 72h --log-level INFO --log-file />
Sep 28 20:16:30 KingKong systemd[1]: Starting Plex-Mount Service...
Sep 28 20:16:30 KingKong fusermount[5929]: /bin/fusermount: entry for /home/craftyclown/Skull not found in /etc/mtab
Sep 28 20:16:33 KingKong systemd[1]: Started Plex-Mount Service.perhaps, there was never a mount for the fuse to unmount?
perhaps for the ExecStartPre run a script.
and add a command like mount > mounts.file as the first line.
then you can see that mounts are active before rclone mount is run.
and perhaps a ps -aux > ps.file
That seems likely
Sorry, I'm not quite sure I understand. My linux skills aren't the greatest I'm afraid
me too, 
for ExecStartPre, run a bash script with all those commands in it, not just a single command.
hopefully, we can learn from @random404 the next step.
It doesn't matter if the pre start command fails, it doesn't affect the other command.
The command will just make sure nothing is mounted on the folder before running your mount command. If there's nothing in you folder after you start your service then it's not the pre start command fault
Also your last log doesn't show you using the command exactly like I posted:
ExecStartPre=-/bin/fusermount -uz /YOUR/FOLDER
Ok, that's odd as I definitely ran it with the command you posted. I did initially try it without the -
So maybe I forgot to do a daemon-reload when I changed it.
This aside, I seem to have found a workaround to get my server up and running (most of the time) however it still doesn't address why this is happening and what is causing multiple rclone processes to clash.
I have disabled the .service file and am instead calling it as a required dependency of docker. This does two things;
- it seems that docker was being called quicker than the rclone mount was previously, now it is a dependency of docker it seems to be getting called earlier and ensuring the mount is up before anything else can conflict.
2 If for any reason it doesn't load before the conflicts start, the dependency fails and docker does not start. This is good because I am immediately aware of the issue, as opposed to before when I didn't always notice the mount wasn't properly up.
I found this post online which shows a similar issue to the one I'm having with rclone's left over processes creating problems for Systemd. https://github.com/systemd/systemd/issues/8630
@ncw is there anything that might have changed in rclone recently that could potentially explain processes being left behind after a restart?
It's definitely due to something with your docker setup and launching an extra process and it's not being cleaned up.
Fuse mounts tend to get stuck if you have something kill the process and there is still IO hitting the mount point as the process may terminate or be killed but there is still IO stuck on the mount point so it cannot fully unmount.
This all relates back to docker/systemd interaction and really has nothing to do with rclone unfortunately other than that's the symptom.
Just for clarity, I double checked and I have entered the ExecStartPre command exactly as you posted it, as you can see from my service script below, however as you can also see from the the status file it seems to append it.
[Unit]
Description=Plex-Mount Service
[Service]
Type=notify
Environment=RCLONE_CONFIG=//home/craftyclown/.config/rclone/rclone.conf
KillMode=none
RestartSec=5
ExecStartPre=-/bin/fusermount -uz /home/craftyclown/Skull
ExecStart=/usr/bin/rclone mount "RichFlixCrypt:/Encrypted/" /home/craftyclown/Skull \
--allow-other \
--buffer-size 256M \
--dir-cache-time 72h \
--log-level INFO \
--log-file /home/craftyclown/logs/PlexMount.log \
--poll-interval 15s \
--timeout 1h \
--umask 002 \
--cache-dir=/home/craftyclown/rclone_cache \
--vfs-cache-mode full \
--vfs-cache-max-size 100G \
--vfs-cache-max-age 24h \
--tpslimit 2
ExecStop=/bin/fusermount -uz /home/craftyclown/Skull
Restart=on-failure
User=craftyclown
Group=craftyclown
[Install]
WantedBy=multi-user.target
Last login: Tue Sep 29 17:41:06 2020 from 185.247.200.174
craftyclown@KingKong:~$ sudo systemctl status Plex-Mount
[sudo] password for craftyclown:
● Plex-Mount.service - Plex-Mount Service
Loaded: loaded (/etc/systemd/system/Plex-Mount.service; disabled; vendor preset: enabled)
Active: active (running) since Tue 2020-09-29 18:10:46 BST; 1min 44s ago
Process: 595 ExecStartPre=/bin/fusermount -uz /home/craftyclown/Skull (code=exited, status=1/FAILURE)
Main PID: 610 (rclone)
Tasks: 14 (limit: 38303)
Memory: 131.4M
CGroup: /system.slice/Plex-Mount.service
└─610 /usr/bin/rclone mount RichFlixCrypt:/Encrypted/ /home/craftyclown/Skull --allow-other --buffer-size 256M --dir-cache-time 72h --log-level INFO --log>
Sep 29 18:10:03 KingKong systemd[1]: Starting Plex-Mount Service...
Sep 29 18:10:03 KingKong fusermount[595]: /bin/fusermount: entry for /home/craftyclown/Skull not found in /etc/mtab
Sep 29 18:10:46 KingKong systemd[1]: Started Plex-Mount Service.Well I'd agree if it wasn't for the fact I still have the issue when docker is completely disabled
If you have dockers removed, it's your systemd configuration then.