Rclone on new server causes Rclone on old server to crash when scanning media

JumpinJack · October 12, 2023, 12:02pm

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

I am in the process of setting up a new server, I have installed Rclone on the new server and began to scan data into Radarr / Sonarr / Jellyfin, but noticed when doing this, it seems to crash Rclone on my old server, and vise versa.

Run the command 'rclone version' and share the full output of the command.

New Server:
rclone v1.64.0
- os/version: debian 12.2 (64 bit)
- os/kernel: 6.1.0-12-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.1
- go/linking: static
- go/tags: none


Old Server:
rclone v1.64.0
- os/version: debian 12.2 (64 bit)
- os/kernel: 6.1.0-12-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.1
- go/linking: static
- go

Are you on the latest version of rclone? You can validate by checking the version listed here: Rclone downloads
-->
Yes

Which cloud storage system are you using? (eg Google Drive)

Dropbox

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

No command, mounted using mergerfs

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

New server
[crypt]
type = crypt
directory_name_encryption = true
filename_encryption = standard
remote = dropbox:/foldername
password = XXX
password2 = XXX

[dropbox]
type = dropbox
client_id = XXX
client_secret = XXX
scope = dropbox
token = XXX


Old Server
[Dropbox]
type = dropbox
token = XXX

[crypt]
type = crypt
directory_name_encryption = true
filename_encryption = standard
remote = Dropbox:/foldername
password = XXX
password2 = XXX

A log from the command that you were trying to run with the `-vv` flag

No command used.

Not sure why this is happening, the 2 servers are using different tokens, the new server is also using a service account, the old server is not. I have never had this happen before so wondering if there is something simple I have missed when setting up Rclone on the new server? This is stopping me from fully setting up the new server while keeping the old one live to continue streaming media on Jellyfin.

Any help would be greatly appreciated Thank you

Animosity022 · October 12, 2023, 12:18pm

What does 'crash' rclone mean?

Help requires a log file of whatever is happening.

JumpinJack · October 12, 2023, 12:43pm

Basically, rclone will become unresponsive on the server not being used, loading folders, reading files, etc stop working until the rclone service is restarted.

How would I get a log file of this?

Animosity022 · October 12, 2023, 12:47pm

Add -vv to the mount and most people will add --log-file /some/directory/rclone.log

JumpinJack · October 12, 2023, 1:07pm

I'm using a service file to mount my dropbox so where would I put these exactly? Sorry for the noobish questions lol

Animosity022 · October 12, 2023, 1:15pm

If you want to share your service file, I can add some input.

I use something like:

github.com

animosity22/homescripts/blob/master/systemd/rclone-movies.service

[Unit]
Description=RClone Service Dropbox Movies
Wants=network-online.target
After=network-online.target

[Service]
Type=notify
Environment=RCLONE_CONFIG=/opt/rclone/rclone.conf
RestartSec=5
ExecStart=/usr/bin/rclone mount2 dcrypt-movies: /media/Movies \
# This is for allowing users other than the user running rclone access to the mount
--allow-other \
# Dropbox is a polling remote so this value can be set very high and any changes are detected via polling.
--dir-cache-time 9999h \
# Log file location
--log-file /opt/rclone/logs/rclone-movies.log \
# Set the log level
--log-level INFO \
# This is setting the file permission on the mount to user and group have the same access and other can read
--umask 002 \

This file has been truncated. show original

My file has a bunch of comments in it as well.

JumpinJack · October 12, 2023, 1:27pm


[Unit]
Description=Dropbox (rclone)
AssertPathIsDirectory=/home/%i/rclone/dropbox
Wants=network-online.target
After=network-online.target

[Service]
User=%i
Group=%i
Type=notify
ExecStart=/usr/bin/rclone mount crypt: /home/%i/rclone/dropbox \
        --config=/home/%i/.config/rclone/rclone.conf \
        --allow-other \
        --buffer-size 256M \
        --dir-cache-time 1000h \
        --drive-chunk-size 128M \
        --poll-interval 15s \
        --timeout 1h \
        --vfs-read-chunk-size-limit off \
        --user-agent user \
        --gid 1001 \
        --uid 1000
ExecStop=/bin/fusermount -uz /home/%i/rclone/dropbox
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

This is my service file

Animosity022 · October 12, 2023, 1:30pm

So with versions, you want fusermount3 instead as I made that change.

[Unit]
Description=Dropbox (rclone)
AssertPathIsDirectory=/home/%i/rclone/dropbox
Wants=network-online.target
After=network-online.target

[Service]
User=%i
Group=%i
Type=notify
ExecStart=/usr/bin/rclone mount crypt: /home/%i/rclone/dropbox \
        --config=/home/%i/.config/rclone/rclone.conf \
        --allow-other \
        --buffer-size 256M \
        --dir-cache-time 1000h \
        --drive-chunk-size 128M \
        --poll-interval 15s \
        --timeout 1h \
        --vfs-read-chunk-size-limit off \
        --user-agent user \
        --gid 1001 \
        --log-level DEBUG \
        --log-file /tmp/rclone.log \
        --uid 1000
ExecStop=/bin/fusermount3 -uz /home/%i/rclone/dropbox
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Something like that should be good. Adjust the log file location if you have somewhere else you want to put it.

JumpinJack · October 12, 2023, 2:03pm

Ok I have added that to my service file started Rclone back up, left it 10 minutes and the issue has once again started, but no log file is generating.

Animosity022 · October 12, 2023, 2:38pm

Can you share the updated service file?

You can share the journalctl -u -b

output as well.

JumpinJack · October 12, 2023, 2:44pm

[Unit]
Description=Dropbox (rclone)
AssertPathIsDirectory=/home/%i/rclone/dropbox
Wants=network-online.target
After=network-online.target

[Service]
User=%i
Group=%i
Type=notify
ExecStart=/usr/bin/rclone mount crypt: /home/%i/rclone/dropbox \
        --config=/home/%i/.config/rclone/rclone.conf \
        --allow-other \
        --buffer-size 256M \
        --dir-cache-time 1000h \
        --drive-chunk-size 128M \
        --poll-interval 15s \
        --timeout 1h \
        --vfs-read-chunk-size-limit off \
        --user-agent user \
        --gid 1001 \
        --log-level DEBUG \
        --log-file /home/mrmister/tmp/rclone.log \
        --uid 1000
ExecStop=/bin/fusermount3 -uz /home/%i/rclone/dropbox
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Just as you pasted in but changed the log file location.

Heres the output of journalctl -u -b

journalctl -u -b
-- No entries --

JumpinJack · October 12, 2023, 3:02pm

Log file has been generated.

Rclone Log File

Animosity022 · October 12, 2023, 3:09pm

Sorry as the service name was not shown in my post.

[felix@gemini ~]$ journalctl -u rclone-tv -b
Oct 12 05:36:13 gemini systemd[1]: Starting rclone-tv.service - RClone Service Dropbox TV...
Oct 12 05:36:18 gemini rclone[1162]: {
Oct 12 05:36:18 gemini rclone[1162]:         "jobid": 1
Oct 12 05:36:18 gemini rclone[1162]: }
Oct 12 05:36:18 gemini systemd[1]: Started rclone-tv.service - RClone Service Dropbox TV.

JumpinJack · October 12, 2023, 3:12pm

Heres the output. I have noticed though since changing the service file Rclone no longer starts up I have had to change it back to what it was previously

Oct 11 22:42:41 myseedbox.cc systemd[1]: Starting rclone@mrmister.service - Dropbox (rclone)...
Oct 11 22:42:43 myseedbox.cc rclone[1176]: mount helper error: fusermount3: unknown parameter in /etc/fuse.conf at line 3: 'user_allow_other - Using the allow_other mount option works fine as root, in'
Oct 11 22:42:43 myseedbox.cc rclone[1176]: mount helper error: fusermount3: unknown parameter in /etc/fuse.conf at line 4: 'order to have it work as user you need user_allow_other in /etc/fuse.conf as'
Oct 11 22:42:43 myseedbox.cc systemd[1]: Started rclone@mrmister.service - Dropbox (rclone).
Oct 11 22:44:00 myseedbox.cc systemd[1]: Stopping rclone@mrmister.service - Dropbox (rclone)...
Oct 11 22:44:00 myseedbox.cc systemd[1]: rclone@mrmister.service: Main process exited, code=exited, status=143/n/a
Oct 11 22:44:00 myseedbox.cc systemd[1]: rclone@mrmister.service: Failed with result 'exit-code'.
Oct 11 22:44:00 myseedbox.cc systemd[1]: Stopped rclone@mrmister.service - Dropbox (rclone).
Oct 11 22:44:00 myseedbox.cc systemd[1]: rclone@mrmister.service: Consumed 8.652s CPU time, received 287.6M IP traffic, sent 4.2M IP traffic.
Oct 11 22:49:45 myseedbox.cc systemd[1]: Starting rclone@mrmister.service - Dropbox (rclone)...
Oct 11 22:49:45 myseedbox.cc rclone[223497]: mount helper error: fusermount3: unknown parameter in /etc/fuse.conf at line 3: 'user_allow_other - Using the allow_other mount option works fine as root, in'
Oct 11 22:49:45 myseedbox.cc rclone[223497]: mount helper error: fusermount3: unknown parameter in /etc/fuse.conf at line 4: 'order to have it work as user you need user_allow_other in /etc/fuse.conf as'
Oct 11 22:49:45 myseedbox.cc systemd[1]: Started rclone@mrmister.service - Dropbox (rclone).
Oct 11 22:50:24 myseedbox.cc systemd[1]: Stopping rclone@mrmister.service - Dropbox (rclone)...
Oct 11 22:50:24 myseedbox.cc systemd[1]: rclone@mrmister.service: Deactivated successfully.
Oct 11 22:50:24 myseedbox.cc systemd[1]: Stopped rclone@mrmister.service - Dropbox (rclone).
Oct 11 22:50:24 myseedbox.cc systemd[1]: rclone@mrmister.service: Consumed 95ms CPU time, received 16.9K IP traffic, sent 8.2K IP traffic.
Oct 11 22:51:20 myseedbox.cc systemd[1]: Starting rclone@mrmister.service - Dropbox (rclone)...
Oct 11 22:51:22 myseedbox.cc rclone[352766]: mount helper error: fusermount3: unknown parameter in /etc/fuse.conf at line 3: 'user_allow_other - Using the allow_other mount option works fine as root, in'
Oct 11 22:51:22 myseedbox.cc rclone[352766]: mount helper error: fusermount3: unknown parameter in /etc/fuse.conf at line 4: 'order to have it work as user you need user_allow_other in /etc/fuse.conf as'
Oct 11 22:51:22 myseedbox.cc systemd[1]: Started rclone@mrmister.service - Dropbox (rclone).
Oct 11 22:52:00 myseedbox.cc systemd[1]: Stopping rclone@mrmister.service - Dropbox (rclone)...
Oct 11 22:52:00 myseedbox.cc systemd[1]: rclone@mrmister.service: Deactivated successfully.
Oct 11 22:52:00 myseedbox.cc systemd[1]: Stopped rclone@mrmister.service - Dropbox (rclone).
Oct 11 22:52:00 myseedbox.cc systemd[1]: rclone@mrmister.service: Consumed 111ms CPU time, received 14.1K IP traffic, sent 5.1K IP traffic.
Oct 11 22:52:43 myseedbox.cc systemd[1]: Starting rclone@mrmister.service - Dropbox (rclone)...
Oct 11 22:52:44 myseedbox.cc rclone[462517]: mount helper error: fusermount3: unknown parameter in /etc/fuse.conf at line 3: 'user_allow_other - Using the allow_other mount option works fine as root, in'
Oct 11 22:52:44 myseedbox.cc rclone[462517]: mount helper error: fusermount3: unknown parameter in /etc/fuse.conf at line 4: 'order to have it work as user you need user_allow_other in /etc/fuse.conf as'
Oct 11 22:52:44 myseedbox.cc systemd[1]: Started rclone@mrmister.service - Dropbox (rclone).
Oct 11 22:58:28 myseedbox.cc rclone[462517]: NOTICE: too_many_requests/..: Too many requests or write operations. Trying again in 300 seconds.
Oct 11 23:00:41 myseedbox.cc systemd[1]: Stopping rclone@mrmister.service - Dropbox (rclone)...
Oct 11 23:00:41 myseedbox.cc systemd[1]: rclone@mrmister.service: Main process exited, code=exited, status=143/n/a
Oct 11 23:00:41 myseedbox.cc systemd[1]: rclone@mrmister.service: Failed with result 'exit-code'.
Oct 11 23:00:41 myseedbox.cc systemd[1]: Stopped rclone@mrmister.service - Dropbox (rclone).

Animosity022 · October 12, 2023, 3:20pm

I don't think your mount is actually working as you need to edit your /etc/fuse.conf

JumpinJack:

ct 11 22:52:44 myseedbox.cc rclone[462517]: mount helper error: fusermount3: unknown parameter in /etc/fuse.conf at line 3: 'user_allow_other - Using the allow_other mount option works fine as root, in'
Oct 11 22:52:44 myseedbox.cc rclone[462517]: mount helper error: fusermount3: unknown parameter in /etc/fuse.conf at line 4: 'order to have it work as user you need user_allow_other in /etc/fuse.conf as'
Oct 11 22:52:44 myseedbox.cc systemd[1]: Started rclone@mrmister.service - Dropbox (rclone).

Animosity022 · October 12, 2023, 3:21pm

And for this:

JumpinJack:

Oct 11 22:58:28 myseedbox.cc rclone[462517]: NOTICE: too_many_requests/..: Too many requests or write operations. Trying again in 300 seconds.
Oct 11 23:00:41 myseedbox.cc systemd[1]: Stopping rclone@mrmister.service - Dropbox (rclone)...

Dropbox is stingy on API calls. I use a different client ID and secret for every mount I have and I use a tps limit

# Set the TPS Limit and Burst
--tpslimit 11 \
--tpslimit-burst 0

JumpinJack · October 12, 2023, 3:24pm

Rclone stopped working so I just redid the mounts.

So I have a tps limit set on the move script, would I also need to set this in the service file for the mount?

Also I am only using a client ID on the new server.

Animosity022 · October 12, 2023, 3:26pm

I use:

Different client ID/secret for mounts
Different client ID/secret for my move script

I use

tpslimit on everything

Not using a tpslimit will make things bad as you'll get a lot worse performance since it gets rate limited and you have to retry a lot.

JumpinJack · October 12, 2023, 3:38pm

Ok, I have set the tpslimit now in my service file.

Will see how things go over the next hour and report back.

Thanks

JumpinJack · October 12, 2023, 4:32pm

This has fixed the issue, as my main post stated it was something simple I over looked.

Thank you for the help I appreciate it !!!