Rclone mount/mounts crashing/disconnecting from the mount folder/point (panic: runtime error: invalid memory address or nil pointer dereference)

[aniel3000 (aniel arias)]

What is the problem you are having with rclone?

am having this issue where my mount will disocnnected/ crash. I beleive it happens when there is a lot of io going on becuase that is the only way i can replicated. i have tried all possible combinations to see if i can pin poin this issue but i have failed. at this point am not sure if this is related to my os, kernel, rclone itself or somehting else.

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.4

• os/version: debian 12.4 (64 bit)
• os/kernel: 6.7.3-zabbly+ (x86_64)
• os/type: linux
• os/arch: amd64
• go/version: go1.22rc2
• go/linking: static
• go/tags: none
  aniel@Hz:~$

Are you on the latest version of rclone? You can validate by checking the version listed here: Rclone downloads
-->

Which cloud storage system are you using? (eg Google Drive)

using dropbox, google, ftp one and few others

The command you were trying to run (eg rclone copy /tmp remote:tmp)

[Unit]
Description=rclone_mounts
#Wants=network-online.target
#After=network-online.target
 
[Service]
Type=notify
User=aniel
Group=aniel
Environment=RCLONE_CONFIG=/home/aniel/.config/rclone/rclone.conf
ExecStart=/usr/bin/rclone mount mounts: /media/mergerfs/mounts \
	#--config=/home/aniel/.config/rclone/rclone.conf \
	--log-file=/media/logs/mounts.log \
	--user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/122.0" \
	#--vfs-write-back 10m \
	#--min-age 1s \
	--ignore-existing \
	#--vfs-fast-fingerprint \
	#--rc-no-auth \
	#--rc-addr 127.0.0.1:5575 \
	#--rc \
	--allow-other \
	#--allow-non-empty \
	--log-level=NOTICE \
	--dir-cache-time 9999h \
	#--vfs-cache-poll-interval 1s \
	--vfs-cache-max-age 9999h \
	#--vfs-refresh=true \
	--vfs-read-chunk-size 1M \
	--vfs-read-chunk-size-limit off \
	#--cache-dir=/media/mergerfs/local_hdd/rclone_cache \
	--vfs-cache-mode full \
	--vfs-cache-max-size 100G \
	#--buffer-size 0M \
	#--vfs-read-ahead 32M \
	--tpslimit 10 \
	--tpslimit-burst 10 \
	#--multi-thread-streams 0 \
	#--cutoff-mode cautious \
	#--network-mode \
	#--read-only \
	#--order-by size,mixed,75 \
	#--max-backlog 10000 \
	--transfers 32 \
	--checkers 64 \
	#--contimeout 30m \
	#--timeout 30m \
	#--retries-sleep 60s \
	--umask 002
ExecStop=/usr/bin/fusermount -uz /media/mergerfs/mounts
#ExecStartPost=/usr/bin/rclone rc vfs/refresh recursive=true --url 127.0.0.1:5575 _async=true
Restart=always
RestartSec=10

[Install]
WantedBy=network-online.target

also tried this one and even simpler ones plus diorecly mounting it on the terminal

rclone mount mounts: /media/mergerfs/mounts --log-file=/media/logs/mounts.log --vfs-cache-mode full --vfs-cache-max-size 100G --umask 002 --ignore-existing --allow-other

[Unit]
Description=rclone_mounts
Wants=network-online.target
After=network-online.target
 
[Service]
User=aniel
Group=aniel
Type=simple

ExecStart=/usr/bin/rclone mount mounts: /media/mergerfs/mounts --log-file=/media/logs/mounts.log --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/122.0" --ignore-existing --allow-other --log-level=NOTICE --dir-cache-time 9999h --vfs-cache-max-age 9999h --vfs-read-chunk-size 1M --vfs-read-chunk-size-limit off --vfs-cache-mode full --vfs-cache-max-size 100G --tpslimit 10 --tpslimit-burst 10 --transfers 32 --checkers 64 --umask 002


# Perform lazy unmount
ExecStop=/usr/bin/fusermount -zu /media/mergerfs/mounts

# Restart the service whenever rclone exists with non-zero exit code
Restart=on-failure
RestartSec=15

[Install]
# Autostart after reboot
WantedBy=default.target

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[aniel_personal_gdrive]
type = drive
scope = drive
token = XXX
team_drive =
root_folder_id =

[dave_ftp]
type = ftp
host = XXX
user = XXX
pass = XXX
port = 21

[dave_ftp_chunker]
type = chunker
remote = dave_ftp:
chunk_size = 5Gi

[dave_ftp_encrypted]
type = crypt
remote = dave_ftp_chunker:xxx
filename_encoding = base32768
password = XXX

[dropbox]
type = dropbox
token = XXX

[dropbox_encrypted]
type = crypt
remote = dropbox:xxx
filename_encoding = base32768
password = XXX
password2 = XXX

[media_merged]
type = union
action_policy = all
create_policy = all
search_policy = ff
upstreams = "xxx_encrypted:" "dropbox_encrypted::ro"

[mounts]
type = combine
upstreams = "media_merged=media_merged:" "dropbox=dropbox:" "dropbox_encrypted=dropbox_encrypted:" "aniel_personal_gdrive=aniel_personal_gdrive:" "xxx=xxx:" "xxx_encrypted=xxx_encrypted:" "dave_ftp=dave_ftp:" "dave_ftp_encrypted=dave_ftp_encrypted:"

[xxx]
type = xxx
access_token = XXX
api_host = XXX
encrypt_files = false

[xxx_encrypted]
type = crypt
remote = xxx:xxx
filename_encoding = base32768
password = XXX
chunk_size = 1Gi

A log from the command that you were trying to run with the -vv flag

this is the log right before the crash for my understanding

2024/02/06 10:18:19 NOTICE: union root '': srcU.f=union root '', srcURemote="media/to fix/shows/1600.Penn.S01E02.The.Skiplantic.Ocean.1080p.AMZN.WEB-DL.DD+5.1.x264-AJP69", dstURemote="media/to fix/shows/_UNPACK_1600.Penn.S01E02.The.Skiplantic.Ocean.1080p.AMZN.WEB-DL.DD+5.1.x264-AJP69.1"
2024/02/06 10:18:25 NOTICE: union root '': srcU.f=union root '', srcURemote="media/to fix/shows/_UNPACK_1600.Penn.S01E02.The.Skiplantic.Ocean.1080p.AMZN.WEB-DL.DD+5.1.x264-AJP69.1", dstURemote="media/to fix/shows/1600.Penn.S01E02.The.Skiplantic.Ocean.1080p.AMZN.WEB-DL.DD+5.1.x264-AJP69.1"
2024/02/06 10:19:03 NOTICE: union root '': srcU.f=union root '', srcURemote="media/to fix/shows/1600.Penn.S01E03.So.You.Dont.Want.to.Dance.1080p.AMZN.WEB-DL.DD+5.1.x264-AJP69", dstURemote="media/to fix/shows/_UNPACK_1600.Penn.S01E03.So.You.Dont.Want.to.Dance.1080p.AMZN.WEB-DL.DD+5.1.x264-AJP69"
2024/02/06 10:19:41 NOTICE: union root '': srcU.f=union root '', srcURemote="media/to fix/shows/_UNPACK_1600.Penn.S01E03.So.You.Dont.Want.to.Dance.1080p.AMZN.WEB-DL.DD+5.1.x264-AJP69", dstURemote="media/to fix/shows/1600.Penn.S01E03.So.You.Dont.Want.to.Dance.1080p.AMZN.WEB-DL.DD+5.1.x264-AJP69"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x195f2a9]

goroutine 79501 [running]:
github.com/rclone/rclone/backend/xxx.(*Fs).findObject(0xc00612e000, {0x2ae3248, 0xc015e5a780}, {0xc011f82600, 0xf5}, {0xc0112b87d5, 0x4d})
	github.com/rclone/rclone/backend/xxx/xxx.go:376 +0x429
github.com/rclone/rclone/backend/xxx.(*Fs).NewObject(0xc00612e000, {0x2ae3248, 0xc015e5a780}, {0xc016160c80, 0x13d})
	github.com/rclone/rclone/backend/xxx/xxx.go:461 +0x59
github.com/rclone/rclone/backend/crypt.(*Fs).NewObject(0xc000540240, {0x2ae3248, 0xc015e5a780}, {0xc015e5522d?, 0xc006108401?})
	github.com/rclone/rclone/backend/crypt/crypt.go:419 +0x8b
github.com/rclone/rclone/backend/union.(*Fs).NewObject.func1(0x0)
	github.com/rclone/rclone/backend/union/union.go:713 +0x8b
github.com/rclone/rclone/backend/union.multithread.func1()
	github.com/rclone/rclone/backend/union/union.go:977 +0x54
created by github.com/rclone/rclone/backend/union.multithread in goroutine 79497
	github.com/rclone/rclone/backend/union/union.go:975 +0x47

i have research this but havent found a fix yet. here are some of those links

--low-level-retries=0 causes panic: runtime error: invalid memory address or nil pointer dereference

What is the problem you are having with rclone? Testing rclone with small/short values around internet connection time outs and retries. I was attempting to force some errors as if I had a bad/spotty internet connection. If I sent --low-level-retires=0 this error occurs every time. If I set --low-level-retires=1 it works correctly. This is not causing me any problems as it is not a --low-level-retries value I expect to use. However, it appears to be boundary condition that should be address…

Panic: runtime error: invalid memory address or nil pointer dereference

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them. What is the problem you are having with rclone? I'm getting a panic: runtime error: invalid memory address or nil pointer dereference error mid-way through a transfer. This seems to happen from dropbox remote to s3 remote. Direct upload from local drive doesn't seem to exhibit the same problems. Run…

Invalid memory address or nil pointer dereference error when copy to Google photos

What is the problem you are having with rclone? When I try to do an automated upload to Google Photos storage, I get a memory error and the upload fails. Sometimes this happens, sometimes it doesn't (i.e., sometimes it uploads fine), and when it does start happening, it lasts for hours (as long as 20 hours). My guess is that it's probably happening when I reach GPhotos' daily upload limit, but when I do, it used to print an error related to Upload Quota, so I could easily see why, but now it d…

Invalid memory address or nil pointer dereference

What is the problem you are having with rclone? Error messages, but works anyway. What is your rclone version (output from rclone version) rclone v1.56.0 os/version: arch rolling (64 bit) os/kernel: 5.13.10-arch1-1 (x86_64) os/type: linux os/arch: amd64 go/version: go1.16.6 go/linking: dynamic go/tags: none Which OS you are using and how many bits (eg Windows 7, 64 bit) OS: EndeavourOS Linux amd_64 I have been using Manjaro Linux and the same problem started to occur recently. For more r…

MEGA: segmentation fault in long-running process

thank you in advance.

[kapitainsky]

@aniel3000

rclone v1.65.4

• os/version: debian 12.4 (64 bit)
• os/kernel: 6.7.3-zabbly+ (x86_64)
• os/type: linux
• os/arch: amd64
• go/version: go1.22rc2
• go/linking: static
• go/tags: none
  aniel@Hz:~$

It is some custom built rclone:) As for now the latest official rclone release is v1.65.2 and beta v1.66.

You have to seek answers with whoever created it.

When running such experimental system (your Linux kernel is also far ahead what Debian supports officially) you have to deal with issues yourself I am afraid.... or within community doing it. I guess there are some reasons for using such on the edge system.

[aniel3000 (aniel arias)]

i can go back to official and stable kernel. do u think that will make a difference? does it even makes a difference in this case. and yes i did forgot to mention that am running acustom rclone build. do u think i should test with the latest rclone beta build?

[kapitainsky]

What I am saying is that if you are running such experimental/custom rclone/OS nobody can even try to replicate your issues. So you have to fix it yourself.

Unless there are other people on this forum who share your setup and can help.

[aniel3000 (aniel arias)]

ok i will try to replicate the issue using official kernel and latest rclone beta build and report back

[kapitainsky]

Let's see - if it helps with your endeavour into unknown even better.

[aniel3000 (aniel arias)]

ahha fair enought.

[kapitainsky]

@aniel3000

[xxx]
type = xxx
access_token = XXX
api_host = XXX
encrypt_files = false

maybe this is the problem - some let's say 3rd party remote.

Because this is what you use in union and mount

[aniel3000 (aniel arias)]

I did not tried taking this out of the equation. however i did took ot out of the main equation by eliminating it from the union and the issue stil happened. do u think it might be causing the issue just by being in the combine group?

[kapitainsky]

I have no clue. You are using something custom built with some "secret" remote.

But genuinely I think you should contact creator of xxx remote to discuss your issues. Or hope that somebody reading this forum is using xxx too and has more knowledge what is doing.

[aniel3000 (aniel arias)]

you have a good point and i will do that, thank for replying to me.

[kapitainsky]

Good luck. And push them to submit PR with this new remote. Might be useful for others too.

[aniel3000 (aniel arias)]

yeah i dont understand why he dosent push this to the official rclone branch. i mean i found his project from this forum so there is no secrecy there. am just the way am and have my reason as to why not reveal said name/ project.

[kapitainsky]

Fair enough. Let's see where it goes

[aniel3000 (aniel arias)]

can confirm swithcing to the latest rclone beta fixes the issue. i will report this to him/her and hopefully it can be fixed and added back to the latest rclone code branch.