Sonarr/radarr/transmission with rclone and mergerfs - *arr automove is super slow

#1

Heres my workflow:

freenas NFS shares two directories - /downloads and /media

dockerbox with sonarr/radarr/transmission-openvpn in containers, and rclone installed on it has
freenas mounted /downloads
freenas mounted /media
google drive mounted /gd

mergerfs has
/gd & /media:/gmedia

sonarr/radarr all download to /downloads and then have the copy (not hardlink) to /gmedia (the mergerfs location)

my understanding is that I can't use hardlinks because the locations for these things are not local drives - but rather NFS shares, spanning multiple disks.

the idea is that sonarr/radarr should download to /downloads then once done, copy and rename to /gmedia relatively quickly, then delete /downloads version.

the problem is that it is not doing relatively quickly this at all - and it didn't start until i started using rclone/mergerfs.
now, its taking ~3 hours to move a 50gb file. 30 minutes or so to move a 5-10gb file. my downloads are accumulating faster than they can be moved, which is causing a whole slew of other problems.

I only ask this here because i assume some of you guys have something similarly setup.
i don't think i have rclone setup with a cache of any sort - there is only one entry in the conf file. i don't know if that would help it or not, or if i should make my *arr's move to /media instead of the mergerfs /gmedia....not sure.

anyone able to help me figure this out?

#2

That's correct. Hard links only work on the same physical disk.

You'd have to share your mount command and mergerfs mount command along with versions and logs from rclone with -vv to show the output to diagnose what's going on as there isn't enough information.

#3

these are my mount/rclone commands

[Service]
Type=forking
ExecStart=/usr/bin/mergerfs /media/transmission:/gd/transmission /gmedia -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true
KillMode=process
Restart=on-failure

Radarr Ver. 0.2.0.1358
Sonarr Ver. 2.0.0.5338

rclone v1.49.5

  • os/arch: linux/amd64
  • go version: go1.12.10

meanwhile i have tried to eliminate the remote datastore aspect of it.
i have installed a 1tb drive in my box and mounted that in the old network location so its all downloading and storing locally until the upload is complete.
did try to set hardlinks but don't seem to be working yet (its moving at burst-ier speeds but still pretty slow - 10MB a sec or so, until all of a sudden a random second will have copied 200MB :?)

i've based most of my install and setup on your stuff, mr. animosity (thanks for the github btw)

will add logs next