Rclone move script with sonarr v3 - no .partial files

Help and Support
1

I was using a combo of scripts, mostly form @Animosity022 's github - and It was working great. I had mergerfs with google drive, and a local folder. With a script that moved the local folder every so many minutes, and it worked great!

But I was working on upgrading to sonarr v3, and it worked great except one part.... The new sonarr does not have .partial files, so my exclude file is not valid anymore, so rclone started moving files that were not fully transfered to the /local folder...

Anyone have some ideas on how to prevent rclone from moving files that are currently being edited? (maybe a check on file size and if it changes?) or has anyone have success with rclone move scripts and sonarr v3?

2

Aren't you using hard links with mergerfs all on the same spot? You shouldn't see any partials at all in the final spot as it copies pretty much instant.

3

I'm not seeing partials, but the episodes or movies can take abit to move from my download client to /local (in FTP I can watch the size go up, and its a regular .mkv file). But since its an .mkv file and if my script runs while its still moving it would cause errors eventually right?

Here's my script for moving

#!/bin/bash
#Check is Lock File exists, if not create it and set trap on exit
if { set -C; 2>/dev/null >~/scripts/rclonemove.lock; }; then
trap "rm -f ~/scripts/rclonemove.lock" EXIT
else
echo "Lock file exists… exiting"
exit
fi
#after check
/home/matt/bin/rclone move /home/matt/local/ gdrive:Videos/ --config=/home/matt/.config/rclone/rclone.conf --exclude-from /home/matt/scripts/exclude --delete-empty-src-dirs --fast-list --transfers=20 --checkers=20 --drive-chunk-size=128M --log-file ~/scripts/move.log

mergefs command:

/home/matt/bin/mergerfs /home/matt/local:/home/matt/rdrive /home/matt/gmedia -o rw,async_read=false,allow_other,use_ino,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true,threads=12

Its very possible I misunderstood something

4

What's your mergerfs setup look like under neath and what are you clients pointing to?

You should be able to test hard linking by just using ln somefile someotherfile on the mergerfs mountpoint to validate things work:

felix@gemini:/gmedia$ cp /etc/hosts .
felix@gemini:/gmedia$ ln hosts test
felix@gemini:/gmedia$ stat hosts
  File: hosts
  Size: 266       	Blocks: 8          IO Block: 4096   regular file
Device: 30h/48d	Inode: 9300096754895801204  Links: 2
Access: (0644/-rw-r--r--)  Uid: ( 1000/   felix)   Gid: ( 1000/   felix)
Access: 2020-04-07 20:35:22.128731318 -0400
Modify: 2020-04-07 20:35:22.128731318 -0400
Change: 2020-04-07 20:35:25.496758291 -0400
 Birth: -

You can see it has 2 links and that tells me the hard link is working.

5 
matt@TheSwagShackRZ:~/local$ cd ~/gmedia
matt@TheSwagShackRZ:~/gmedia$ stat hosts
  File: hosts
  Size: 194             Blocks: 8          IO Block: 4096   regular file
Device: 37h/55d Inode: 9895625424999  Links: 2
Access: (0644/-rw-r--r--)  Uid: ( 1000/    matt)   Gid: ( 1000/    matt)
Access: 2020-04-07 20:42:13.005393119 -0400
Modify: 2020-04-07 20:42:13.005393119 -0400
Change: 2020-04-07 20:42:18.129278146 -0400
 Birth: -

So I have google drive mounted @ ~/rdrive - which is merged with ~/local/ in ~/gmedia
I have sonarr read ~/gmedia so it picks up all the episodes I've downloaded.

Then new episodes will ofc show in the ~/local folder, which is automatically uploaded every 10ish minutes to my google drive.

But for some reason when an episode is done downloading via NZBGet, for example, the linking is not instant and I can slowly watch the filesize go up. I can send a vid too if needed

6

The NZB / Torrent client should also point to something in ~/gmedia

Where are they pointing at?

7

It should?? I just have them going to a normal directory :open_mouth:
they should be pointed to ~/gmedia too? I mean they're both on the same HDD

8

That's probably the main point of my setup and using mergerfs.

Everything points to /gmedia so I can use hard linking and Sonarr/Radarr can use the same mount points.

I use an exclude file in local to keep my other stuff there and any TV or Movies get uploaded:

felix@gemini:/opt/rclone/scripts$ cat excludes
*partial~
*sidecar
lost+found/**
torrents/**
seed/**
NZB/**
9

Do they need to be in the same directory for hard linking? All the stuff is on the same hard drive.
~/usenet/downloads are on the same HDD as ~/gmedia. sorry for all the dumb questions

10

Hard linking works when it's on the same physical disk underneath. Mergerfs allows for using hard links when it's the same disk.

The reason I use megerfs is I can use hard links and keep all my media / programs / everything pointed to /gmedia.

The directory doesn't matter as it's just needs to be on the same physical disk.

So in my case, something might go to /gmedia/torrents/TV/blah.mkv and that links to /gmedia/TV/Show/Blah.mkv

I only exclude anything I don't want to upload as I shared above and that's how I use hard links so when something finishes, it becomes available instantly rather than duplicating disk space.

11

hmmm maybe I'm trying to do the wrong thing then?
I just want for sonarr to view my google drive for what shows I have downloaded, then have a script move my local stuff to my gdrive. I used mergerfs so it would read my google drive, but not write to my rclone mount (as that causes issues and slowness) - and I just had a script auto upload the local files after.

12

That's what I do. I don't write anything to my rclone mount directly. I only upload at night.

13

But the issue I've been having is when a download is complete on say NZBGet, I can see the MKV moving to /local/ via the files size

Episode1.mkv - 500mb
episode1.mkv - 800mb

and rclone will move the file (in theory) even if its being transferred/unzipping to the /local folder, causing incomplete files. Sonarr V2 when it was moving a file named the file .partial till it was completely moved/ changing size

So I was wondering if rclone had a feature to not move/copy files that are being moved.

14

Yeah, rclone does that automatically as if a file is being modified, it's skipped. You should see that in the rclone log.

15

Oh I did not know that was built into RClone!!! I made a problem out of nothing...
Sorry about that and thank you for your time!

16

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.