Mergerfs need heeelp :)

Morphy · February 15, 2019, 12:38pm

Hi guys.

Im trying to find the logic and how I should work with this mergerfs system.

As several has posted already, I find it a bit hard to figure out how its working.

I want to test it out, and work a bit more with moving rather than copy my files.

My current setup is rtorrent for download client.
Rclone latest version , not beta.
Mergerfs

Rclone setup:
gdrive = my googledrive
gcrypt = encrypt/decrypt

Paths:
Download folder
/home/plex/media/movies etc.

Mount folder(s)?
/home/plex/mnt
/home/plex/mnt/gdrive
/home/plex/mnt/media
/home/plex/mnt/move

Service file rclonemount.service:

[Unit]
Description=RClone Service
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
ExecStart=/usr/bin/rclone mount gcrypt: /home/plex/mnt/gdrive
–allow-other
–dir-cache-time 72h
–vfs-read-chunk-size 10M
–vfs-read-chunk-size-limit 512M
–buffer-size 1G
–umask 002
–log-level INFO
–log-file /home/plex/scripts/logs/rclone-mount.txt
ExecStop=/bin/fusermount -uz /home/plex/mnt/gdrive
Restart=on-abort
User=plex
Group=plex

[Install]
WantedBy=default.target

Mergerfs.service

[Unit]
Description=Megerfs Service
After=rclonemount.service
RequiresMountsFor=/home/plex/mnt/gdrive
[Service]
Type=forking
User=plex
Group=plex
ExecStart=/usr/bin/mergerfs -o defaults,sync_read,allow_other,category.action=all,category.create=ff /home/plex/mnt/move:/home/plex/mnt/gdrive /home/plex/mnt/media
ExecStop=/home/plex/mnt/gdrive
Restart=on-abort
RestartSec=5
StartLimitInterval=60s
StartLimitBurst=3
[Install]
WantedBy=rclonemount.service

my /home/plex/mnt/gdrive shows my encrypted content, so far so good.
my /home/plex/mnt/media shows the exact same content??
my /home/plex/mnt/move is empty

I assume that I need to move my newly downloaded content to the /home/plex/mnt/move folder.
Run a script to upload files after xx minutes . Got that sorted as well.

But what I dont understand is… Why is the content the exact same in gdrive and media?
Where should plex libraries be build from? media? gdrive? IM SO lost

Thanks

Morphy

darthShadow · February 15, 2019, 2:53pm

Everything should be pointed to /home/plex/mnt/media since that reflects the latest state of your media and combines both the local storage and the remote mount. Any new media should also be moved to /home/plex/mnt/media

After all the local content has been uploaded, the data should be the same in the gdrive & media folders.

Basic Structure:

/home/plex/mnt/move - All the Local content waiting to be uploaded
/home/plex/mnt/gdrive - All the Remote content that has been uploaded
/home/plex/mnt/move - Combined Local + Remote content

Morphy · February 15, 2019, 3:31pm

Thanks for the explanation .

But what happends to my download folder? Should i download directly to /home/plex/mnt/media/subfoldershere… ?

And how is Plex updating? I asume it Would rescan the entire library ?

My concern is if I download something, Plex will see it ASAP, even though the content aint done downloading?

Im a bit confused , sorry

darthShadow · February 15, 2019, 3:45pm

The only advantage to downloading content directly to /home/plex/mnt/media/subfoldershere is that you avoid an extra move operation in Sonarr/Radarr when importing content.

Plex only scans the configured folders. So, if you don’t add the download paths to it, it won’t scan them.

Animosity022 · February 15, 2019, 3:46pm

So my use case looks like this.

All my items (Sonarr/Radarr/rTorrent/Plex) point to /gmedia.

/gmedia is comprised of:
/GD - My Google Encrypted Drive
/data/local - my local 6TB mirrored set of disks.

I use mergerfs to combine them and always write to the /data/local first.

[Mount]
What = /data/local:/GD
Where = /gmedia
Type = fuse.mergerfs
Options = defaults,sync_read,allow_other,category.action=all,category.create=ff

Normally if I have something written locally, I can see my torrents which I don’t move to the cloud and my local tv shows /movies:

felix@gemini:/data/local$ ls -al
total 8
drwxrwxr-x.  5 felix felix   69 Feb 14 14:01 .
drwxrwxr-x  13 felix felix  217 Feb 14 20:33 ..
drwxrwxr-x. 50 felix felix 4096 Feb 15 10:32 Radarr_Movies
drwxrwxr-x.  6 felix felix   81 Feb 13 14:07 torrents
drwxrwxr-x. 29 felix felix 4096 Feb 15 10:36 TV

Sonarr/Radarr move completed items to /gmedia/TV/blah and /gmedia/Radarr_Movies/blah, which underneath the covers gets written to /data/local as you see above.

Each night at 3am, I move the contents of /data/local excluding /data/local/torrents to get my local content to the cloud.

# Move older local files to the cloud
/usr/bin/rclone move /data/local/ gcrypt: -P --checkers 3 --log-file /home/felix/logs/upload.log -v --tpslimit 3 --transfers 3 --drive-chunk-size 32M --exclude-from /home/felix/scripts/excludes --delete-empty-src-dirs

So that point Sonarr/Radarr/Plex all see content on /gmedia regardless of if it’s local or in the cloud and it operates the same way all the time.

benba · February 15, 2019, 3:48pm

Look at this pictue, maybe this clears your disorder:

Source: https://www.linuxserver.io/2016/02/02/the-perfect-media-server-2016/

Morphy · February 15, 2019, 4:11pm

Thanks for all the replies.

I dont use Radarr or Sonar. I use Rutorrent as standalone . All my filters etc are made in Rutorrent, so Rutorrent downloads to the specific folders TV series, movies etc.

So what you are telling me Ani, I have to skip /home/plex/media/foldersxx and download directly to /home/plex/mnt/media ? Once the items are downloaded there, the system should move them to /home/plex/mnt/move , and then upload the content to gdrive?

I dont really need /home/plex/mnt/gdrive for anything except for decrypting my gdrive?

So after the content is downloaded to /home/plex/mnt/media / which = your /local/data Ani, Plex will scan and see the content. ? But if the data is moved to /move folder, how in the world does it work?

Sorry dont get it

Animosity022 · February 15, 2019, 4:29pm

It’s probably good to take a step back and figure out what you are trying to accomplish.

For me, I use rTorrent/Radarr/Sonarr with Plex so my use case works very well for that in my setup.

The question comes down to what are you trying to accomplish and what is your desired work flow for your setup?

The work flow will dictate the solution.

If you are just using rTorrent, I really can’t see any reason to use mergerfs as it’s just makes it complex.

You “could” use rTorrent and use autotools to move completed torrents to somefolder and you can upload said folder to your Google Drive every x hours / minutes or whatever frequency you want.

Use Plex to solely read your GD and be done with it.

That’s probably how I’d approach if based on how I understand your setup and how your work flow is.

Morphy · February 15, 2019, 6:02pm

I got it to work as I want it too… I LOVE IT

Thanks for all the help

Morphy · February 15, 2019, 6:03pm

With the setup I had without mergerfs Ani… My content would first be available at night ( uploading at night )

With this setup I can download, see my tv show asap, without waiting the next day

Morphy · February 20, 2019, 10:09am

Hi again .

I have a question regarding this setup.

My new download folder is /home/plex/media , mount folder for gdrive is /home/plex/mnt , and then I have my /home/plex/move folder.

All new items are downloaded to my media folder, so the question is . Once the item is downloaded - I assume the DL client is seeding from the current local folder? After upload to gdrive - where is the DL client seeding from? gdrive?

If the items isnt deleted from the DL client , I must assume its locally right? Or am I missing something here?

Cheers

Animosity022 · February 20, 2019, 12:25pm

I can’t figure out why you’d have 3 folders.

As I noted I above, I have a /data/local (local disk) and /GD (My Google Drive)

They are combined via mergerfs to /gmedia

Everything points to /gmedia (Sonarr/Radarr/ruTorrent/Plex).

So example from last night is a show The Is Us.

You can see it’s seeding from /gmedia:

felix@gemini:/gmedia/torrents/seed/TV$ ls -al *S03E13*
-rw-rw-r-- 2 vpn felix 3155318943 Feb 20 03:38 This.Is.Us.S03E13.1080p.WEB.H264-METCON.mkv

Plex sees it here:

felix@gemini:/gmedia/TV/This Is Us$ ls -al *S03E13*
-rw-rw-r-- 2 vpn felix 3155318943 Feb 20 03:38 This.Is.Us.S03E13.mkv

By my mergerfs setup, they both exist on /data/local

My seeding location

felix@gemini:/data/local/torrents/seed/TV$ ls -al *S03E13*
-rw-rw-r-- 2 vpn felix 3155318943 Feb 20 03:38 This.Is.Us.S03E13.1080p.WEB.H264-METCON.mkv

My Plex location

felix@gemini:/data/local/TV/This Is Us$ ls -al
total 3081368
drwxrwxr-x 2 felix felix         43 Feb 20 03:38 .
drwxrwxr-x 6 felix felix        108 Feb 20 05:23 ..
-rw-rw-r-- 2 vpn   felix 3155318943 Feb 20 03:38 This.Is.Us.S03E13.mkv

Overnight tonight, I’ll rclone move the plex location and exclude/leave alone the /data/local/torrents area. I use mergerfs as this lets me use a hard link (think file copy without diving into details so if you delete any one of them, it doesn’t impact the other).

For now, I’ll manually run my upload script and you can see it moved from local to my GD:

2019/02/20 07:24:56 INFO  : TV/This Is Us/This.Is.Us.S03E13.mkv: Copied (new)
2019/02/20 07:24:56 INFO  : TV/This Is Us/This.Is.Us.S03E13.mkv: Deleted

Technically, it will disappear for ~1 minute as that’s my polling on my Google Drive from the Plex area and re-appear.

felix@gemini:/gmedia/TV/This Is Us$ ls -al *S03E13*
ls: cannot access '*S03E13*': No such file or directory
felix@gemini:/gmedia/TV/This Is Us$ date
Wed Feb 20 07:26:01 EST 2019
felix@gemini:/gmedia/TV/This Is Us$ ls -al *S03E13*
-rw-rw-r-- 1 felix felix 3155318943 Feb 20 03:38 This.Is.Us.S03E13.mkv

Since I move in the middle of the night, I’m not concerned about a 1 minute blip.

Hope that helps.

Morphy · February 20, 2019, 12:35pm

Thanks for always your fast replies Ani and good answers
Well I made the folders that way , but I could have collected them under /home/plex/mnt … oh well I can fix that

Still new into this , so learning by doing .

But if I download a file to /home/plex/media ( locally ) via rutorrent, then I must assume its seeding from there.
My setup - downloads file, hardlinks to move , upload at 1:00 am. at 05am it removes the empty folders in the move folder.

But I like to seed a bit more than 1 day, so my rutorrent deletes the torrent and the files after 8 days. So the files has to be local there right?

But I will try use the commands and see what happends from your fine explanation

3 folders

/home/plex/mnt = gdrive mounted to my server encrypted
/home/plex/media = download folder ( but also your /gmedia )
/home/plex/move = uploading to gdrive.

Morphy · February 20, 2019, 12:47pm

Here is my mergerfs.service

[Unit]
Description=Megerfs Service
After=rclonemount.service
RequiresMountsFor=/home/plex/mnt

[Service]
Type=forking
User=plex
Group=plex
ExecStart=/usr/bin/mergerfs -o defaults,sync_read,allow_other,category.action=al l,category.create=ff /home/plex/move:/home/plex/mnt /home/plex/media
ExecStop=/home/plex/mnt
Restart=on-abort
RestartSec=5
StartLimitInterval=60s
StartLimitBurst=3

[Install]
WantedBy=rclonemount.service

Animosity022 · February 20, 2019, 1:19pm

I can’t follow your work flow as it seems to be the as mine but with 3 folders it just doesn’t work as first write does that as it always write to the first location listed so everything would write to /home/plex/move based on your config.

My workflow/use case is really identical to yours so if you aren’t getting my workflow or how it works, I’d work on that as opposed to trying to figure out how make it work with 3 folders as my setup does not work with 3 folders.

My use case is that I download torrents and keep them on local disk. I seed them for however long I want and eventually they get removed or stay depending on my ratio groups and how I have it setup. Torrents are always local as I don’t think GD is a great use case for torrenting.

My Sonarr/Radarr/Plex at the end all point to my GD as that is a great use case for it. Once a download is completed, it is hard linked and loses any relationship / need / anything for the torrent area so if the torrents are deleted it doesn’t matter.

Morphy · February 20, 2019, 2:05pm

Hi Ani

1st my /home/plex/mnt = gdrive is read-only via rclonemount.service
2nd /home/plex/media = my merged folder via mergerfs.service
3rd /home/plex/move = which uploads my content

I can try explain what my setup does, since I dont have sonar etc… all the fancy thingys… Im not there yet

Before I migrated to mergerfs I had these folders as my download folders which has the exact same location on my gdrive with names etc

/home/plex/Media/TV-Series
/home/plex/Media/4K
/home/plex/Media/Movies
/home/plex/Media/Animations
etc…

Via Rutorrent - I downloaded to those folders - overnight I had an upload script which uploaded the content to gdrive in same folders.

Now I made a new setup as you can see with mergerfs

I download my media to the new /home/plex/media folder with the exact same folder setup as on gdrive.
Once downloaded - the files are seeded 8 days, and deleted via rutorrent.
The files are uploaded overnight everyday.
Since the rclonemount is read-only the files wont be deleted on the gdrive via mergerfs. Plex sees everything, everything is working.

I was just curious if the files seeding where the files that was downloaded , since the folder structure is the exact same.

Does it make sense?

Animosity022 · February 20, 2019, 2:13pm

I don’t think Sonarr/Radarr have much to do as they just happen to be the thing that hard links the files.

I still can’t follow your flow to be honest as it’s overly complex. If you take mergerfs out of the situation, how does it work now? mergerfs shouldn’t add any new workflow, just simplify the mounts to make everything unique. I’ve said before, I don’t get why you are using mergerfs in your setup as it doesn’t make sense to me and overly complicates it.

Walk through one example of a TV show or movie and that would help as I think your work flow is.

ruTorrent snags a movie
ruTorrent finishes download
ruTorrent uses autotools or something else to copy (or hardlink) to finished location
something happens to move it to a final plex location (?)
plex scan sees files and adds it to the library

If that’s the flow, I’d just rclone move to the final plex location rather than playing around with anything else as I ran a similar setup before I got rid of my seedbox.

Morphy · February 20, 2019, 3:09pm

My old setup would require instant upload to gdrive and then scan entire library to watch my content. My ssd drive is not that big, so I need the gdrive as support to store my content. So my content would be available the day after.

With mergerfs as it is now :
Rutorrent downloads to ie. moviefolder
Once downloaded it seeds, and plex sees the new content. ( locally )
Once the download is complete, it hardlinks to move folder ( not rutorrent, but the mergerfs )
System uploads the content at night and after upload it delete the content from the move folder. The downloaded content still is available in the media folder and seeding.
After the 8 days rutorrent delete the content locally, and plex still see the content since its the same media folder.

It may sounds confusing, but it rather cool if you could see it in action .
The move folder is also there if I have downloaded something via RSS that I wont need, I can manually delete the torrent, and it deletes the content from the move folder before uploading to gdrive.

I know im not an expert , but believe me, the setup I have now seems more fancy than the old one ^^ .

Morphy · February 20, 2019, 3:11pm

By the way . IF I was seeding directly onto the gdrive, wouldnt I hit API bans then?

Morphy · February 20, 2019, 3:25pm

Here is my API hits from the past 6 days.!