Multiple Servers Same Mount / MergerFS - Brainstorming Logic of Cycled Rclone Moves, same data presenting to all servers?

Off Topic
1

What is the problem you are having with rclone?

No problem with Rclone - The actual problem if anything is that its got new features I'm not familiar with and haven't had to learn about because its worked so well lol.

Preface: A few years back, I setup a box with the use of a 'docker setup' service many of you probably have heard of called PG. It didn't work well enough, and I stumbled upon this direct forum, and the guy thats super popular - Animosity022. Using his scripts, I setup what I needed and it worked FLAWLESSLY for years. Until now...enter synchronous gigabit fiber to the home. lol.

Now I started running into two issues (neither were to do with rclone) - one is that my bandwidth was too much for the small hardware box to run with my media server transcoding, and also utilizing a VPN connection for downloading ISOs.
The other was a certain limit of 750GB/24 hours, which was easy to hit now.

Enter solutions.
I came back to the forums to research and found that Animosity has made quite a few tweaks to his stuff. I started doing some research and still like his idea the best, which is not to live-upload (downloader copying directly into the rclone mount...some friends and colleagues use this, and it works well enough for them) but rather to upload on a script cycle. I have a few reasons for this :
-This will allow a single 'mount' to serve the data to my media server, without issue with that rclone remote.
-This will allow use of several rclone remotes configured to run the 'move' command, throughout the day, cron style.

As I start trying to architect this solution, I now realize there are things similar to this 'pause before uploading' built into rclone that (may not be?) are new...since I last researched, in 2019. For example, I see some using vfs-cache and some commands with vfs that will auto hold the files, storing them in a cache directory then uploading on a time-after-edit usecase. This was an approach I was going to go down, until (i think? was testing last night) this caused my downloader to crash several times, my remote/rclone service to kill itself, and etc..etc..etc... lol

So my troubleshooting today is going to lead me to the idea of this>

One box running docker. In it, an rclone mount to /gdrive. Additionally, a mergerfs between it, and a location for staged files, waiting to be uploaded. This merger fs will be something like /glocal:/gdrive -> /gmedia. This is how I started with animosity's stuff 3 years ago.

This is where my head hits a 'hopefully some people have tried some of this stuff before, run into issues and I'm not going to try to reinvent the wheel' because my thought process ORIGINALLY was to serve the merged /gmedia over NFS, to another box also running docker, which consisted of my media server application (starts with a P :)). This is EXACTLY how I have this stuff setup now, but with no NFS share, all of this is handled on one box, so the mergerfs is a 'local' directory passed through to docker from within itself.

I don't know if presenting this over the network over NFS is going to be a super bottleneck, or if there are certain settings to give to rclone mount to mitigate this....or even to the individual applications. (transmission, the *arr's, and pl...)

This 'downloader docker' box will then run a series of scripts, probably via cron, to utilize rclone move, with a bandwidth limit on the command, that will use remote1 at 0100 to move ~600GB, then exit...remote2 at 0700 to move ~600G, then exit....remote3 at 1200 to move ~600GB then exit. each of these scripts will probably have a bit of logic built in to run a little more frequently, and only check to see if rclone is running then holdoff til the next run. I doubt I'll ever hit more than 2TB in a day.

It seems, at least to my perspective, I have a really good handle on how to utilize the move scripts and the scheduling around that, but not so much to handle the fact that a different box is serving the media to another box over NFS. The biggest problem I think I have is that the *arr's need to move it locally, and for them to move it locally, pl3x needs to be made aware of where they get moved to.
Either I mount the /glocal and /gdrive both as NFS on another box as well, and mergerfs the NFS mounts there AS WELL AS on the host box...or just serve the /gmedia over NFS...but that might have some serious performance issues?

Solutions ? Thoughts ? Call me a crazy person ?

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.0-beta.6078.bab91e440

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 5.4.0-107-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.18.1
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

GDrive - Teams now.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

This is not actually applicable, however this is the series of things I was testing last night.
/usr/bin/rclone mount tcrypt: /gdrive
--allow-other
--allow-root
--dir-cache-time 48h
--log-level INFO
--log-file /opt/rclone/logs/rclone.log
--config /opt/rclone/rclone.conf
--poll-interval 15s
--fast-list
--umask 002
--vfs-cache-mode full
--vfs-cache-max-size 10G
--vfs-cache-max-age 336h
--vfs-write-back 300s
--vfs-read-chunk-size-limit 500M
--cache-dir /p1r4t3/.cache/
--allow-non-empty
--rc
--rc-addr 127.0.0.1:5572

2

hi
small detail from a small mind, --fast-list does nothing on a mount.

we will have to wait for @Animosity022 for the big mind :wink:

3

lol this is true. i see this in the logs for everrrrrr. i just never took it out as it didnt fail. thanks for the knowledgeable input though :slight_smile:

4

So my take on things are make them as simple as possible and only add complexity when it really has a big benefit.

I just recently slimmed my setup down and removed mergerfs as dropbox has no daily upload limits so I just use cache-mode full mounts and upload automatically after an hour. Much less to worry about other than running out of disk space on my stage drive, but it's a 2TB SSD so it seems to be fine for what I'm doing.

Two servers make things complex as once something is deleted from the cloud drive, you need to have it locally on both which you can do via NFS mount or something like you described, but the question comes back to why do that.

Plex has great hardware transcoding support on Linux or running via Docker on any Linux as it removes the need for a lot of driver fuss.

HDR to SDR Tone Mapping | Plex Support

So I'm going through my setup now and slimming down complexity by adding a little complexity with putting everything in dockers. I even toyed with looking at Windows but the lack of the proper tone mapping hardware support made my stick with any Linux flavor. You can get by transcoding 4ks with Intel Quicksync with pretty much any recent i5/7 so a small single server without too much cost up front will last for a long time.

I have gigabit fios as well and I basically use my router to rate limit things and handle all that stuff at the router level rather than touching anything in rclone or trying to do it elsewhere. IPFire/OPNSense/pfSense/Firewalla all do great jobs with a bit of time investing into doing really great QoS.

When I had to deal with Google Drive limits, I went mergerfs with a 8-10TB drive and just let it upload at most 750GB per day if I did hit more than that.

For my whole goal is hands off with as little intervention as possible so I'll pay for adding the right space / speccing the right server out so I never touch it. Worse case, I burn out SSDs in 4-5 years and they'd get replaced anyway. My one 250GB SSD is already 5+ years old as that's my boot disk :slight_smile:

1 Like
5

So, all great ideas.

I originally wondered, (and then wondered again when I was reading over your 'updated to me' scripts and services) why you were using just straight OS. I've been using docker with all of it for years, no real issues - but like you said the one huge benefit was transcoding support in Intel. My setup was the same as yours just docker-ed.

The box I've been using is an i3-9100 with QS passthrough to the container for plex. This has been flawless. The actual CPU doing transcodes of audio is meh, but even 4K...lol.

But the next issue is, like you said, vpn at the firewall level. I've questioned it, and probably should just use that - but wanted to use WG in favor of OVPN (pia subscriber) for the cpu cycles...and still using pfsense as my main firewall. Otherwise, the VM's or actual piece of hardware (re: i3-9100) just can NOT hold its own trying to do all that. Not while plex is going anyways.

Altogether, your suggestions is don't over-engineer the stuff which is great....I wish I'd not. I just bought 3 new 2U servers, moved pfsense to a 1u custom supermicro chassis and board (over the same i3 box hardware as my dockerbox) and setup a proxmox cluster because.....still figuring out the rest of that sentence. lol.

To be honest, all of this, all of the stuff I borrowed from you worked GREAT until I had more bandwidth. Even then, not even using it just having the ceiling choked everything. I at least moved transmission off, and that worked well-ish...but the workflow was transmission VM -> dl to dockerbox /ssd (over local network) which caused sonarr/radarr a bit of a choke.

I'm deciding just exactly how to remove all the over-engineering aspects of this because this is one thing I don't really want to go to down when I want to tinker with other stuff....but at the same time, I've moved almost every other thing in my network now to rackmounted fun stuff and this one last box is all that remains...plex on an hp 290-0043w. lol.

I'm probably going with your suggestion, and am looking into docker routing because I do indeed have a vlan setup just for PIA on pfsense. As long as the bandwidth doesnt choke the pfsense processor out over openVPN, i should be alright....but if it doesn't, you already know I'm probably looking at switching to opnsense just for wg support lol.