Using rclone mount for an FTP server causing plex issues

uStream · September 12, 2023, 3:28pm

What is the problem you are having with rclone?

I moved from Google Suite to Hetzner cloud storage running Windows Server, and I'm mounting it as FTP to my Whatbox server (where plex is installed) using rclone to scan for content.
Moving from google rclone mount, I assumed there are less limitations with ftp since it's me who's running the server, so I tried removing a few parameters from the mount command to make it as simple as possible, but I can't seem to find the right command. I keep running into issues that cause either plex to crash, or for the mount itself to break. The issues that I'm facing:

Scanning libraries after adding new files takes a very long time as it goes through every single folder again, and I have terabytes of data. With gsuite mount, it would detect the new files and scan the newly modified directories only. Scan would only take seconds. Now it takes 10-20 mins. I think this has to do with cache?
While scanning libraries, it tends to crash a lot. By crash, I mean either Plex just hangs while scanning the same movie/show and stops playing anything, resulting in having to restart it.
Another way it crashes while scanning for a while is all folders in the mount become empty when I browse them. So Plex thinks that the files were deleted and my content would move to the trash, and I would have to unmount then remount for it to work
I keep getting errors in the SSH of the whatbox server while it's playing or scanning content. It's usually like this:

ERROR : ftp/folder/video.mp4: ReadFileHandle.Read error: open: write tcp XXX.XXX.XX.XX:XXXXX->XX.XX.XXX.XX:XX: write: connection reset by peer

These issues have been causing my plex to constantly scan, constantly crash, and just cause a headache as I haven't been able to figure out how to fix it. I'm not an expert when it comes to parameters for the mount command, and I've tried browsing the forums here but couldn't find the right modifications that I need to make for an ftp mount.

What I would like to achieve:

Be able to scan the libraries quickly without going through every single folder in the mount
Be able to detect new files a few seconds after I've added them to the mounted server (I'm assuming this has to do with --poll-interval)
Not get errors often resulting in having to remount

I hope this isn't too much to ask. Any help would be greatly appreciated. Thank you

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.1-DEV

os/version: gentoo 2.14 (64 bit)
os/kernel: 5.15.74-gentoo-whatbox (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.20.5
go/linking: dynamic
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Hetzner storage running Windows Server, mounting as ftp to Whatbox server (seedbox) with plex

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone mount ftp: ~/mnt/ftp_plex --cache-db-purge --buffer-size 32M --use-mmap --dir-cache-time 72h --drive-chunk-size 16M  --timeout 1h --vfs-cache-mode minimal --poll-interval=1s &

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[ftp]
type = ftp
host = XX.XXX.XXX.XX
user = user
port = port
pass = pass

A log from the command that you were trying to run with the `-vv` flag

-bash: -vv: command not found

Please let me know the proper way to log what you need. It doesn't seem that I'm doing it the right way with the mount command.
Thank you

asdffdsa · September 12, 2023, 3:41pm

welcome to the forum,

not sure what you mean, ftp server inside windows server, ftp with hetzner storagebox or what?
what does window server have to do with it?

when i rclone mount with hetzner stoargebox, i use webdav.
ftp is terrible, sftp slightly less terrible.

need to prime/pre-load the vfs dir cache.

to the mount command, add --rc --rc-no-auth
once the mount is live, run+wait rclone rc vfs/refresh
trigger plex to scan the library.

gdrive supports ChangeNotify, ftp,sftp,webdav do not.
using debug output would show that.

so if you add a new file to ftp server, rclone will not auto-magically detect it.
plex would have to scan the mountpoint, which would trigger rclone to scan the ftp server.

--cache-db-purge - does nothing.
--drive-chunk-size 16M - does nothing, that is for gdrive, not ftp.

for debug output, -vv is a flag to be added to the rclone command.

uStream · September 12, 2023, 5:51pm

Thank you for the detailed and quick response.

Yes I agree that ftp is terrible haha. I'm honestly not sure how to use webdav.
Using Hetzner Robot I installed Windows Server just so can manage my files on the server with a UI, and through that I was able to start the ftp server so that I can mount it on rclone.
It sounds like webdav is the way to go. Do you by any chance know a tutorial on how to activate it and mount it to a seedbox?

Maybe I wasn't clear. Right now using --poll-interval=1s, the files do show up right away in the mount, so that's working. Only issue is the scan taking long. I guess I will have to run rclone rc vfs/refresh like you listed above, before scanning, every time?

asdffdsa · September 12, 2023, 6:21pm

that should work for gdrive, ftp does not support polling:

really need to use -vv or at least -v and in the output, would see
INFO : ftp://redacted:37245: poll-interval is not supported by this remote
the docs can be very confusing,
"will only be picked up once the directory cache expires if the backend configured does not support polling for changes"

correct.
with rclone mount, there are two caches.

each cache has:
a. different purpose
b. different data
c. different storage
d. different levels of volatility

vfs-dir-cache
a. to speed up the listing of directory/file names from slow cloud based storage.
b. a list of directory/file names.
c. stored in memory.
d. once rclone mount is killed, the cache is lost. does not survive a reboot.

vfs-file-cache
a. to store the contents of files read from slow cloud based storage.
b. the contents of files.
c. stored in local storage.
d. once rclone mount is killed, the file cache contents are perseved. does survive a reboot.
if you re-run the same mount command, the rclone should re-use the cache.

alneven · September 12, 2023, 8:43pm

You can also disable auto library scan and use on your whatbox server a curl script which triggers the only changed folders for a scan.

asdffdsa · September 12, 2023, 9:04pm

here is my remote.

[hetznersbox01_webdav]
type = webdav
url = https://username.your-storagebox.de
vendor = other
user = username
pass = redacted

[hetznersbox01_webdav_crypt]
type = crypt
remote = hetznersbox01_webdav:store
password = redacted
password2 = redacted

well, my setup is a bit different, as per where the media server is located.

on your setup, plex runs on whatbox shared server.
on my setup, emby runs on cheap hetzner vm, located in the same data center as the storagebox.

only need cache when copying/writing files into the mount.
so i use something like, --vfs-cache-mode=writes --vfs-cache-max=1m --vfs-cache-max-size=1g

side note: in my setup, due to low latency, sftp works fine, same as webdav.

uStream · September 25, 2023, 3:51pm

Hello, thank you everyone for your replies and help. I wanted to take some time and make sure that my issues have been fixed before updating this post.
So as others suggested, I have switched from FTP to SFTP mount from my windows machine, and was able to add 4 of my partitions to the mount by using mklink in CMD.
Scanning libraries has been much faster, and I haven't had any playback issues since then. I stopped getting write IO errors in the whatbox server SSH.

I've been using this with my rclone sftp mount command:

rclone mount sftp_plex: ~/mnt/sftp_plex --rc --rc-no-auth --buffer-size 32M --use-mmap --dir-cache-time 72h  --timeout 1h --vfs-cache-mode minimal --poll-interval=1s &

Only issue is that running rclone rc vfs/refresh would stop working after a while, so the files wouldn't refresh. It works for a few hours after mounting, but maybe after a day or so, it just stops working. I would either have to remount, or in my FTP program (Filezilla) I would rename the folder to refresh the files inside of it. Maybe the refresh command stops working due to some wrong parameter in my mount command? Any help would be appreciated.

Either way, most of my issues have been fixed by making this switch. Thanks again everyone for your help

asdffdsa · September 25, 2023, 4:00pm

each time you need to refresh the vfs dir cache, have to run the command again.

uStream · September 25, 2023, 4:13pm

Yes, I run the command and get the OK message:

rclone rc vfs/refresh
{
        "result": {
                "": "OK"
        }
}

but the directories wouldn't refresh. It works at first, but after a while, I run the command, get the OK message, but the new files would still not show up. Am I doing something wrong?

asdffdsa · September 25, 2023, 4:29pm

i would think so.
run the mount command with debug output, looks for problems.

uStream · September 25, 2023, 4:33pm

I created a txt file locally and then run the refresh command with debug:

rclone rc vfs/refresh -vv
2023/09/25 16:31:52 DEBUG : rclone: Version "v1.63.1-DEV" starting with parameters ["rclone" "rc" "vfs/refresh" "-vv"]
{
        "result": {
                "": "OK"
        }
}
2023/09/25 16:31:52 DEBUG : 6 go routines active

The txt file has still not shown up. Hope this helps

asdffdsa · September 25, 2023, 4:37pm

the refresh command applies to the remote, which is sftp server, not local.

uStream · September 25, 2023, 4:43pm

Sorry, I meant that I created it in the directories that are within the SFTP shared folders. It usually shows up if I run the refresh command after mounting, but now after a while, it doesn't seem to be properly refreshing.

asdffdsa · September 25, 2023, 4:44pm

that is the next step

uStream · September 25, 2023, 10:02pm

Is there a way I can exclude all irrelevant info from the debug so I can share here? Getting lots of ChunkedReader.Read logs from plex playback. I ran the mount command with debug and now a few hours later, the refresh command is already not working properly.

alneven · September 26, 2023, 7:46am

you can use Notepad++ and search and mark and remove ALL those lines (in 3 seconds)

uStream · September 28, 2023, 9:45pm

Sorry for the late reply.
After running rclone rc vfs/refresh, I get these 2 lines in the mount debug:

2023/09/28 21:43:46 DEBUG : rc: "vfs/refresh": with parameters map[]
2023/09/28 21:43:46 DEBUG : rc: "vfs/refresh": reply map[result:map[:OK]]: <nil>

New files still aren't showing unless I remount the sftp.

system · October 1, 2023, 9:45pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.