Avoid plex scan on crypted ACD by doing it locally?

Hi all

I’m currently moving all my media to ACD using rclone crypt from my dying nas. (an old xpenology which is becoming quite insecure btw…)

When the upload will be done, is it possible to scan my media library on my nas (mounted on the new PMS via sftp) and then unmount it, mount the crypted ACD on the same mountpoint in order to avoid a looooooong plex scan and all the ACD ban thing…

I know it sounds like a plex specific question but I’m concerned about files’ modtime which depends on rclone mount settings (I might be wrong, I’m doing things beyond my understanding at the moment, thanks to all of you guys on this amazing forum!!)

What would be the mount flags?

If you are using ACD, you can just mount and bite the bullet and let it scan as it’ll work. Personally, I never got any bans or anything from scanning on ACD. The only time I’ve seen/read/heard of bans was via GDrive.

The problem with ACD is that is doesn’t support mod times so if you upload, the mod time will be when you uploaded it so it’ll have to rescan it. Kinda annoying.

GDrive supports mod times so you can scan locally and it just notes the path change when uploaded.

I think I would scan like ~8TB overnight when I went to bed, but heavily dependent on how ACD is playing and your bandwidth.

Unfortunately this will eventually force ACD to start throttling/banning if everyone is rescanning many TBs over and over. The service has already sunk over the past year.

I’m not sure what the fix would be as plex and emby both run off mod times of the files now and use that to figure out if they need to analyze/ffprobe them respectively.

That’s really why I moved over to GD and ocamlfuse, but my luck with ocamlfuse seems to be much better than most I’ve read.

yea I know… and agree. The solution would be a cloud safe scanner and metadata caching. I’ve thought about forking one of the many simple fuse mounts out there to overlay and explicitly set the times to one time forever as a parameter. Kind of like:

fusefs --settime ‘01:01:01 01-21-15’ /srcmount /destmount

This would effectively fix it. then you can selectively scan new media and switch the underlying mounts to your heart’s content as long as the content itself was synced.

Something like this. :slight_smile:

python test.py /src /mnt

#!/usr/bin/env python

from __future__ import with_statement

import os
import sys
import errno

from fuse import FUSE, FuseOSError, Operations


class Passthrough(Operations):
    def __init__(self, root):
        self.root = root

    # Helpers
    # =======

    def _full_path(self, partial):
        if partial.startswith("/"):
            partial = partial[1:]
        path = os.path.join(self.root, partial)
        return path

    # Filesystem methods
    # ==================

    def access(self, path, mode):
        full_path = self._full_path(path)
        if not os.access(full_path, mode):
            raise FuseOSError(errno.EACCES)

    def chmod(self, path, mode):
        full_path = self._full_path(path)
        return os.chmod(full_path, mode)

    def chown(self, path, uid, gid):
        full_path = self._full_path(path)
        return os.chown(full_path, uid, gid)

    def getattr(self, path, fh=None):
        full_path = self._full_path(path)
        st = os.lstat(full_path)
        #return dict((key, getattr(st, key)) for key in ('st_atime', 'st_ctime', 'st_gid', 'st_mode', 'st_mtime', 'st_nlink', 'st_size', 'st_uid'))
        return dict((key, getattr(st, key)) for key in ('st_gid', 'st_mode', 'st_nlink', 'st_size', 'st_uid'))

    def readdir(self, path, fh):
        full_path = self._full_path(path)

        dirents = ['.', '..']
        if os.path.isdir(full_path):
            dirents.extend(os.listdir(full_path))
        for r in dirents:
            yield r

    def readlink(self, path):
        pathname = os.readlink(self._full_path(path))
        if pathname.startswith("/"):
            # Path name is absolute, sanitize it.
            return os.path.relpath(pathname, self.root)
        else:
            return pathname

    def mknod(self, path, mode, dev):
        return os.mknod(self._full_path(path), mode, dev)

    def rmdir(self, path):
        full_path = self._full_path(path)
        return os.rmdir(full_path)

    def mkdir(self, path, mode):
        return os.mkdir(self._full_path(path), mode)

    def statfs(self, path):
        full_path = self._full_path(path)
        stv = os.statvfs(full_path)
        return dict((key, getattr(stv, key)) for key in ('f_bavail', 'f_bfree',
            'f_blocks', 'f_bsize', 'f_favail', 'f_ffree', 'f_files', 'f_flag',
            'f_frsize', 'f_namemax'))

    def unlink(self, path):
        return os.unlink(self._full_path(path))

    def symlink(self, name, target):
        return os.symlink(name, self._full_path(target))

    def rename(self, old, new):
        return os.rename(self._full_path(old), self._full_path(new))

    def link(self, target, name):
        return os.link(self._full_path(target), self._full_path(name))

    def utimens(self, path, times=None):
        return os.utime(self._full_path(path), times)

    # File methods
    # ============

    def open(self, path, flags):
        full_path = self._full_path(path)
        return os.open(full_path, flags)

    def create(self, path, mode, fi=None):
        full_path = self._full_path(path)
        return os.open(full_path, os.O_WRONLY | os.O_CREAT, mode)

    def read(self, path, length, offset, fh):
        os.lseek(fh, offset, os.SEEK_SET)
        return os.read(fh, length)

    def write(self, path, buf, offset, fh):
        os.lseek(fh, offset, os.SEEK_SET)
        return os.write(fh, buf)

    def truncate(self, path, length, fh=None):
        full_path = self._full_path(path)
        with open(full_path, 'r+') as f:
            f.truncate(length)

    def flush(self, path, fh):
        return os.fsync(fh)

    def release(self, path, fh):
        return os.close(fh)

    def fsync(self, path, fdatasync, fh):
        return self.flush(path, fh)


def main(mountpoint, root):
    FUSE(Passthrough(root), mountpoint, nothreads=True, foreground=True)

if __name__ == '__main__':
    main(sys.argv[2], sys.argv[1])

Could simple put this over the local volume, scan it, then change to acd and keep it overlayed. Times will never change.

HS -> stat g
  File: ‘g’
  Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
Device: 2fh/47d	Inode: 2           Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/  robert)   Gid: ( 1000/  robert)
Access: 1969-12-31 19:00:00.000000000 -0500
Modify: 1969-12-31 19:00:00.000000000 -0500
Change: 1969-12-31 19:00:00.000000000 -0500
 Birth: -
✓ robert [~/testm] $

So a trick could be to set mod times on my Nas the same it’ll be on crypted ACD once the upload is done…? (absolutely no idea how to perform this…)

@calisro python is something I clearly don’t understand… Sucks to be a noob :weary:

I could be mistaken, but my understanding was that ACD doesn’t support mod times so rclone I think maps upload time to modtime so no real way around it.

8TB overnight?!

I thought it would actually take waaaaaaayyyyyy more time. (because it took like 3-4 minutes to scan 1 file when I was testing the thing and I’ve got like 7000+ files to be scanned)

I haven’t try to ‘optimize’ the mount settings for plex though, it might be better if I do.

Little test:

felix@plex:~$ stat /etc/hosts
  File: '/etc/hosts'
  Size: 184       	Blocks: 8          IO Block: 4096   regular file
Device: 10302h/66306d	Inode: 11796824    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2017-05-05 15:26:20.995135225 -0400
Modify: 2017-04-24 15:25:50.190074037 -0400
Change: 2017-04-24 15:25:50.190074037 -0400

And I uploaded that file just now:

felix@plex:~$ rclone copy /etc/hosts ACD:

felix@plex:~/ACD$ stat hosts
  File: 'hosts'
  Size: 184       	Blocks: 1          IO Block: 4096   regular file
Device: 32h/50d	Inode: 9551839091146849951  Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/   felix)   Gid: ( 1000/   felix)
Access: 2017-05-06 12:19:28.891000000 -0400
Modify: 2017-05-06 12:19:28.891000000 -0400
Change: 2017-05-06 12:19:28.891000000 -0400
 Birth: -

And now if you do
rclone sync ACD: /etc/hosts

Does it change hosts?

Edit : just saw this, I guess it won’t then https://github.com/ncw/rclone/issues/1365

Edit 2 : sorry, I should have tried it when testing…

What I’m saying is have the fuse always show the same time. It then wouldn’t matter what is stored on ACD or Google drive or whatever. Plex would always see a consistent timestamp for all files. My python would always show that same date for all files.

“.plexignore” file would be a solution if something is already scanned and you didn’t want to scan it over and over, like cancelled tv shows, or finished tv series with all their seasons…

or movies which never got re-released on Bluray, or something like that

That sounds great actually

Could be an option after the library is scanned