Get changeNotify in other Log Levels

peterbaumert · November 17, 2021, 11:29am

What is the problem you are having with rclone?

Is there a way to get the ChangeNotify in INFO or NOTICE log-level?
The DEBUG level is working fine in general but produces a lot unneccessary info ( for my use case ).
Or maybe is there another way to trigger something external on changeNotify?

KR
Pete

What is your rclone version (output from `rclone version`)

rclone v1.56.0                       
- os/version: debian 10.11 (64 bit)  
- os/kernel: 4.19.0-17-amd64 (x86_64)
- os/type: linux                     
- os/arch: amd64                     
- go/version: go1.16.5               
- go/linking: static                 
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

ExecStart=/usr/bin/rclone mount union: /mnt/union \
--allow-other \                                                
--dir-cache-time 1000h \                                       
--log-level DEBUG \                                            
--log-file /var/log/rclone/runion.log \                   
--poll-interval 15s \                                          
--user-agent Mozilla \                                         
--rc \                                                         
--rc-web-gui \                                                 
--rc-addr MYIP:5572 \                               
--rc-user rcuser \                                             
--rc-pass rcpass \                                             
--rc-web-gui-no-open-browser \                                 
--rc-serve \                                                   
--cache-dir /srv/cache \                                       
--vfs-cache-mode full \                                        
--vfs-cache-max-size 75G \                                     
--vfs-cache-max-age 48h \                                      
--tpslimit 10

The rclone config contents with secrets removed.

[data_11]                                                                                                                   
type = drive                                                                                                                 
client_id = ***                                        
client_secret = ***                                                                                     
scope = drive                                                                                                                
team_drive = ***                                                                                           
root_folder_id =                                                                                                             
service_account_file = /root/.config/rclone/creds.json                                                                       
impersonate = ***                                                                                                
                                                                                                                             
[union_uncrypted]                                                                                                      
type = union                                                                                                                 
upstreams = data_01::ro --> data_11::ro ( here are all listed, just removed for readability )
action_policy = epall                                                                                                        
create_policy = rand                                                                                                         
search_policy = epff                                                                                                         
                                                                                                                             
[union]                                                                                                                
type = crypt                                                                                                                 
remote = union_uncrypted:                                                                                              
filename_encryption = standard                                                                                               
directory_name_encryption = true                                                                                             
password = ***                                   
password2 = ***

A log from the command with the `-vv` flag

2021/11/17 10:07:58 DEBUG : : >changeNotify:
2021/11/17 10:10:12 DEBUG : : changeNotify: relativePath="TV-1080P/Show/Season 01/File.mp4", type=1

ncw · November 17, 2021, 12:23pm

I guess the question would be what are you doing with the change notifies?

We could potentially make them appear in the rc?

I did have the idea of making an rclone command which just showed the change notifies for testing purposes as that is sometimes useful.

peterbaumert · November 17, 2021, 12:29pm

Hi Nick,

thanks for replying.

I am using the changeNotify's to trigger my Plex Partial rescan.
In rc would be great.
I saw the command for testing but it doesnt output anything in my case.

Is that enough information? Im happy to provide anything needed

KR
Peter

Animosity022 · November 17, 2021, 12:43pm

You probably don't want to recreate the wheel.

Why not using something that's already there?

Does what you are looking for already.

Or use the automatic connectors into to plex to trigger scans.

peterbaumert · November 17, 2021, 1:01pm

Hi Animosity022,

I know about this. But this builds another "own cache" and has a lot of overhead that i don't think is really needed. For example my script I use has like 40 lines and no big dependencies. I appreciate your tips, don't get me wrong. Was just wondering since those notifications exist in rclone anyway, why not use thme?

KR
Peter

Animosity022 · November 17, 2021, 1:08pm

They exists to trouble / debug issues with files being added as there have been a few over time.

Moving them out of a debug level I don't think is a good idea as the logs become to chatty with changes.

In my environment, plex autoscan uses ~50MB of memory and less than 0.1% of my CPU.

What's your use case on how the changes are getting there?

peterbaumert · November 17, 2021, 1:52pm

If that's the case I'm fine with it, was just thinking there might be an easy way. If not I'm ok with the DEBUG level option.

I add stuff mostly manual to gdrive ( not on my plex server).
Yes it doesn't use many resources but still it does things that are already done elsewhere

I appreciate all the time you take!

KR
Peter

Animosity022 · November 17, 2021, 1:57pm

If you add them manually, why not just kick off a scan when done as part of the process?

You can do that via Plex CLI as part of your process.

You can also just scan hourly on Plex and let it fire and forget to:

Scanning with a primed mount is near instant on Linux. I'm super lazy so without diving into to much of the use case, I'd go with the least amount of work and worse case, you wait an hour for media to show up.

peterbaumert · November 17, 2021, 2:02pm

Because my Library has several thousand items and i dont want plex to always run over all again

Animosity022 · November 17, 2021, 2:04pm

Not sure what you mean as if they are scanned, it finishes almost instantly and takes almost no resources.

Your library isn't really that big

peterbaumert · November 17, 2021, 2:08pm

Then i might have an other issue, everytime my scan starts it takes a while and often after I get "downloadlimitexceeded" things. yet i have everything off like thumbnails and so on, its a dedicated box, gbit uplink and all. rclone conf you see up.

Animosity022 · November 17, 2021, 2:21pm

The way Plex works is when you add a file, it needs to analyze the file. That means it does something similar to a ffprobe/mediainfo on a file.

If the file is not analyzed and you scan, it'll analyze when the scan happens.

I use a version of this to view my library stats:

github.com

ajkis/scripts/blob/master/plex/plex-library-stats.sh

#!/bin/bash
# Generate Plex Library stats
# OS: Ubuntu 16.04 ( change dbpath if using diffrent OS )
# Output:
# Media items in Libraries
# Library = Movies
#  Items = 2815

# Library = TV Shows
#  Items = 13105

# Time to watch
# Library = Movies
# Minutes = 312664
#   Hours = 5211
#    Days = 217

# Library = TV Shows
# Minutes = 525251
#   Hours = 8754

This file has been truncated. show original

That gives you an idea of what media is analyzed vs not.

If the library is analyzed, a scan runs very fast.

You'd see something like:

0 files missing analyzation info

If you have Intro Detection on, that's also a killer as it converts the show which requires a full download and checks for intros.

Quotas for Google are a pain. A faster connection would cause you to run out faster.

Animosity022 · November 17, 2021, 2:25pm

And here is mine that has intros added:

github.com

animosity22/homescripts/blob/master/scripts/plex-library-stats.sh

#!/bin/bash

logfile="/home/felix/logs/plexstats.log"
db="/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db"
echo "$(date "+%d.%m.%Y %T") PLEX LIBRARY STATS" | tee -a $logfile
echo "Media items in Libraries" | tee -a $logfile
query="SELECT Library, Items \
                        FROM ( SELECT name AS Library, \
                        COUNT(duration) AS Items \
                        FROM media_items m  \
                        LEFT JOIN library_sections l ON l.id = m.library_section_id  \
                        WHERE library_section_id > 0 GROUP BY name );"
sqlite3 -header -line "$db" "$query" | tee -a $logfile
echo " " | tee -a

query="select count(*) from media_items"
result=$(sqlite3 -header -line "$db" "$query")
echo "${result:11} files in library" | tee -a $logfile

query="select count(*) from media_items where bitrate is null"

This file has been truncated. show original

peterbaumert · November 17, 2021, 2:25pm

wow thank you for that hint, will look into it!
again i very much appreciate all the time you take for answering.
proof why rclone is great to use, besides the tool itself, there are great people trying to help

Animosity022 · November 17, 2021, 2:27pm

Best way to learn is to ask a lot of questions. I've learned so much more from seeing what other people do and how they get around things to help improve my own process.

peterbaumert · November 17, 2021, 2:30pm

that so true!

ok so the analyzing stuff is the issue

365039 files in library
123156 files missing analyzation info
1070 media_parts marked as deleted
1206 metadata_items marked as deleted
4 directories marked as deleted
357898 files missing deep analyzation info.

0 analyzed for intros
321867 not analyzed for intros
0 have intros

intros are off anyways, but i guess the 123156 files missing analyzation info part is the issue ^^
Will start to put some more love in the maintaining i guess, will checkout and steal some of your scripts though thx again!

Animosity022 · November 17, 2021, 2:33pm

I use:

github.com

animosity22/homescripts/blob/master/scripts/plex-analyze-cli

#!/usr/bin/env python3
# Plex analayze all files that are missing analyzation info
# OS: Ubunti 16.04 ( in case of other OS's make sure to change paths )

from subprocess import call
import os
import requests
import sqlite3
import sys

conn = sqlite3.connect('/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db')

c = conn.cursor()
c.execute('Select media_items.metadata_item_id As id, metadata_items.title As title From media_items Inner Join metadata_items On media_items.metadata_item_id = metadata_items.id Where media_items.bitrate Is Null And Not metadata_items.metadata_type = "12"')
items = c.fetchall()
conn.close()

print("To analyze: " + str( len(items) ))

for row in items:

This file has been truncated. show original

And that goes through one by one and analyzes the file. Note that does take quite some time and going through 123k files will take a few days if I had to guess.

You can stop that script at any point as I'd run it a few hours a day or something.

Once that missing analysis goes to 0, a scan should just legit walk the file system and do nothing else.

system · December 17, 2021, 2:33pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Get changeNotify in other Log Levels

What is the problem you are having with rclone?

What is your rclone version (output from rclone version)

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

The rclone config contents with secrets removed.

A log from the command with the -vv flag

What is your rclone version (output from `rclone version`)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

A log from the command with the `-vv` flag