My setup: VPS with 8 cores 8 ram with SSD. 1G connection. Ubuntu 14.04. Rclone mount to ACD with ENCFS and UnionFS-Fuse.
My mount: sudo nice -n -10 su jason -c “rclone mount --max-read-ahead 100M --read-only --allow-other --acd-templink-threshold 0 --stats 1s --buffer-size 1G --timeout 5s --contimeout 5s --log-file=$RCLONE_LOG --verbose ACD:Videos/edrive $ACDMOUNT_ENC &”
Tested this with regular linux amd64 and beta
When streaming all is nice and cozy. But if you start a library scan…whoa. It takes about 5 minutes to burn through all the ram and then the load average goes parabolic. I can see a streaming difference with the buffer size set at 1G so I would like to use it but the scanning shuts it all down. If I set the buffer at 100M then it runs ok but large streams start to buffer. Is this fixable? If not I guess i will have my script remount with a lower buffer to scan then again with 1G after the scan is done.
What’s the unionfs-fuse mount command you are using? You want to make sure to use auto_cache on that.
unionfs-fuse -o cow,allow_other,direct_io,auto_cache,sync_read “$LOCALMOUNT_DEC”=RW:"$ACDMOUNT_DEC"=RO “$UNIONMOUNT” 2>> “$LOGFILE”
Have you checked out the plex scripts from Ajki?
https://github.com/ajkis/scripts
If you run the library stats one, you can see how many files you have to analyze and it'll open up quite a number of files when it analyzes, which is probably the your buffer size is running away with all the memory.
If you run a:
lsof / you can see how many are open during the scan.
The key line is the analyze from the library-stats script:
16926 files in library
0 files missing analyzation info
0 media_parts marked as deleted
0 metadata_items marked as deleted
0 directories marked as deleted
15245 files missing deep analyzation info.
i have seen his stuff. library stats i keep getting Error: unable to open database file. not sure how to fix that. I have tried making the owner me, root and plex but still cant get it working
You can “sudo su - plex” and run them from the plex user or add in the plex group to your user.
For me, I added my local user that i normally use to the ‘plex’ group in /etc/group and my plexlibrary.db file has group read/write on it:
drwxr-xr-x 2 plex plex 11 May 9 05:02 .
drwxr-xr-x 7 plex plex 7 Apr 10 15:28 ..
-rw-rw-r-- 1 plex plex 69632 Apr 10 15:46 com.plexapp.dlna.db
-rw-rw-r-- 1 plex plex 32768 Apr 10 15:46 com.plexapp.dlna.db-shm
-rw-rw-r-- 1 plex plex 292552 Apr 10 15:46 com.plexapp.dlna.db-wal
-rw-rw-r-- 1 plex plex 84635648 May 10 20:14 com.plexapp.plugins.library.db
-rw-rw-r-- 1 plex plex 32768 May 10 20:14 com.plexapp.plugins.library.db-shm
-rw-rw-r-- 1 plex plex 1454656 May 10 20:14 com.plexapp.plugins.library.db-wal
-rw-rw-r-- 1 plex plex 811008 May 10 20:10 com.plexapp.plugins.trakttv.db
-rw-rw-r-- 1 plex plex 32768 May 10 20:14 com.plexapp.plugins.trakttv.db-shm
-rw-rw-r-- 1 plex plex 4136512 May 10 20:14 com.plexapp.plugins.trakttv.db-wal
felix@plex:/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases$ pwd
/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases
ugh that still didn’t work
What’s the error once you try to run it?
jason@TRZG001:~/plex/scripts$ . plex-library-stats.sh
10.05.2017 19:33:27 PLEX LIBRARY STATS
Media items in Libraries
Error: unable to open database file
Time to watch
Error: unable to open database file
Error: unable to open database file
files in library
Error: unable to open database file
files missing analyzation info
Error: unable to open database file
media_parts marked as deleted
Error: unable to open database file
metadata_items marked as deleted
Error: unable to open database file
directories marked as deleted
Error: unable to open database file
files missing deep analyzation info.
Are you running Ubuntu or something similiar?
What does this look like:
felix@plex:~/scripts$ ls -al "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db"
-rw-rw-r-- 1 plex plex 84638720 May 10 20:34 /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db
felix@plex:~/scripts$ id felix
uid=1000(felix) gid=1000(felix) groups=1000(felix),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),110(lxd),115(lpadmin),116(sambashare),117(plex)
felix@plex:~/scripts$
I have this also:
felix@plex:~/scripts$ grep plex /etc/group
video:x:44:plex
plex:x:117:felix
jason@TRZG001:~$ ls -al “/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db”
-rwxrwxrwx 1 plex plex 250880 Sep 11 2016 /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db
jason@TRZG001:~$ id jason
uid=1000(jason) gid=1000(jason) groups=1000(jason),27(sudo),106(fuse),110(plex),118(libvirtd)
jason@TRZG001:~$ grep plex /etc/group
plex:x:110:jason
also running Ubuntu 14.04
Can you change the first line to be and just re-run it. We can see better if it’s actually breaking on permissions or something else. Seems like your permissions are good.
#!/bin/bash -x
still exactly the same output
It should generate quite a bit of extra output:
felix@plex:~/scripts$ ./plex-library-stats
+ logfile=/home/felix/logs/plexstats.log
+ db='/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db'
+ tee -a /home/felix/logs/plexstats.log
++ date '+%d.%m.%Y %T'
+ echo '10.05.2017 20:43:50 PLEX LIBRARY STATS'
10.05.2017 20:43:50 PLEX LIBRARY STATS
+ echo 'Media items in Libraries'
+ tee -a /home/felix/logs/plexstats.log
Media items in Libraries
+ query='SELECT Library, Items FROM ( SELECT name AS Library, COUNT(duration) AS Items FROM media_items m LEFT JOIN library_sections l ON l.id = m.library_section_id WHERE library_section_id > 0 GROUP BY name );'
+ sqlite3 -header -line '/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db' 'SELECT Library, Items FROM ( SELECT name AS Library, COUNT(duration) AS Items FROM media_items m LEFT JOIN library_sections l ON l.id = m.library_section_id WHERE library_section_id > 0 GROUP BY name );'
+ tee -a /home/felix/logs/plexstats.log
Library = TV
Items = 13515
Library = Movies
Items = 1413
Library = Exercise
Items = 270
Library = MMA
Items = 41
Library = Misc
Items = 7
+ echo ' '
+ tee -a
+ query='select count(*) from media_items'
++ sqlite3 -header -line '/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db' 'select count(*) from media_items'
+ result='count(*) = 16926'
+ echo '16926 files in library'
+ tee -a /home/felix/logs/plexstats.log
16926 files in library
+ query='select count(*) From media_items Inner Join metadata_items On media_items.metadata_item_id = metadata_items.id Where media_items.bitrate Is Null And Not metadata_items.metadata_type = 12'
++ sqlite3 -header -line '/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db' 'select count(*) From media_items Inner Join metadata_items On media_items.metadata_item_id = metadata_items.id Where media_items.bitrate Is Null And Not metadata_items.metadata_type = 12'
+ result='count(*) = 0'
+ echo '0 files missing analyzation info'
+ tee -a /home/felix/logs/plexstats.log
0 files missing analyzation info
+ query='SELECT count(*) FROM media_parts WHERE deleted_at is not null'
++ sqlite3 -header -line '/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db' 'SELECT count(*) FROM media_parts WHERE deleted_at is not null'
+ result='count(*) = 0'
+ echo '0 media_parts marked as deleted'
+ tee -a /home/felix/logs/plexstats.log
0 media_parts marked as deleted
+ query='SELECT count(*) FROM metadata_items WHERE deleted_at is not null'
++ sqlite3 -header -line '/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db' 'SELECT count(*) FROM metadata_items WHERE deleted_at is not null'
+ result='count(*) = 0'
+ echo '0 metadata_items marked as deleted'
+ tee -a /home/felix/logs/plexstats.log
0 metadata_items marked as deleted
+ query='SELECT count(*) FROM directories WHERE deleted_at is not null'
++ sqlite3 -header -line '/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db' 'SELECT count(*) FROM directories WHERE deleted_at is not null'
+ result='count(*) = 0'
+ echo '0 directories marked as deleted'
+ tee -a /home/felix/logs/plexstats.log
0 directories marked as deleted
+ query='select count(*) from metadata_items meta join media_items media on media.metadata_item_id = meta.id join media_parts part on part.media_item_id = media.id where part.extra_data not like '\''%deepAnalysisVersion=2%'\'' and meta.metadata_type in (1, 4, 12) and part.file != '\'''\'';'
++ sqlite3 -header -line '/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db' 'select count(*) from metadata_items meta join media_items media on media.metadata_item_id = meta.id join media_parts part on part.media_item_id = media.id where part.extra_data not like '\''%deepAnalysisVersion=2%'\'' and meta.metadata_type in (1, 4, 12) and part.file != '\'''\'';'
+ result='count(*) = 15245'
+ echo '15245 files missing deep analyzation info.'
+ tee -a /home/felix/logs/plexstats.log
15245 files missing deep analyzation info.
+ exit
felix@plex:~/scripts$
I also changed the logfile to my user directory as well:
logfile="/home/felix/logs/plexstats.log"
weird. could there be a setting somewhere to enable -x
Edit the script and change the first line from:
#!/bin/bash
to
#!/bin/bash -x
Ah, I think I see.
Are you running it via:
. plex-library-stats
as opposed to
./plex-library-stats
I think yours has .sh at the end, but that shouldn’t matter.
hmm didn’t know there was a difference.
-
logfile=/home/jason/plex/logs/plexstats.log
-
[[ ! -f /home/jason/plex/logs/plexstats.log ]]
-
db=’/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db’
-
tee -a /home/jason/plex/logs/plexstats.log
++ date ‘+%d.%m.%Y %T’
-
echo ‘10.05.2017 19:58:18 PLEX LIBRARY STATS’
10.05.2017 19:58:18 PLEX LIBRARY STATS
-
echo ‘Media items in Libraries’
-
tee -a /home/jason/plex/logs/plexstats.log
Media items in Libraries
-
query=‘SELECT Library, Items FROM ( SELECT name AS Library, COUNT(duration) AS Items FROM media_items m LEFT JOIN library_sections l ON l.id = m.library_section_id WHERE library_section_id > 0 GROUP BY name );’
-
sqlite3 -header -line ‘/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db’ ‘SELECT Library, Items FROM ( SELECT name AS Library, COUNT(duration) AS Items FROM media_items m LEFT JOIN library_sections l ON l.id = m.library_section_id WHERE library_section_id > 0 GROUP BY name );’
-
tee -a /home/jason/plex/logs/plexstats.log
Error: unable to open database file
-
echo ’ ’
-
tee -a
-
echo ‘Time to watch’
-
tee -a /home/jason/plex/logs/plexstats.log
Time to watch
-
query=‘SELECT Library, Minutes, Hours, Days FROM ( SELECT name AS Library, SUM(duration)/1000/60 AS Minutes, SUM(duration)/1000/60/60 AS Hours, SUM(duration)/1000/60/60/24 AS Days FROM media_items m LEFT JOIN library_sections l ON l.id = m.library_section_id WHERE library_section_id > 0 GROUP BY name );’
-
sqlite3 -header -line ‘/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db’ ‘SELECT Library, Minutes, Hours, Days FROM ( SELECT name AS Library, SUM(duration)/1000/60 AS Minutes, SUM(duration)/1000/60/60 AS Hours, SUM(duration)/1000/60/60/24 AS Days FROM media_items m LEFT JOIN library_sections l ON l.id = m.library_section_id WHERE library_section_id > 0 GROUP BY name );’
-
tee -a /home/jason/plex/logs/plexstats.log
Error: unable to open database file
-
echo ’ ’
-
tee -a
-
query=‘Select count() From media_items Inner Join metadata_items On media_items.metadata_item_id = metadata_items.id Where Not metadata_items.metadata_type = 12’
++ sqlite3 -header -line ‘/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db’ 'Select count() From media_items Inner Join metadata_items On media_items.metadata_item_id = metadata_items.id Where Not metadata_items.metadata_type = 12’
Error: unable to open database file
-
result=
-
echo ’ files in library’
-
tee -a /home/jason/plex/logs/plexstats.log
files in library
-
query=‘Select count() From media_items Inner Join metadata_items On media_items.metadata_item_id = metadata_items.id Where Not metadata_items.metadata_type = 12 And media_items.bitrate Is Null’
++ sqlite3 -header -line ‘/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db’ 'Select count() From media_items Inner Join metadata_items On media_items.metadata_item_id = metadata_items.id Where Not metadata_items.metadata_type = 12 And media_items.bitrate Is Null’
Error: unable to open database file
-
result=
-
echo ’ files missing analyzation info’
-
tee -a /home/jason/plex/logs/plexstats.log
files missing analyzation info
-
query=‘SELECT count() FROM media_parts WHERE deleted_at is not null’
++ sqlite3 -header -line ‘/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db’ 'SELECT count() FROM media_parts WHERE deleted_at is not null’
Error: unable to open database file
-
result=
-
echo ’ media_parts marked as deleted’
-
tee -a /home/jason/plex/logs/plexstats.log
media_parts marked as deleted
-
query=‘SELECT count() FROM metadata_items WHERE deleted_at is not null’
++ sqlite3 -header -line ‘/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db’ 'SELECT count() FROM metadata_items WHERE deleted_at is not null’
Error: unable to open database file
-
result=
-
echo ’ metadata_items marked as deleted’
-
tee -a /home/jason/plex/logs/plexstats.log
metadata_items marked as deleted
-
query=‘SELECT count() FROM directories WHERE deleted_at is not null’
++ sqlite3 -header -line ‘/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db’ 'SELECT count() FROM directories WHERE deleted_at is not null’
Error: unable to open database file
-
result=
-
echo ’ directories marked as deleted’
-
tee -a /home/jason/plex/logs/plexstats.log
directories marked as deleted
-
query=‘select count() from metadata_items meta join media_items media on media.metadata_item_id = meta.id join media_parts part on part.media_item_id = media.id where part.extra_data not like ‘’’%deepAnalysisVersion=2%’’’ and meta.metadata_type in (1, 4, 12) and part.file != ‘’’’’’;’
++ sqlite3 -header -line ‘/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db’ 'select count() from metadata_items meta join media_items media on media.metadata_item_id = meta.id join media_parts part on part.media_item_id = media.id where part.extra_data not like ‘’’%deepAnalysisVersion=2%’’’ and meta.metadata_type in (1, 4, 12) and part.file != ‘’’’’’;’
Error: unable to open database file
-
result=
-
echo ’ files missing deep analyzation info.’
-
tee -a /home/jason/plex/logs/plexstats.log
files missing deep analyzation info.
-
exit