Can you share the full debug log as you have snippets missing.
Unfortunately there isn't a way to customize where it works. Plex Optimize is very poorly documented as well. It's all point'n'click stuff, very basic.
I've got buffer-size set to 0 now. I'm running on an SSD anyway, so that shouldn't be a problem.
Here's the (full) log from my latest optimize run.
Command:
rclone mount plex: mount --allow-other --buffer-size 0 --vfs-cache-max-size 18G --dir-cache-time 96h --log-level DEBUG --log-file /home/ubuntu/writes-r.log --timeout 1h --umask 002 --rc --vfs-cache-mode writes
EDIT: In my google drive trash, I'm finding tons of 1 byte files in there that I think plex is creating and rclone is uploading. Can I apply a --min-age filter on a mount? Would that help or complicate things?
I just did a very simple test as I converted 1 tv show:
felix 5212 25049 2 19:22 pts/0 00:00:14 rclone mount gcrypt: /Test --vfs-cache-mode writes -vv
I optimized it and it wrote the file properly. I can mediainfo the file.
[felix@gemini The Orville]$ pwd
/Test/test/Plex Versions/Optimized for Mobile/The Orville
[felix@gemini The Orville]$ mediainfo S02E06.mp4
General
Complete name : S02E06.mp4
Format : MPEG-4
Format profile : Base Media
Codec ID : isom (isom/iso2/avc1/mp41)
File size : 647 MiB
Duration : 48 min 26 s
Overall bit rate mode : Variable
Overall bit rate : 1 867 kb/s
Writing application : Lavf58.9.100
Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L5.1
Format settings : CABAC / 2 Ref Frames
Format settings, CABAC : Yes
Format settings, Reference frames : 2 frames
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 48 min 26 s
Bit rate mode : Variable
Bit rate : 1 734 kb/s
Maximum bit rate : 2 487 kb/s
Width : 1 280 pixels
Height : 720 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 23.976 (23976/1000) FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.078
Stream size : 601 MiB (93%)
Codec configuration box : avcC
Audio
ID : 2
Format : AAC LC SBR
Format/Info : Advanced Audio Codec Low Complexity with Spectral Band Replication
Commercial name : HE-AAC
Format settings : Implicit
Codec ID : mp4a-40-2
Duration : 48 min 26 s
Bit rate mode : Constant
Bit rate : 127 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Frame rate : 23.438 FPS (2048 SPF)
Compression mode : Lossy
Stream size : 43.9 MiB (7%)
Default : Yes
Alternate group : 1
The file plays fine in plex as I picked that version and it played about 2 minutes of it before I stopped it.
From the rclone logs, I'm having a hard time figuring out what part of the optimize is going on as you have a bunch of things in motion as I can see some sidecar files, some files named .106 on the end.
Are you running some filter or something to optimize a group of stuff? You may want to pick one file and make sure things are working before going big.
You can't filter the mount, but you could use a --min-size to tell rclone to not transfer files smaller than that. I think that would pass through to rclone even for files going through a mount but I never tested something like that.
It's very hard to say if that would do anything without know what exactly those tiny files are... what are their names and extensions? I know a little bit about video encoding, so maybe I can decipher what their function are.
I had to re-post the log as a google drive link. The Github Gist only had 7K lines, when the log actually has about 50K lines.
I'm only optimizing one video in there, nothing else. I'm trying to keep it as simple as possible, and even sometimes optimizing to a very low quality so that it finishes quickly- trying to get one working test run going before going big.
Whatever else is going on in there must be plex. (sidecar/106 or whatever) I only run rclone and plex on this server, nothing else. Nobody is streaming anything, just that one optimize job of one video going on there.
EDIT: In case it's relevant, I'm using the latest version of plex media server.
Their names look like encrypted rclone files, and they have no extensions. All jibberish.
o56rhg0328q7gp293h0948ty0q3arhgoerh09t348
I'm not using a crypt here though. Just plain regular mount.
When it finishes, what do you see in the optimized folder? Are you keeping it with the media?
My best guess is they are some sort of temporary workfile for plex while it's processing the file - but no idea how that factors in to this problem, sorry.
Yes, I'm keeping it in the Plex Versions folder. Sometimes there is another (subtitle) file in there (.srt file, I believe)
The file size looks right, but it won't play. Not in plex, not in VLC. Broken, for some reason.
I'm seeing all of these little files copied then removed.
Could this be, for some reason, the cause of my troubles? Maybe plex needs these, and rclone is offloading them too soon, and plex isn't seeing them in the mount because it hasn't updated lately?
Can I tell rclone, in my mount command, to use --min-size, as @thestigma suggested? Will that work?
Little Files Being Moved, Log Level INFO (copied, then removed from cache)
2019/07/30 17:58:39 INFO : Movies/Plex Versions/Optimized for Mobile/2f56ee96-ad20-4c43-9083-48024d37cf7a: Copied (new)
2019/07/30 17:58:39 INFO : Movies/Plex Versions/Optimized for Mobile/2f56ee96-ad20-4c43-9083-48024d37cf7a: Removed from cache
2019/07/30 17:58:41 INFO : Movies/Plex Versions/Optimized for Mobile/d07811e6-9269-4bcb-986f-ddcf045f9345: Copied (new)
2019/07/30 17:58:41 INFO : Movies/Plex Versions/Optimized for Mobile/d07811e6-9269-4bcb-986f-ddcf045f9345: Removed from cache
2019/07/30 17:58:42 INFO : Movies/Plex Versions/Optimized for Mobile/45081da0-e49d-4bf5-8933-77e4f52e274b: Copied (new)
2019/07/30 17:58:43 INFO : Movies/Plex Versions/Optimized for Mobile/45081da0-e49d-4bf5-8933-77e4f52e274b: Removed from cache
2019/07/30 17:58:44 INFO : Movies/Plex Versions/Optimized for Mobile/555c2b74-8977-4e4b-ab80-b3e7a51cf70e: Copied (new)
2019/07/30 17:58:45 INFO : Movies/Plex Versions/Optimized for Mobile/555c2b74-8977-4e4b-ab80-b3e7a51cf70e: Removed from cache
2019/07/30 17:58:46 INFO : Movies/Plex Versions/Optimized for Mobile/5d657412-cbdd-46d8-8970-ce82f42f63ac: Copied (new)
2019/07/30 17:58:47 INFO : Movies/Plex Versions/Optimized for Mobile/5d657412-cbdd-46d8-8970-ce82f42f63ac: Removed from cache
2019/07/30 17:58:48 INFO : Movies/Plex Versions/Optimized for Mobile/edebdabb-fd9c-4965-bfe3-48bb93d0124f: Copied (new)
2019/07/30 17:58:49 INFO : Movies/Plex Versions/Optimized for Mobile/edebdabb-fd9c-4965-bfe3-48bb93d0124f: Removed from cache
2019/07/30 17:58:50 INFO : Movies/Plex Versions/Optimized for Mobile/a31770a6-2c72-441d-99be-5487aa1d06bc: Copied (new)
2019/07/30 17:58:51 INFO : Movies/Plex Versions/Optimized for Mobile/a31770a6-2c72-441d-99be-5487aa1d06bc: Removed from cache
2019/07/30 17:58:54 INFO : Movies/Plex Versions/Optimized for Mobile/edab5f46-cc3f-498d-9a2d-90346799afcb: Copied (new)
2019/07/30 17:58:55 INFO : Movies/Plex Versions/Optimized for Mobile/edab5f46-cc3f-498d-9a2d-90346799afcb: Removed from cache
2019/07/30 17:58:57 INFO : Movies/Plex Versions/Optimized for Mobile/7150f5b1-d089-4e53-9665-66b210461d26: Copied (new)
2019/07/30 17:58:57 INFO : Movies/Plex Versions/Optimized for Mobile/7150f5b1-d089-4e53-9665-66b210461d26: Removed from cache
2019/07/30 17:58:59 INFO : Movies/Plex Versions/Optimized for Mobile/9baa4c8e-3520-4612-8f9b-c2500e748a16: Copied (new)
2019/07/30 17:58:59 INFO : Movies/Plex Versions/Optimized for Mobile/9baa4c8e-3520-4612-8f9b-c2500e748a16: Removed from cache
2019/07/30 17:59:00 INFO : Movies/Plex Versions/Optimized for Mobile/a2900f54-f887-41e1-8bbd-a13cdfa8368e: Copied (new)
2019/07/30 17:59:01 INFO : Movies/Plex Versions/Optimized for Mobile/a2900f54-f887-41e1-8bbd-a13cdfa8368e: Removed from cache
2019/07/30 17:59:02 INFO : Movies/Plex Versions/Optimized for Mobile/ac80d809-5c61-465a-9414-bb4ab10594b6: Copied (new)
2019/07/30 17:59:03 INFO : Movies/Plex Versions/Optimized for Mobile/ac80d809-5c61-465a-9414-bb4ab10594b6: Removed from cache
2019/07/30 17:59:04 INFO : Movies/Plex Versions/Optimized for Mobile/6be05039-68cc-4b51-ad41-884e85502117: Copied (new)
EDIT: And here is a good link to the full log from before, all 50K+ lines instead of only 7K.
I'm also finding that the files in the Plex Versions folder in Google drive and in the extension .mp4, but if I download the file, it ends with a number. Movie.mp4.126
While it's transcoding in plex, if I ls the directory, I see it with that number as well. I'm not sure if that's something plex needs, or not.
It's plausible it could be related - but at least in theory there should be a seamless swapping from files being in the cache to being on the drive, so files shouldn't be getting "lost" or be a different one than expected because something was outdated. But that's in theory... and if in this case these tiny workfiles are being constantly overwritten at a rapid pace then I wouldn't be super surprised if they trigger some edge-case that rclone just can't keep up with.
That's why generally these sorts of complex file-writes are best done locally and uploaded when done. Without any config or option to direct the temp-folder to be local I don't know exactly what options or workaround you have in this regard.
Maybe this isn't addressing the real problem, but you can't just make them locally and then move them in plex after? Or even use a separate software to handle this spesific task of making mobile-optimized versions? Just ideas to consider.
Yeah, I've thought about both of those last options you suggested.
Local: I can't automate the process of doing it locally. I have 30GB space local, and several TB of files in the cloud. I'd have to manually move about 12GB at a time, optimize with plex, then move the new version off and delete the old version, keeping a checklist of what has/hasn't been optimized. Not a good deal there. That would take so long, I'd sooner ditch plex and just watch straight out of Google Drive and try to figure out how to cast that to a TV screen. (Not ideal)
Other Software: This isn't a terrible idea. I've had some success with for f in *; do rclone copy "$f" ~/work -P; HandBrakeCLI -Z "Vimeo YouTube 720p30" -s scan -F --subtitle-burned -N eng -i "$f" -o ~/work/"${f%.*}.mp4"; rm ~/work/*; rclone move ~/work plex:720 -P; done, but the problem with doing this is that I won't have a consolidated library. I think I would have to create a new plex library specifically for the optimized versions. It would be like going onto netflix and choosing which network you want to view the available media in, 480, 720, 1080, etc. Keeping all that organized over time would be a total nightmare, I think.
I'm open to other ideas though.
I tried Kodi and did not like that one bit.
Those could possibly be partial segments of an incomplete file. Video encoding / transcoding is often a multi-stage process where individual streams are rendered, video, audio, subtitles, metadata ect. (sometimes first in segments) and then they are all re-assembled (muxed) into a modern media-file which is really a sort of container for all that various data.
So unless the software does this all internally in memory before writing the resulting file (which is rarely done because of the sizes involved) there is often a lot of "temporary assembly parts" at work. These parts may or may not have human-readable names, but my best guess would be that the 126 signifies it's part 126 of some part of it.
Problem of course is that rclone and the mount cache doesn't know what is temporary data and what isn't. it will simply upload files once they are closed - believing them to be finished and thus ready for upload. If you could delay the mount from syncing the files after they close that might alleviate some of this but... I don't think an option for that exists.
Makes sense.
I thought there was a way to set up a cache remote such that it won't upload/move files unless they are older than a certain age. Is that not possible? I might have been dreaming about rclone move with a filter.
You can filter rclone in many ways:
https://rclone.org/filtering/
But this applies to rclone itself. It's not a setting for the mount or cache.
Doing it on a move or sync is certainly very possible, because that's purely rclone working on it without any middleman backends involved. That's the kind of use it's meant for.
But I believe the method of upload for the mount cache is basically a rclone sync command (rather than the mount having some internal mechanism to do it), so hopefully any filtering flag you use should also apply to files being uploaded from the mounts cache.
You can by all means try to add a --min-size 1M or something and see what happens but I really have no idea how that ends up affecting things (and you probably only want to use that temporarily even if it does work since it will obviously block your intentional small uploads too)
I just tried your method.
rclone mount plex: mount/ --vfs-cache-mode writes --log-level DEBUG --log-file rclone.log
I tried to optimize Season 1 Episode 1 of Planet Earth, it failed immediately.
Here's my debug log...
2019/07/30 19:16:07 DEBUG : rclone: Version "v1.48.0" starting with parameters ["rclone" " mount" "plex:" "mount/" "--vfs-cache-mode" "writes" "--log-level" "DEBUG" "--log-file" "/home/ubuntu/rclone.log"]
2019/07/30 19:16:07 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2019/07/30 19:16:07 DEBUG : Google drive root '': Mounting on "mount/"
2019/07/30 19:16:08 DEBUG : vfs cache root is "/home/ubuntu/.cache/rclone/vfs/plex"
2019/07/30 19:16:08 DEBUG : Adding path "vfs/forget" to remote control registry
2019/07/30 19:16:08 DEBUG : Adding path "vfs/refresh" to remote control registry
2019/07/30 19:16:08 DEBUG : Adding path "vfs/poll-interval" to remote control registry
2019/07/30 19:16:08 DEBUG : : Root:
2019/07/30 19:16:08 DEBUG : : >Root: node=/, err=<nil>
2019/07/30 19:16:08 DEBUG : Movies/Apollo 11 (2019) (1080p HD).mp4: updateTime: setting atime to 2019-07-30 18:23:26.137637859 -0600 MDT
2019/07/30 19:16:08 DEBUG : Movies/Plex Versions/Optimized for Mobile/.inProgress/Apollo 11 (2019).mp4.126: updateTime: setting atime to 2019-07-30 19:00:51.476492486 -0600 MDT
2019/07/30 19:16:08 DEBUG : Movies/Plex Versions/Optimized for Mobile/.inProgress/Apollo 11 (2019).mp4.126.148.sidecar: updateTime: setting atime to 2019-07-30 19:07:24.107599233 -0600 MDT
2019/07/30 19:16:08 DEBUG : Movies/Plex Versions/Optimized for Mobile/Apollo 11 (2019).mp4: updateTime: setting atime to 2019-07-30 19:11:16.774692696 -0600 MDT
2019/07/30 19:16:08 DEBUG : Movies/Plex Versions/Optimized for Mobile/Apollo 11 (2019).mp4.version: updateTime: setting atime to 2019-07-30 19:10:59.938764462 -0600 MDT
2019/07/30 19:16:08 DEBUG : Movies/Plex Versions/Optimized for Mobile/Beauty and the Beast (1991).mp4: updateTime: setting atime to 2019-07-30 18:20:29.16762507 -0600 MDT
2019/07/30 19:16:08 DEBUG : Movies/Plex Versions/Optimized for Mobile/Beauty and the Beast (1991).mp4.version: updateTime: setting atime to 2019-07-30 18:16:04.50651778 -0600 MDT
2019/07/30 19:16:08 DEBUG : Movies/Plex Versions/Optimized for Mobile/d5a98f96-c8d2-4a9e-88b4-084d6939d179: updateTime: setting atime to 2019-07-30 19:12:03.334490068 -0600 MDT
2019/07/30 19:16:08 INFO : Movies/Plex Versions/Optimized for Mobile/Beauty and the Beast (1991).mp4.version: Removed from cache
2019/07/30 19:16:08 INFO : Cleaned the cache: objects 12 (was 0), total size 12.189G (was 0)
2019/07/30 19:16:56 DEBUG : /: Attr:
2019/07/30 19:16:56 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>I recreated your issue so let me take a look as it requires a bit of time to go through the logs.
The first time worked for me as I was dropping some files in my mergerfs mount by mistake.
I'm willing to implement mergerfs. Just, when I did, it performed just the same as if I didn't have it set up.
I can't figure out a way to get things to write to the local disk and not back to the cache mount.
I used this command to attempt to get that to work as my solution, but no joy:
mergerfs /home/ubuntu/local:/home/ubuntu/gdrive /home/ubuntu/mount/ -o async_read=false,use_ino,allow_other,auto_cache,func.getattr=newest,category.action=all,category.create=ff
/local/ was an empty folder, /gdrive/ was my rclone mount, and /mount/ was an empty folder that plex is pointed at. With that setup, running optimize, I had the same problems as without mergerfs.
mergerrfs works by combining two mounts.
You'd have:
/home/ubuntu/local
and
/home/ubuntu/gdrive
That would combine to
/home/ubuntu/mount
and when you write something to /home/ubuntu/mount
You should see something write to /home/ubuntu/local
If not, you got something not quite right.