I'm having some difficulty getting the plex optimization feature to work properly when using an rclone mount. If I move the files to the hard drive and optimize there, they work. When I optimize from/to GDrive with rclone, I end up with corrupt files, although they seem to be the right size after a conversion takes place. (Can't be played in Plex or VLC)
What is your rclone version (output from rclone version)
rclone v1.48.0
- os/arch: linux/amd64
- go version: go1.12.6
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Meaning you are using the Optimize media feature in Plex and configured a particular format and are trying to let it optimize via that route?
If you run a test, I'd expect you'd see some errors in the logs as when it's writing the media, it would need vfs-cache-mode writes
I've played around with optimize and I always read from my GD and write locally and upload later via my whole mergerfs setup rather than trying to read from GD and write to GD as it's super slow and I rather just upload after it's done anyway.
I used to use a filter for my 4k stuff but since I've been hardware decoding, I don't care anymore as I have 4k and if people see bad HDR conversations, it's on them
I think vfs-cache-mode writes would work as it's essentially doing the same thing in terms of writing it locally first and uploading once it's done.
You can always toss it into debug mode and run a test and see if everything works properly and if not, we have a log to look at.
I use mergerfs in my setup as I have my own requirement for using hard links as I do not want double space when Radarr/Sonarr move things and mergerfs gives me that ability.
In your use case, it sounds like overhead and I try to keep it as simple as possible generally as less things can break.
Do you mean the plex debug log? (If so) It's miles long, and I can't discern what is important and what is sensitive. I'll need guidance on what to actually post.
I'm trying mergerfs, basically copying your setup from github. Still having all kinds of problems though. Some of them have to do with chown, which I think is sorted out now...
I'm optimizing a movie now, but in my local side of mergerfs, the directory is empty. From what I understand, I should be able to see what's written locally, then run rclone move on that local directory.
EDIT: That optimization failed as well.
I added a few flags to my mount command after reading through the mount docs some more.
Here's my current mount setup, which I'm starting with a cronjob instead of systemd (Don't yet know how to use systemd)
I just tried optimizing with --vfs-cache-mode full but the optimize process in plex hangs at 100% and the file never moves from .inProgress to the normal folder, then rclone moves the file to the cloud. That doesn't work as well as I expected it to.
I just tried optimizing with a cache remote and still it did not work. I found this in the logs. Looks like some kind of rename failure or something. Can't tell what's going on for sure. This is producing a file that cannot be played by plex or VLC. Something about rclone is causing the disruption, because they convert fine on local.
Log File
2019/07/30 12:12:28 DEBUG : Movies/Plex Versions/Optimized for Mobile/: Lookup: name="World Trade Center (2006).mp4.version"
2019/07/30 12:12:28 DEBUG : Cache remote plexcache:: list 'Movies/Plex Versions/Optimized for Mobile'
2019/07/30 12:12:28 DEBUG : Movies/Plex Versions/Optimized for Mobile: list: cold listing: 2019-07-28 12:12:28.269078107 -0600 MDT
2019/07/30 12:12:28 DEBUG : Movies/Plex Versions/Optimized for Mobile: list: read 0 from temp fs
2019/07/30 12:12:28 DEBUG : Movies/Plex Versions/Optimized for Mobile: list: temp fs entries: []
2019/07/30 12:12:29 DEBUG : Movies/Plex Versions/Optimized for Mobile: list: read 1 from source
2019/07/30 12:12:29 DEBUG : Movies/Plex Versions/Optimized for Mobile: list: source entries: [Movies/Plex Versions/Optimized for Mobile/.inProgress]
2019/07/30 12:12:29 DEBUG : Movies/Plex Versions/Optimized for Mobile: list: cached directories: 1
2019/07/30 12:12:29 DEBUG : Movies/Plex Versions/Optimized for Mobile: list: cached dir: 'Movies/Plex Versions/Optimized for Mobile', cache ts: 2019-07-30 12:12:29.266582715 -0600 MDT m=+700.030208467
2019/07/30 12:12:29 DEBUG : Movies/Plex Versions/Optimized for Mobile/: >Lookup: node=<nil>, err=no such file or directory
2019/07/30 12:12:29 DEBUG : /: Lookup: name="Movies"
2019/07/30 12:12:29 DEBUG : /: >Lookup: node=Movies/, err=<nil>
2019/07/30 12:12:29 DEBUG : Movies/: Attr:
2019/07/30 12:12:29 DEBUG : Movies/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2019/07/30 12:12:29 DEBUG : /: Lookup: name="Movies"
2019/07/30 12:12:29 DEBUG : /: >Lookup: node=Movies/, err=<nil>
2019/07/30 12:12:29 DEBUG : Movies/: Attr:
2019/07/30 12:12:29 DEBUG : Movies/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2019/07/30 12:12:29 DEBUG : Movies/: Lookup: name="Plex Versions"
2019/07/30 12:12:29 DEBUG : Movies/: >Lookup: node=Movies/Plex Versions/, err=<nil>
2019/07/30 12:12:29 DEBUG : Movies/Plex Versions/: Attr:
2019/07/30 12:12:29 DEBUG : Movies/Plex Versions/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2019/07/30 12:12:29 DEBUG : Movies/Plex Versions/: Lookup: name="Optimized for Mobile"
2019/07/30 12:12:29 DEBUG : Movies/Plex Versions/: >Lookup: node=Movies/Plex Versions/Optimized for Mobile/, err=<nil>
2019/07/30 12:12:29 DEBUG : Movies/Plex Versions/Optimized for Mobile/: Attr:
2019/07/30 12:12:29 DEBUG : Movies/Plex Versions/Optimized for Mobile/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2019/07/30 12:12:29 DEBUG : Movies/Plex Versions/Optimized for Mobile/: Lookup: name=".inProgress"
2019/07/30 12:12:29 DEBUG : Movies/Plex Versions/Optimized for Mobile/: >Lookup: node=Movies/Plex Versions/Optimized for Mobile/.inProgress/, err=<nil>
2019/07/30 12:12:29 DEBUG : Movies/Plex Versions/Optimized for Mobile/.inProgress/: Attr:
2019/07/30 12:12:29 DEBUG : Movies/Plex Versions/Optimized for Mobile/.inProgress/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2019/07/30 12:12:29 DEBUG : Movies/Plex Versions/Optimized for Mobile/.inProgress/: Lookup: name="World Trade Center (2006).mp4.110"
2019/07/30 12:12:29 DEBUG : Movies/Plex Versions/Optimized for Mobile/.inProgress/: >Lookup: node=Movies/Plex Versions/Optimized for Mobile/.inProgress/World Trade Center (2006).mp4.110, err=<nil>
2019/07/30 12:12:29 DEBUG : Movies/Plex Versions/Optimized for Mobile/.inProgress/World Trade Center (2006).mp4.110: Attr:
2019/07/30 12:12:29 DEBUG : Movies/Plex Versions/Optimized for Mobile/.inProgress/World Trade Center (2006).mp4.110: >Attr: a=valid=1s ino=0 size=772978378 mode=-rw-rw-r--, err=<nil>
2019/07/30 12:12:29 DEBUG : Movies/Plex Versions/Optimized for Mobile/: Lookup: name="World Trade Center (2006).mp4.version"
2019/07/30 12:12:29 DEBUG : Movies/Plex Versions/Optimized for Mobile/: >Lookup: node=<nil>, err=no such file or directory
2019/07/30 12:12:29 DEBUG : Movies/Plex Versions/Optimized for Mobile/.inProgress/: Rename: oldName="World Trade Center (2006).mp4.110", newName="World Trade Center (2006).mp4.version", newDir=Movies/Plex Versions/Optimized for Mobile/
2019/07/30 12:12:29 DEBUG : Cache remote plexcache:: new object 'Movies/Plex Versions/Optimized for Mobile/World Trade Center (2006).mp4.version'
2019/07/30 12:12:29 DEBUG : Movies/Plex Versions/Optimized for Mobile/World Trade Center (2006).mp4.version: find: error: couldn't find object (World Trade Center (2006).mp4.version)
2019/07/30 12:12:29 DEBUG : Movies/Plex Versions/Optimized for Mobile/World Trade Center (2006).mp4.version: find: not found in local cache fs
2019/07/30 12:12:29 DEBUG : find failed: not found in either local or remote fs
2019/07/30 12:12:29 DEBUG : Cache remote plexcache:: moving obj 'Movies/Plex Versions/Optimized for Mobile/.inProgress/World Trade Center (2006).mp4.110' -> Movies/Plex Versions/Optimized for Mobile/World Trade Center (2006).mp4.version
2019/07/30 12:12:30 DEBUG : Movies/Plex Versions/Optimized for Mobile/.inProgress/World Trade Center (2006).mp4.110: move: queued file moved to Movies/Plex Versions/Optimized for Mobile/World Trade Center (2006).mp4.version
2019/07/30 12:12:30 DEBUG : Movies/Plex Versions/Optimized for Mobile/World Trade Center (2006).mp4.version: move: file moved
2019/07/30 12:12:30 DEBUG : Movies/Plex Versions/Optimized for Mobile/.inProgress/World Trade Center (2006).mp4.110: move: removed from cache
2019/07/30 12:12:30 DEBUG : Movies/Plex Versions/Optimized for Mobile/.inProgress: cache: expired Movies/Plex Versions/Optimized for Mobile/.inProgress
2019/07/30 12:12:30 DEBUG : Movies/Plex Versions/Optimized for Mobile/.inProgress: cache: expired Movies/Plex Versions/Optimized for Mobile
2019/07/30 12:12:30 DEBUG : Movies/Plex Versions/Optimized for Mobile/.inProgress: cache: expired Movies/Plex Versions
2019/07/30 12:12:30 DEBUG : Movies/Plex Versions/Optimized for Mobile/.inProgress: cache: expired Movies
2019/07/30 12:12:30 DEBUG : Movies/Plex Versions/Optimized for Mobile/.inProgress: cache: expired
2019/07/30 12:12:30 INFO : Movies/Plex Versions/Optimized for Mobile/.inProgress: move: cache expired
2019/07/30 12:12:30 DEBUG : Movies/Plex Versions/Optimized for Mobile: invalidating directory cache
2019/07/30 12:12:30 DEBUG : Movies/Plex Versions/Optimized for Mobile/.inProgress: forgetting directory cache
2019/07/30 12:12:30 DEBUG : Movies/Plex Versions/Optimized for Mobile/World Trade Center (2006).mp4.version: pending upload found
2019/07/30 12:12:30 DEBUG : Movies/Plex Versions/Optimized for Mobile/World Trade Center (2006).mp4.version: move: added to cache
2019/07/30 12:12:30 DEBUG : Movies/Plex Versions/Optimized for Mobile: cache: expired Movies/Plex Versions/Optimized for Mobile
2019/07/30 12:12:30 DEBUG : Movies/Plex Versions/Optimized for Mobile: cache: expired Movies/Plex Versions
2019/07/30 12:12:30 DEBUG : Movies/Plex Versions/Optimized for Mobile: cache: expired Movies
2019/07/30 12:12:30 DEBUG : Movies/Plex Versions/Optimized for Mobile: cache: expired
2019/07/30 12:12:30 INFO : Movies/Plex Versions/Optimized for Mobile: move: cache expired
2019/07/30 12:12:30 DEBUG : Movies/Plex Versions: invalidating directory cache
2019/07/30 12:12:30 DEBUG : Movies/Plex Versions/Optimized for Mobile/.inProgress: forgetting directory cache
2019/07/30 12:12:30 DEBUG : Movies/Plex Versions/Optimized for Mobile: forgetting directory cache
2019/07/30 12:12:30 INFO : Movies/Plex Versions/Optimized for Mobile/.inProgress/World Trade Center (2006).mp4.110: Moved (server side)
2019/07/30 12:12:30 DEBUG : Movies/Plex Versions/Optimized for Mobile/.inProgress/World Trade Center (2006).mp4.110: Updating file with Movies/Plex Versions/Optimized for Mobile/World Trade Center (2006).mp4.version 0xc00029cb40
2019/07/30 12:12:30 DEBUG : Movies/Plex Versions/Optimized for Mobile/.inProgress/: >Rename: err=<nil>
It would be nice to be able to run some kind of for loop and just rclone copy the file to local disk, plex transcode it, then rclone move it to the appropriate folder. I can't figure that out either though.
EDIT: I also found this error right at the end of the actual transcode, as far as I could tell.
2019/07/30 12:12:27 ERROR : Movies/Plex Versions/Optimized for Mobile/.inProgress/World Trade Center (2006).mp4.110: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2019/07/30 12:12:27 DEBUG : &{Movies/Plex Versions/Optimized for Mobile/.inProgress/World Trade Center (2006).mp4.110 (w)}: >Write: written=0, err=illegal seek
2019/07/30 12:12:27 DEBUG : /: Lookup: name="Movies"
full will download the file in its entirety before allowing it to be read which generally isn't what you want. Whereas writes will stream a file from the internet when only reading.
Is this feature simply not compatible with rclone? Or am I still doing something wrong? It seems like I've tried every combination of things, and there's no way to optimize. The only thing I haven't tried is vfs-cache-mode full in conjunction with a cache remote. Is that even worth trying? Hours upon hours of labor, nothing to show for it. I'm no closer than before, I've just eliminated all the possibilities of getting it working, it seems.
Trying afresh with rclone mount plex: mount --allow-other --buffer-size 1G --dir-cache-time 96h --log-level DEBUG --log-file /home/ubuntu/r.log --timeout 1h --umask 002 --rc --vfs-cache-mode full
based on a post by Animosity 11 days ago elsewhere in the forums.
I'd like to keep this as simple as possible. Is there anything else I should add to this? Maybe --vfs-cache-mode full? I don't mind it taking a minute to start playing a video, as long as I can optimize my media somehow.
Cache mode writes is definitely the safest option. It certainly shouldn't make anything worse at the very least. with writes it enables the OS to act upon the file much more like it is a local file (because it IS a local file that gets uploaded later when closed). Not using writes mode will mean the file-access will be very limited in terms of what can be done to it - so I would not recommend that, especially not for something like this.
Do you use a cache-backend as well by any chance? .. if you do (and your dir-cache-time is larger than your caches info-age) then it can sometimes cause a problem potentially leading to corruption in some cases. Longshot, but I want to just eliminate that potential problem.
Your buffer size is also abnormally large. I very much doubt you need that. I think you are misunderstanding what that does when you decided to set it to 1G. It shouldn't directly affect this issue, but it's worth mentioning.
I'm trying it right now with cache mode full. Cache mode writes didn't help.
I've tried the cache backend on and off several times, but it hasn't helped. I enjoy the simplicity of not having the extra cache layer since it doesn't seem to do anything (for my use case).
I can try again and check on my dir-cache-time and info-age numbers, make sure that stuff is right.
I think cache mode full is rather pointless. It will work just as writes for actually writing things to the cloud, but full will make opening large files for read downright painful. Don't use it unless you have a very good reason for it IMO.
Not using the cache backend certainly won't hurt you in this case. Unless you use the cache-writes or temp-upload features for it then it won't touch your uploads at all and do nothing. If you can leave it off for the purpose of troubleshooting this specific issue at least, then do so.
A large buffer-size isn't "wrong", 1GB is just beyond excessive (default is 16MB). I think Animosity just said 1G in that case because the posters original value was an even more absurd number. What this actually does (to the best of my understanding) is try to smooth out the delays that your for example your local drive (particularly if a HDD) might have in reading or writing requested data for example. So rather than pausing the transfer when these small pauses happen the data goes into the buffer so you don't waste bandwidth. In some spesific cases where data is actually being moved around locally a buffer might be kind of pointless and 0 could be "all you actually need", but having a buffer shouldn't ever hurt anything I think (except maybe wasting a little memory?).
Is there an option in plex to control what temp-folder is used for "in progress" files like this? Because it obviously uses a temp-folder according to your log in first post. I would think there is a decent chance for there being an option or config to set this, just like in torrent programs for example (I don't use plex myself). If that exists then see if you can set that folder to a local one. The problem here most likely stems from how plex is accessing files during it's creation. If it can do that in peace on the local system and then just move it automatically to the final destination when done then that would avoid a whole host of potential problems.
This is the solution I have to use if I want to download a torrent to my cloud drive - because trying to let it assemble the file directly there leads to all sorts of issues. Since it probably opens and closes it multiple times - unfinished files get sent out, then asked to be read back, then sent out again (still unfinished) and aside from it being horrendously inefficient and slow it's also likely prone to errors. I imagine something similar may be the case here.
