Plex server with Amazon drive - rclone crypt vs encFS speed

Suspected Bug
1

Hello,

I suspect what I’m about to mention may become a hot topic now that Amazon Drive is more widely available internationally. This post will be in the thread of this other one: Mounting ACD with decryption for reading but I did not want to hijack that thread.

I’ve been doing some testing tonight with a remote Plex Server (not Plex Cloud) on a Linode VPS, connected in the backend to Amazon Cloud Drive to host the files (VPS storage is ridiculously expensive). This is basically a mimic of Plex Cloud that was just released, but my objective is to control it and also use encryption.

Now that rclone has encryption, I thought I’d test some of the speed with no encryption, encFS and rclone crypt.

Setup

  1. Linode 4GB - https://www.linode.com/pricing
  2. Amazon Cloud Drive
  3. rclone, acd_cli, encfs
  4. ACD mounted with acd_cli to /mnt/acd

Folders

On my ACD, I’ve setup a “Media” folder and three sub-folders:

  1. encfs
  2. rclone
  3. native (this is non-encrypted)

encFS

This assumes you have encFS installed.

  1. Mount an encfs folder: ENCFS6_CONFIG='~/encfs.xml' encfs /mnt/acd/Media/encfs /mnt/encfs
  2. make your movie folder: mkdir -p /mnt/encfs/movies/MyMovie
  3. Copy your movie: cp MyMovie.mkv /mnt/encfs/movies/MyMovie
  4. Your movie will be copied to ACD and encrypted using encFS

rclone crypt

This assumes you have rclone installed and a crypt setup.

  1. rclone your movie to acd encrypted: rclone -v copy ./path/to/MyMovie/ acdenc:Media/rclone/movies/MyMovie
  2. Mount your encrypted media folder: rclone -v mount acdenc:Media/rclone/ /mnt/rclone

Native

This assumes you have ACD mounted to /mnt/acd

  1. Copy your movie file directly to ACD: cp -r /path/to/MyMovie /mnt/acd/Media/native/movies/MyMovie/

Plex

This assumes you know how to setup Plex.

  1. Create a new Movies library in Plex: Encryption Comparison
  2. Try native first by adding to this new library the /mnt/acd/Media/native folder.
  3. Test playback (should be very quick).
  4. Edit the library and remove the folder above, add a new folder: /mnt/encfs.
  5. Rescan the library to pick up the new changes
  6. Test playback (should be quick, but not as quick as native)
  7. Edit the library and remove the folder from #4 and add a new folder: /mnt/rclone.
  8. Rescan the library to pick up the new changes
  9. Test playback - this will be hit or miss, but it’s unreliable.

Conclusion

I am certain that the cryptographic algorithms used for rclone crypt add a significant overhead that isn’t seen with encFS. Using encFS, the playback is almost as fast as native for me. Seeking is also very reliable. I would say that the remote Plex server is almost as fast as a local one (those Linode VPS are very quick).

The rclone crypt + mount option (which I was hoping to use for simplicity) is very slow and highly unreliable. Plex often chokes trying to transcode the files (I’m guessing due to overhead). I’ve tried different mount options but no joy I’m afraid.

I’m not sure if any improvements to the speed of crypt can be done to make it more in line with encFS. I know there is a feature request to support encFS as a legacy mechanism.

It’s too bad, rclone + crypt is far simpler, it’s an all-in-one package. I know it was probably never devised to be used this way, but I’m hoping some performance gains can yet be had.

Hope this helps other people thinking of doing the same.

1 Like
2

An interesting set of tests - thanks for writing them up.

I think the weak part of the rclone setup is the FUSE mount rather than the crypto. The crypto only adds a small overhead (my laptop can encrypt at 400 MB/s per core).

The FUSE mount has been improving greatly - make sure you try the latest beta: http://beta.rclone.org/v1.33-79-g77b975d/ if you didn't already. I find I can play encrypted media quite well off ACD now, though I haven't compared it to encfs+acd_cli.

The FUSE mount for rclone needs more caching I think.

I'd like it to be useable in exactly that way and I'm sure it will be one day when all the planned FUSE improvements are in :slight_smile:

1 Like
3

Hi @ncw

Thanks for the feedback. Just one clarification to add.

In my tests, ACD was always mounted using rclone mount. So even when I used encFS, I did not use acd_cli to mount the ACD. I don’t like the fact that you have to run acd_cli sync to pick up the new changes, so I tend to stay about from it. Even when I sent the data to ACD, I’d do an encFS reverse encryption and sync the encrypted folder to ACD via rclone.

Which means that the speed differences between encFS vs Crypt tests I did I believe are purely based on crypt vs encFS, not crypt+rclone mount vs encFS+acd_cli. I do not believe the limitations (from a Plex standpoint) are because of FUSE.

It’s possible it’s the way Plex handles the media files for streaming. In Plex, you can navigate to an object and download it via the web interface. The downloads (both Crypt/encFS) are very fast so I know that that the decryption is happening well on both. It’s just starting to stream something from Plex when using crypt often takes 20-30 seconds, times out (errors in the Plex logs that it can’t seem to buffer the data fast enough), but none of this with EncFS.

I haven’t tried yesterday’s beta. Any improvements over Friday’s beta?

4

lol took long time to post . bussy

I am using rclone + crypt.
Direct play on anything higher than 20 gb will buffer but that’s due to I need faster home internet.
I have used VPS for plex for a long time now normally just have VPN from the VPS to my NAS at home and best result to cost for me is to force streaming.

Fastest I have seen is 250-300 Mbps from amazon to the VPS with encryption on
(I have since downgraded the system) but when using stable release, it was slower was lucky to see above 100Mbps but there has been some work done, the Beta’s run well.
This is all encrypted
Yes, some optimization would be great, there was a suggestion in one of the other posts.
Some other things implanted/address in last beta of to test it later.
I Have not used encFS so I cannot compare.
I watched about 5 hours of tv/movies from it last night was very stable only issue is starting a movie 2/3 in on a large file.
I decided to spend a little more on a VPS this time, so I am using a VPS with guaranteed resources so the issues I was seeing due to network or host issue have now gone.

My ACD structure
• Plex (unencrypted files for testing)
• Plex-Crypt (encryption folder)
• Folder A (unencrypted files)
• Folder B (unencrypted files)

Plex install-
Operating system: CentOS 7

  1. Update system.
  2. Install any utility you would like to use.
  3. Install and configure plex.
  4. Add IPatbles/firewall rule.
  5. Create folders where you intend to mount the network drive give full ownership to plex (better to mount to a sub-folder in main directory).
  6. Install fuse.
  7. Copy pre-configured rclone to the VPS place it in Plex home drive give read, write and execute. access to required folders and files.
  8. Create a mount.service file then do the required system work. 
  9. I have plex start the mount, I did have a limit CPU usage command in this when I was using a cheap VPS

[Unit]
Description=Mount

[Service]
ExecStart=/var/lib/plexmediaserver/rclone/rclone mount -v --no-modtime secret: /your/path
ExecStop=/bin/fusermount -u /your/path
Type=oneshot
User=plex
Group=plex
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

My sync command
On local sync server, I just have a script or even a crontab entry. it just runs only sync one way
./rclone --exclude-from exclude-file.txt --transfers=1 -v sync /your/files/ secret:

5

I think it may have to do with the way Plex streams and segments the file using HLS (HTTP Live Streaming) that is different with crypt and encfs.

Trying to be methodical here, here’s the constants/variables in my tests:

Constants

  1. Linode 4GB VPS
  2. ACD mounted via rclone mount command in two spots: /mnt/rclone (using encrypted config) and /mnt/encfs (using raw config)
  3. The Plex Media Server + settings
  4. My Apple TV / iPhone that are being used as test clients.
    5.Same original movie file (unencrypted file I’m testing with.

Variables

  1. Two libraries in Plex: Movies (rclone) and Movies (encFS)
  2. Encryption format of the movie file
  3. For encFS, the encfs is layered on top of the ACD rclone mount to allow Plex to read the files.

More results

rclone encrypted file

  1. Movie takes longer to start streaming
  2. Movie will often pause or “jump” ahead, sometimes the movie will just stop, sending you back to the Plex menu.
  3. Movie sometimes does not start at the beginning, but instead 5-30 seconds into the movie
  4. Skipping ahead is hit or miss (although there is a known issue for this).
  5. Downloading the raw file from Plex shows speeds of 5-10MB/s, more than enough bandwidth from the VPS to my house to stream a few streams at a time. This tests the raw decryption speed + download from the VPS to my house.

encfs encrypted file

  1. Movie starts much quicker than the rclone encrypted file
  2. Movie does not jump ahead
  3. Movie always starts at beginning
  4. Skipping ahead is much smoother than rclone, but can still takes 5-10 seconds before it picks it back up.
  5. Downloading the raw file from Plex shows speeds of 5-10MB/s, more than enough bandwidth from the VPS to my house to stream a few streams at a time. No difference between rclone and encfs on the download.

As I mentioned, the Plex logs are littered with errors that it cannot pull enough segments when using rclone. By default, Plex segments the file for HLS streaming using 10 second segments and feeds those to the client. By default again, it will buffer 6 segments (60 seconds - this is configurable) ahead. As sessions are played, it queues up more segments. If it runs out of segments or can fill its cache, that is when buffering/complete stop will occur. I’ve tried bumping this to 120, even 300, but it doesn’t change anything.

Again, I feel this may be specific to Plex. I will attempt streaming via VLC a bit later, but I do not suspect it to be slow. If the raw download speed shows 5-10MB/s, chances are VLC will play fine.

I’m not entirely certain what is going on, but I can vouch that the rclone encrypted data is much more sporadic of getting a valid stream and holding it. With EncFS, it’s practically a guarantee that the stream will work and complete successfully - important for a good user experience.

6

I’m seeing pretty much the same results, playback in Plex is a lot of hit n’ miss

Also while scanning this error is very common (nearly every file), https://github.com/ncw/rclone/issues/802

7

Hey @jkaberg

Would you be willing to try a similar test as the one I did, do 1-2 files with EncFS?

Here’s how to do it (assuming you know how to install EncFS), you can do this from the box Plex is on:

1. Mount ACD non-encrypted folder (assuming acd is acd: in rclone config):

Do not mount the secret or encrypted folder or this won’t work.

mkdir -p ~/acdmount
rclone mount acd: ~/acdmount --allow-other --max-read-ahead 1024 --no-modtime &

2. Setup an EncFS folder:

If this is the first time you run encfs, it will ask you for some questions and a password. Save it. It will also create an encfs6.xml file with your config, don’t lose this, assuming you want to continue your tests later on.

encfs ~/acdmount/.encfstest ~/acdmount/encfstest

3. Copy a media file (i.e.: movie) from ACD (rclone encrypted) to the newly created encfs folder:

mkdir -p ~/acdmount/encfstest/MyMovie
cp /path/to/acd/movie/MyMovie/* ~/acdmount/encfstest/MyMovie

4. Validate that it’s encrypted:

ls -l ~/acdmount/encfstest/MyMovie (you should see the actual filename)
ls -l ~/acdmount/.encfstest (you should see the encrypted movie folder)

5. Add the encfstest folder as a new library in Plex:

  1. Go into Plex via web interface (https://app.plex.tv/web)
  2. Add new library from the left menu bar
  3. Type: Movies
  4. Folder: ~/acdmount/encfstest
  5. Let it scan, it should find your movie.

6. Try Playback

From various Plex clients, connect to your Plex Server and try playing the new movie in the new library.

Really curious about others comparisons of EncFS vs Crypt.

8

I will test comparisons of EncFS vs Crypt later tonight ,

9

Wow..

So with Linode 4GB VPS ($20/month) I was able to hit 6 streams with EncFS encrypted media. This is Plex Media Server running on Linode (Ubuntu 16.04) with ACD mounted (with rclone) and encrypted with EncFS.

CPU usage was around 66%, all but 1 of the 6 streams were Direct Play, the rest were Transcodes:

Unbelievable! Hope rclone crypt is up to the challenge :slight_smile:

10

Here is a roundup reply to lots of posts!

Ok thanks for clarifying that. It looks like there is a problem with crypt indeed!

Seeking is quite special on an encrypted file system so if I was going to point the finger at anything I'd point it at that.

eba0a3633b5c2a68e2210d54e3a25fc67a9a0234 was the important fix for that, so any beta after or including http://beta.rclone.org/v1.33-75-geba0a36/ has that fix.

So we can get to the bottom of it I'd appreciate

  • logs from rclone mount -v when it is mis-behaving
  • a note of which version of rclone you are using (rclone version) - if you built it from source then which commit it was built from (git describe --tags in the rclone source directory)

1-4 sound like seeking problems. And 5 which dosn't seek and doesn't cause problems again points the finger at the seeking.

Does this mean that Plex is continuously seeking in the stream even when playback has started? That would explain why the performance is so bad - seeks are really slow. rclone doesn't attempt to buffer the file, relying on the OS to do that (wihch may be a mistake). Though I'm not sure encfs has a cache either (I looked at the man page).

Hmm, I haven't figured out issue 802 yet - maybe it is the key to everything :slight_smile:

Do other people see that in their logs?

11

I’ve found a data corruption problem with crypt+seek which I’ve now fixed. Have a go with http://beta.rclone.org/v1.33-81-g9d2dd2c/ (will be uploaded in 15-30 minutes) and tell me if that makes a difference.

12

Maybe streaming brain doing deep analysis at the start .

plex does prob the file for HI and low bitrate/bandwith so it can give a smooth streaming Experience

A cache drive option would be nice. start the movie and the file is just cached on the system when played and used for rewind fast f, and so on, have it delete when the file finish.

Was looking at adding the mount with fastcache or bcache .

Note:

Only error’s I am seeing in my Plex logs.
handle_stream_read error 335544539 short read
handle_stream_read error 2 End of file

I was having a few issues when I was on a cheap VPS, I had to limit the CPU use for rclone so plex would run properly.

I moved to better infrastructure, most of the issue’s have gone away,
yes, rclone+crypt does need some optimization, but until it is fine tuned, force streaming, better overall experience

@stokkes PM me and email address so I can send an invite , I would like to test the streams.

13

@ncw

Was this a big bug? I’m not home yet so can’ stress, but I’m streaming from 2 clients (iPhone on LTE Cellular) and one on Wifi (Plex Web client) and the streams are working perfectly.

I’m optimistic!

EDIT
In the past 15 minutes I’ve had the dreaded transport endpoint is not connected twice with the newest beta, requiring me to remount.

Unsure if some other bug was introduced.

14

It was a bug I introduced when optimising crypt seeking...

Good news!

I haven't seen that bug yet - can you post a log from rclone mount -v if possible? I suspect it means that rclone crashed in which case it probably gave a backtrace...

15

Hey @ncw

I’m back with my hopefully 2nd to last post for this topic.

I am quite confident the fix for the stream+corruption bug you did for v1.33-81-g9d2dd2c seems to have resolved pretty much all my problems.

I had 5 streams running from an rclone encrypted mount point, no dropped frames and I was actively monitoring the Plex log and the buffer stayed full on all the streams (so it was successfully pulling segments).

I will do another test this weekend with 6+ streams (the 5 I did were from my own house), but I am expecting it should be positive.

Caveat of course is that skipping ahead isn’t as fast as EncFS over an rclone non-encrypted mount. If you ever want the rclone log file from a stream from Plex (just to see how much Plex is accessing the file), I can PM you a snippet. But just to let you know that the rclone log fills very quickly when streaming with Plex.

Hoping to see some more tuning and improvements to the streaming and FUSE you were talking about soon!

Cheers,

16

Hello , sorry work ugg.

Yes, I am seeing the same thing . resume on a movie at 1 hour worked fine ,

that use to fail, nice work .

17

Thank you all for testing - it is important that the streaming is correct first, then we can make it go faster!

18

Yes, faster would be good :slight_smile:
But it’s more than acceptable and passed the WAF (Wife Acceptance Factor)