I have the same (almost same) setup as animosity022.
I also do not know of a remux that has a 80Mb/sec rate.
Braveheart is at 51…
Saving Private Ryan 52
iron man 56…
all play fine…which is one that is 80+?
JP3 is 73, let me test
no issues Direct Stream/Play
The bitrate can peak up to 128mbps according to blu-ray standards. What shows on plex is only the average bitrate. If you use Plex Media Player and press control shift d, it’ll show you the bitrate and how much has cached. Good for seeing the exact speed of what something is going at.
What chunk size do you use?
I’m just the default 10M chunk size:
Constant bitrate on my 50GB 4k movie:
Video
ID : 1
Format : HEVC
Format/Info : High Efficiency Video Coding
Commercial name : HDR10
Format profile : Main 10@L5.1@High
Codec ID : V_MPEGH/ISO/HEVC
Duration : 2 h 14 min
Bit rate : 44.3 Mb/s
Width : 3 840 pixels
Height : 2 160 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 23.976 (24000/1001) FPS
Color space : YUV
Chroma subsampling : 4:2:0 (Type 2)
Bit depth : 10 bits
Bits/(Pixel*Frame) : 0.223
Stream size : 41.7 GiB (89%)
Default : Yes
Forced : No
Color range : Limited
Color primaries : BT.2020
Transfer characteristics : PQ
Matrix coefficients : BT.2020 non-constant
Mastering display color primaries : Display P3
Mastering display luminance : min: 0.0001 cd/m2, max: 1000 cd/m2They just seemed right for my use case. The first implementation I tested had --vfs-read-chunk-size 1G hard coded, which just worked fine too.
Right, filling the buffer would cause some unnecessary connections and after the buffer is full, the chunk size would probably be --buffer-size or more, depending on the consumer speed.
The --buffer-size as a default value would probably be fine for line speeds less than 1 Gbit/s.
Assuming a 1 Gbit/s connection and the default --buffer-size 16M, the first chunk would be read in 150ms. This could cause 4 requests in the first second (realistically probably 2 - 3).
This seams a bit excessive, so twice the --buffer-size is probably a better value.
All these numbers primarily consider Google Drive as the remote. What about other services? Are there known limits for req/s, req/day and download bytes/day on other remotes that we should take into account?
This feature, seems to me (unless I have misunderstood something very badly), to be the best available option for seeding torrents.
I tried a google drive -> cache -> crypt -> mount setup for seeding torrents, and I observed huge amounts of downstream traffic from the cloud. However this traffic didn’t correspond at all with the actual torrent upload traffic to peers. (Download from the cloud was in the order of 150-300 mbit/s, while I was seeding at most 30-40 kbytes/sec).
Now despite my efforts, I couldn’t get these numbers to change. Although I didn’t spend a lot of hours testing it, I gave up, coming to the conclusion that using a cache is the very antithesis of seeding a torrent.
The moment you upload a piece to a torrent peer, the chances that you will have to send the exact same data to another peer in the swarm drop dramatically.
In addition, torrent pieces are requested by peers semi-randomly, on a rarest first approach. This, of course, doesn’t correspond to a cache that assumes that the next data you’ll need, is what comes after what you just requested.
So, I’m thinking of testing this new feature for seeding torrents, possibly some time in the following weekend, unless of course I have totally misunderstood what this feature does, in which case feel free to enlighten me.
That is interesting. 1G seems quite large…
Yes. Which isn’t that far from your recommended settings in this thread…
I don’t know about specific limits of other remotes, but I like the general principle of not requesting more than we need to start with.
I guess my concern is to get the initial chunk size as small as possible, but maybe I’m worrying too much as effectively the values are infinite at the moment so making the defaults 128M/2G rather than 32M/whatever isn’t worth worrying about.
I played around a bit more and seems to work well with some smaller values:
/usr/bin/rclone mount gcrypt: /gmedia \
--allow-other \
--dir-cache-time 672h \
--vfs-cache-max-age 675h \
--vfs-read-chunk-size 64M \
--vfs-read-chunk-size-limit 1G \
--buffer-size 32M \
--syslog \
--umask 002 \
--rc \
--log-level INFO
I tested writing directly to the mount as well and that worked well as too. The updates went quickly.
Works better than your CACHE mount?
Even with 1G chunk size, this was at least 10x reduce in requested bytes for most files and a up to 100x reduction for the largest files, so it seemed right to me 
Right, every number will be a big save compared to the current defaults. A initial value of 64M or 128M and a limit of 2G should be good numbers for speeds up to 1 Gbit/s.
I’m going to let it run a few days and really see. Subjectively, start time seems better in a few test cases. Listing out files and such works the same. Once the cache is primed up, it operates just as fast as my cached instance before without the overhead of running the cache.
It feels like a simpler solution for me with one part removed so I’m going to test out a few days and see.
It is definitely worth a try. I don’t know what block sizes are used for torrents, but I would suggest to try --vfs-read-chunk-size 16M or even smaller.
This is probably caused by a combination of cache fetches and --buffer-size. The default value of 16M to much for seeding. Something between 1M and 4M should reduce the downstream traffic.
Are you sure it’s constant bitrate, it’s showing constant frame rate. If this is a remux then it could vary a lot based on the amount of colors in the scene and movement.
I’m assuming this is not variable, but let me know if not:
Bit rate : 44.3 Mb/s
Width : 3 840 pixels
Height : 2 160 pixels
Display aspect ratio : 16:9
Frame rate mode : ConstantI’m betting it’s variable bitrate mode.
You could test it using ffmpeg, found from https://video.stackexchange.com/questions/17182/how-to-measure-bitrate-of-video-over-time since I’m not sure if HEVC has average and peak bitrate metadata, since mediainfo and ffmpeg don’t display peak.
You’d run ffmpeg -i gmedia/hevcfile.mkv -an -c:v copy -segment_time 0.00001 -f segment i%0d.mp4 but make sure you run it inside of a folder outside your gdrive mount and in its own folder because it will create many .mp4 files with each frame from the video, I cancelled it a few seconds after running. Then you can do ffprobe i0.mp4 or any of the other files to see the bitrate, for me it was variable while mediainfo only shows the one bitrate field. If you play the file in Plex Media Player and hit control shift D, itll open an info window with the current bitrate on the right.
Download from the cloud was in the order of 150-300 mbit/s, while I was seeding at most 30-40 kbytes/sec).
I’m using rclone cache+mount with the following parameters to seed torrents and avoid the issue you mentioned above.
/usr/local/bin/rclone mount crypt: ~/mnt/crypt --no-checksum --allow-other --dir-cache-time=36h --cache-info-age=36h --max-read-ahead=512K -v --buffer-size=0k --cache-chunk-no-memory --cache-read-retries=20 --cache-chunk-size=512K --cache-total-chunk-size=50M --cache-db-purge --cache-dir=/dev/shm/rclone --cache-chunk-path=/dev/shm/rclone --cache-db-path=/dev/shm/rclone --cache-chunk-clean-interval=1m --cache-rps=4 --tpslimit=3 --low-level-retries=20 --retries=20 --contimeout=2m --log-file=/home/mydir/bin/rclone.log &
(The tpslimit can be set higher, till 9 for example. I set it low because I also seed with rclone cache+mount from home and have a script to automatically rclone copy (upload) to GD at the same time.)
It does allow rtorrent to seed torrents from mount, but the problem is that maximum upload speed is typically only 1x kb/s. Very rarely would the upload speed reach 100 kb/s. Using vfs-read-chunk-size sounds like a good idea because cache is indeed not useful for seeding. I wonder if using vfs-read-chunk-size may speed up the upload! Please do keep us posted on the result of your experiment.
I think Lucy has a high Bitrate
Lucy:
Complete name : Lucy.2014.German.Dubbed.DTS.DL.2160p.UHD.BluRay.HDR.HEVC.Remux-Lame4K.mkv
Format : Matroska
Format version : Version 4 / Version 2
File size : 53.8 GiB
Duration : 1 h 29 min
Overall bit rate mode : Variable
Overall bit rate : 86.0 Mb/s
Movie name : Lucy 4K
Encoded date : UTC 2017-10-17 10:47:39
Writing application : mkvmerge v9.7.1 ('Pandemonium') 64bit
Writing library : libebml v1.3.4 + libmatroska v1.4.5
John Wick 2:
Complete name : John.Wick.Kapitel.2.2017.German.Dubbed.DTSHD.DL.2160p.UHD.BluRay.HDR.HEVC.Remux-NIMA4K.mkv
Format : Matroska
Format version : Version 4 / Version 2
File size : 72.7 GiB
Duration : 2 h 2 min
Overall bit rate mode : Variable
Overall bit rate : 85.1 Mb/s
Encoded date : UTC 2017-10-05 12:23:28
Writing application : mkvmerge v15.0.0 ('Duel with the Devil') 64-bit
Writing library : libebml v1.3.5 + libmatroska v1.4.5FYI… Those bitrates shown in the mediainfo are all average and they are not constant. It depends a lot on the scene you are watching.
For instance, if you are watching a dark scene there is less information (=less bitrate) that needs to be transmitted.
Yeah I know this. But it is a good indicator for stream testing.
In most cases I test 5-10 min.
A good method to watch the current bitrate is to start Kodi debug player mode (Strg + Shift + O)
Tried John Wick 2, in Plex DIRECT stream…
damn…does not play well…
Tried it in VLC, worse.
Not sure if its a DATA transfer issue (need to up the cache size to “download” before playing) or what…
Might make a quick test mount with these settings to try it WITHOUT the cache
Edit: Using the Console on PLEX, the READ AHEAD size does NOT get lower than its actual read ahead. So I have a feeling its my computer cannot handle the bitrate. Let me try it on the Android TV.
EDIT2: It was my Computer unable to play the file. PLayed JOhn Wick 2 to the TV and the frame rate is perfect. I am getting some pausing here and there. Unsure if its internet related at this point or file or cache size related. I do not get that much onscreen information from the android TV/PLEX. I can not say it is X amount of seconds or that it pauses for X amount of seconds. But it is clearly unwatchable.
I will do further testing to see if internet related or cache related
I have a feeling it is INTERNET related as Truilla is saying I need 180mbit connection for this file to work properly and of course WIRED on the SONY XBR is only 100mbit. So will try and switch to WIFI 5G tomorrow and test again. Just got through 3 minutes of non interruption. Wish the TV had true 1gbit LAN. Might look for an external ethernet that works. Because if it is the built in issue, I will be switching.
EDIT3: couldn’t wait. Connected to 5g wifi (which theoretically has a higher bandwidth. Did a speed test through built in web browser in android TV. and pauses similarly. (more actually)
ACTIVITY Streams: 1 stream (1 direct play) | Bandwidth: 185.2 Mbps (LAN: 185.2 Mbps)
So basically the CACHE is fine, its the stupid piece of shit network connection on my TV =(…Unless I can somehow up the “read ahead”
@greenxeyezz if you want to test the consistency of the cache, I’d run your rclone mount command with all of its variables and leave out a log file and just monitor it in the terminal window with -vvv which I think is one up --log-level=DEBUG which is only -vv according to the docs.
Then run rsync --progress fileongdrive tofolderonlocal and monitor the speed. For me, using Animosity022’s settings, I got slow speeds, not enough to sustain UHD remuxes effectively. But since I am on a 10gbps+ box, I upped the workers to 8 and chunk size from 10M to 50M and ran the rsync command again. Supposedly the cache workers only works when using Plex (if a server and credentials are added to your config file) and for all other tasks it will resort to using 1 worker, I’m guessing for rsync here. I got 100 MB/s now vs only getting 6-8 MB/s before on the lower chunk size.
It will be easier to run the rclone mount with the trace logging since for me at least, file copying isn’t always consistent and maybe some packet loss or gdrive connection/server failure can happen and it sometimes hangs on grabbing some of the chunks which causes 50 MB/s to go down to 12 MB/s for at least 15-30 seconds. It may be doing this with Plex reading and causes the stutters to happen.
Last thing if you run it manually, add --cache-db-purge (or rm -r .cache/rclone) to your temporary mount so the cache is cleaned each time you change your rclone settings, without cleaning the cache, you’ll get errors. Also too if you try playing the same file again, it likely won’t be a fair test since it could be reading from the cache at that point, depending on your cache size. Rclone pulls in the chunks and then reads them if they still exist before grabbing from gdrive again, if the stutter comes from Gdrive instead of the Plex server.