Mediainfo/ffprobe timings

It's been a while since I've retested things and was curious if some folks had other results.

I was doing some non cached testing with various read chunk sizes and curious on the results other folks got:

  541  rclone mount dcrypt-tv: /home/felix/test -vvv --vfs-read-chunk-size 16M
  542  rclone mount dcrypt-tv: /home/felix/test -vvv --vfs-read-chunk-size 256M
  543  rclone mount dcrypt-tv: /home/felix/test -vvv --vfs-read-chunk-size 128M
  544  rclone mount dcrypt-tv: /home/felix/test -vvv --vfs-read-chunk-size 1M

128/256 gave me consistently 2-3 mediainfo results in my timing and 1M/16M almost doubled the time to 5-6 seconds per file. The file does matter so as long as you test with the same file over and over, that would be fine as your results might not be exactly the same as mine.

Cache mode gives me faster results as expected if the file is already there.

1M - real 0m8.783s
8M - real 0m4.580s
16M - real 0m3.640s
32M - real 0m2.984s
64M - real 0m3.529s
128M - real 0m3.804s
256M - real 0m3.183s

32M generally seemed like a good suite spot for me with Dropbox so I was trying to see some other results.

I was just using time mediainfo for the same test file and umounting after each test and retesting a few times to see what a general average was.

if you describe in more detail the exact steps and exact commands to replicate,
then i can do some testing.
keep in mind, not a linux expert.

I'm just running a mount and then cd'ing into a directory and timing the mediainfo on a media item.

So I basically have one terminal for a mount command and another I paste in something like:

cd /home/felix/test/TV
cd "TESTSHOW"
time mediainfo 'TESTFILE.mkv'
cd
fusermount -uz /home/felix/test

and just repeat by changing the mount command to my settings. I try to run through it a few times on each setting to get an idea of a baseline (not perfect) but a good general idea of the performance.

I guess for a better approach, maybe test using a standard jellyfish test file?

https://jell.yfish.us/media/jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv

Might be a good one to use as it's not huge.

Hmm, I take that back as that's just video and no audio tracks so a real bad "real world" test. I'd just find a decent TV show or Movie and test.

does it matter if file is .mkv, mp4, subtitles, etc?

best would be to pick a file, upload it dropbox and share it,
we can download it and test with that

I wouldn't think so as just trying to get an idea of other folks doing a similar test. I'd pick just a normal file in your library and test using that and see how the results match up with different tweaks.

Plex generally does a mediainfo/ffprobe when it analyzes a file and I was just seeing if I'd make any other tuning changes on my mounts to help that process.

1M means much less 'wasted' download but it's much slower. The default of 128M still seems good but you get a little bit of waste at the end as you tend to read more and the file closes out. More problematic in the Google area as they have some algorithm it seems on number of opens/how much data is read per file and some aggregation of all of that which without any formal documentation is just awful to figure out as the results people post are all over the board.

When I redid my library, I butchered some stuff and had to redo a lot and without a download quota on Dropbox, they were easy to fix and re-analyze and get things right at the end. I can't imagine how many quota days I would have blown on Google, but that's a different problem all together :slight_smile:

did some quick testing, for 16M, 256M and 1M, in that order, tested two runs,
i made sure to kill the mount each time.
rclone mount wasabi01:zork /home/user01/rclone/mountpoints/zork --read-only --allow-other -v --vfs-read-chunk-size=xxx

here is the summary of the two runs, orderd by chunk size

--vfs-read-chunk-size=1M
real    0m1.891s
real    0m2.161s

--vfs-read-chunk-size=16M
real    0m1.900s
real    0m1.665s

--vfs-read-chunk-size=256M
real    0m2.656s
real    0m3.842s

using a virtual machine.

rclone version
rclone v1.57.0
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.10.60.1-microsoft-standard-WSL2 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.2
- go/linking: static
- go/tags: none

Roughly, what's the size of the file?

I've noticed the following does seem to play into the speeds:

  • size
  • number of audio trackers
  • chapter info

The more items in the container, it definitely takes longer. Movies as a whole seem to take 2-3 times as long as a TV show from my observations.

General
Unique ID                                : 34972992319628066984384588793422566019 (0x1A4F8DA612A246D4EA7C9DAB2A2C5283)
Complete name                            : file.mkv
Format                                   : Matroska
Format version                           : Version 4
File size                                : 1.55 GiB
Duration                                 : 1 h 40 min
Overall bit rate                         : 2 218 kb/s
Movie name                               : 
Encoded date                             : UTC 2020-07-05 16:45:07
Writing application                      : mkvmerge v48.0.0 ('Fortress Around Your Heart') 64-bit
Writing library                          : libebml v1.4.0 + libmatroska v1.6.0

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4
Format settings                          : CABAC / 4 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 4 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 1 h 40 min
Bit rate                                 : 2 119 kb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 30.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.034
Stream size                              : 1.48 GiB (96%)
Writing library                          : x264 core 155 r2917 0a84d98
Encoding settings                        : cabac=1 / ref=1 / deblock=1:0:0 / analyse=0x3:0x113 / me=hex / subme=2 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=0 / me_range=16 / chroma_me=1 / trellis=0 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=0 / threads=12 / lookahead_threads=4 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=2 / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=1 / weightb=1 / open_gop=0 / weightp=1 / keyint=90 / keyint_min=46 / scenecut=0 / intra_refresh=0 / rc_lookahead=10 / rc=crf / mbtree=1 / crf=23.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / vbv_maxrate=5136 / vbv_bufsize=7200 / crf_max=0.0 / nal_hrd=none / filler=0 / ip_ratio=1.40 / aq=1:1.00
Language                                 : English
Default                                  : Yes
Forced                                   : No

Audio
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : A_AAC-2
Duration                                 : 1 h 40 min
Bit rate                                 : 96.0 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 32.0 kHz
Frame rate                               : 31.250 FPS (1024 SPF)
Compression mode                         : Lossy
Delay relative to video                  : -32 ms
Stream size                              : 68.8 MiB (4%)
Language                                 : English
Default                                  : Yes
Forced                                   : No

Text
ID                                       : 3
Format                                   : UTF-8
Codec ID                                 : S_TEXT/UTF8
Codec ID/Info                            : UTF-8 Plain Text
Duration                                 : 1 h 39 min
Bit rate                                 : 104 b/s
Count of elements                        : 2037
Stream size                              : 76.2 KiB (0%)
Language                                 : English
Default                                  : No
Forced                                   : No

Menu
00:00:00.000                             : :Chapter 1
00:03:44.224                             : :Chapter 2
00:07:03.381                             : :Chapter 3
00:10:51.860                             : :Chapter 4
00:14:49.347                             : :Chapter 5
00:19:16.405                             : :Chapter 6
00:24:40.980                             : :Chapter 7
00:27:32.943                             : :Chapter 8
00:30:52.351                             : :Chapter 9
00:34:14.469                             : :Chapter 10
00:38:21.591                             : :Chapter 11
00:42:30.673                             : :Chapter 12
00:48:03.881                             : :Chapter 13
00:52:49.375                             : :Chapter 14
00:55:47.970                             : :Chapter 15
00:57:46.505                             : :Chapter 16
00:59:57.511                             : :Chapter 17
01:04:23.401                             : :Chapter 18
01:07:27.627                             : :Chapter 19
01:10:20.800                             : :Chapter 20
01:13:42.293                             : :Chapter 21
01:19:59.962                             : :Chapter 22
01:22:00.374                             : :Chapter 23
01:25:33.420                             : :Chapter 24
01:29:35.787                             : :Chapter 25
01:32:17.741                             : :Chapter 26
01:34:47.682                             : :Chapter 27
01:38:14.347                             : :Chapter 28

It's a bit tough without getting super into tracing to see the exact.

As a rough estimate, you can run something like nethogs, check the rclone process and do a test.

There's a little noise in there as it does capture the cd into the directory and a poll, but overall, that's an example of how much data it pulls for a file. That's 4.6GB file I'm checking and while I could only snag a quick screenshot, it download a healthy bit of data.

so back to my point, need to compare the same file

In my experience (I use MediaInfo embedded into Windows Explorer all day long), it takes significantly longer if a file has a lot of subtitles. This is usually the case with shows on Netflix and Amazon.

Yep, probably getting a little off topic as I was curious to see different chunk sizes and their influence on a 'normal' type media file that resides in one's library.

S3 is a ton faster as I expected, albeit the smaller calls seem better. It's a way more expensive service compared to Google/Dropbox so I'd imagine it should be faster.

1 Like

so this is off topic of an off topic,
imho, our ideal meeting place :wink:

I guess two off topics make it on topic so I stand corrected.

Simple math! :nerd_face:

(Need MOAR characters...)