I’ve been checking out omxh264enc & nvv4l2h264enc for encoding video from a capture source (a Magewell Eco Capture Dual HDMI), and I’ve noticed that the GStreamer elements/codecs are behaving slightly differently when under heavier load, depending (my best guess) on scene complexity, amount of “stuff” going on in the video, input resolution/FPS, selected bandwidth/profiles, etc. I’m sharing my observations and got some followup questions.
My source material was an action scene from a blockbuster movie, somewhat simplistically played out from YouTube on one screen, and cloned via OS-level screen mirroring into the Eco Capture, where it was subsequently encoded & packaged to HLS on the NX, and played out on a third device - so that I had two screens to visually compare the output, and no other task running on the NX.
The basic pipelines can be recreated as follows:
gst-launch-1.0 v4l2src ! video/x-raw ... ! omxh264enc ... ! h264parse ! hlssink2 target-duration=...
gst-launch-1.0 v4l2src ! video/x-raw ... ! nvvidconv ! nvv4l2h264enc ... ! h264parse ! hlssink2 target-duration=...
The dots stand in for capsfilters, encoder/packager parameters, etc. I’ve mostly stuck with width=1920,height=1080,framerate=60/1 (or 30/1).
The target-duration was one of the parameters that also seems to have an impact (likely because hlssink2 seems to be communicating back to the encoder about when to emit key frames). In order to play back an HLS stream, I’ve used a basic HTTP server, started with “python3 -m http.server”. My playback device was an iPhone SE2 (Safari / AVPlayer).
Observations and questions:
- It seems that omxh264enc outputs variable frame rate in the encoded H264 stream (as reported by ffprobe, Apple’s HLS validator tools, etc), even when the input frame rate is fixed and constant. When reading the element documentation (gst-inspect-1.0 omxh264enc) I didn’t find any obvious parameters / hints about forcing the codec to output constant frame rate, or to select any other trade-offs. Is it even possible, or is vFPS the only option here?
- When using a very short (1s) segment length, it seems like nvv4l2h264enc is struggling to keep up, but the “side effects” are different depending on the source frame rate. All tests were done with baseline profile (profile=0) and slow preset (preset-level=4). With 1080p60, we’ve occasionally found the video to stutter. With 1080p30, there are visible encoding artefacts, characteristic to H264, but no stutter. We’ve observed all of the presets to be quite stable when we’d bump the segment length to the more conventional 6s. The question is, are there any tweaks we can apply here to push for lower latency when streaming with HLS, while maintaining acceptable video quality (eg no stutter, artifacts)? There are plenty of options available in the codec which I’m less familiar with (quant-*-frames, qp-range), and some that I’ve played with (curiously, bitrate) don’t seem to have the expected impact. I would love some guidance here :)