Summary: Hardware H.264 encoding via GStreamer nvv4l2h264enc works on the Jetson host but fails inside NVIDIA DeepStream L4T containers on JetPack 6.2.2. Investigation shows libnvtvmr.so shells out to lsmod | grep nvgpu during encoder initialization. When lsmod is absent (no kmod package in the image), the check fails and NVENC context creation aborts with a generic error.
Workaround: apt-get install -y kmod (or add kmod to the container image).
Product / scope
| Area | Details |
|---|---|
| Platform | Jetson Orin NX (Engineering Reference Developer Kit) |
| BSP | JetPack 6.2.2 (L4T R36.5.0, kernel 5.15.185-tegra, OOT kernel modules) |
| Container image | nvcr.io/nvidia/deepstream-l4t:7.1-samples-multiarch |
| Runtime | Docker with --runtime=nvidia --privileged |
| Component | GStreamer nvv4l2h264enc / NVMM (libnvtvmr.so) |
Symptom
Host: The following pipeline completes successfully:
gst-launch-1.0 videotestsrc ! nvvidconv ! 'video/x-raw(memory:NVMM), framerate=5/1' \
! nvv4l2h264enc ! fakesink
Expected log excerpt includes NvMMLiteOpen, ===== NvVideo: NVENC =====, and normal preroll.
Container (same command, same board, privileged NVIDIA runtime): Pipeline fails before NVENC initializes:
Opening in BLOCKING MODE
ENC_CTX(0xffff88008460) Error in initializing nvenc context
ERROR: from element /GstPipeline:pipeline0/nvv4l2h264enc:nvv4l2h264enc0:
Could not get/set settings from/on resource.
Device is in streaming mode
There is no NvMMLiteOpen / NvVideo: NVENC sequence; initialization stops earlier in the stack.
What was ruled out
The following were verified not to explain the failure:
-
Device nodes — Present in-container with host-matching major:minor, including
/dev/nvmap,/dev/host1x-fence,/dev/dri/renderD128,/dev/nvgpu/igpu0/*,/dev/v4l2-nvenc. -
NVIDIA user libraries — Bind-mounted from host via
nvidia-container-runtime; checksums matched forlibnvtvmr.so,libnvrm_host1x.so,libtegrav4l2.so,libv4l2_nvvideocodec.so,libnvmmlite_video.so, etc. -
DRM / host1x — Direct test opening
/dev/dri/renderD128and issuingDRM_IOCTL_TEGRA_CHANNEL_OPENfor NVENC class (0x21) succeeded on both host and container (context=1, version=35, capabilities=1). -
Permissions / cgroups — With
--privileged, seccomp, AppArmor, and cgroup device filtering are not blocking access;NVIDIA_VISIBLE_DEVICES=alldid not change behavior. -
sysfs / SoC identity — Relevant sysfs paths readable; SoC reports as Tegra234 (Orin),
soc_id=35.
Investigation
IOCTL comparison (host vs container)
strace -f -e trace=ioctl showed a large gap in DRM activity during encoder bring-up:
| Metric | Host | Container |
|---|---|---|
Total ioctl calls |
~2016 | ~1644 |
DRM_IOCTL calls |
~236 | ~36 |
DRM_IOCTL_TEGRA_CHANNEL_OPEN |
Called | Never called |
On the host, the encoder-related path opens a Tegra DRM channel to NVENC. In the container, the stack queries DRM_IOCTL_VERSION on render nodes but does not proceed to channel open; initialization aborts earlier.
Thread-level behavior (strace -f)
The thread responsible for encoder initialization was traced on both environments. On the host it proceeds from NvMMLiteOpen into DRM channel setup. In the container it spawned child processes via clone() + execve() and then failed immediately.
Child processes observed in-container:
-
lsmod— exit code 127 (command not found). -
grep nvgpu— read an empty pipe, non-zero exit.
Inferred call chain
The NVMM / video resource path appears to invoke a shell pipeline equivalent to:
lsmod | grep nvgpu
to confirm the nvgpu kernel module is loaded before NVENC context creation. When lsmod is missing (package kmod not installed in the image), the pipeline fails and the library treats the GPU stack as unavailable.
Approximate dependency chain (observed via behavior and tracing):
libnvv4l2.so
→ libv4l2_nvvideocodec.so
→ libnvtvmr.so (Tegra Video Resource Manager)
→ shell-out: lsmod | grep nvgpu
The failure mode is absence of the lsmod userspace binary, not missing kernel module or inaccessible devices (the module is loaded on the host kernel shared by the container).
Root cause
The container image does not install the kmod package, so /sbin/lsmod (or equivalent) is not present. NVIDIA’s closed-source libnvtvmr.so relies on that utility during encoder initialization instead of reading /proc/modules or another in-process API. A failed shell-out leads to aborted NVENC init and the generic GStreamer error above.
Why it shows up on JetPack 6.x containers: On older JetPack 5.x–era base images, transitive dependencies often pulled in kmod. The DeepStream L4T 7.1 image for JetPack 6.x uses a leaner base where kmod may be absent. The Jetson host filesystem always includes kmod from the Ubuntu/L4T rootfs, so the issue is container-specific.
Observability: stderr from the failed lsmod invocation may be easy to miss (e.g. mixed with plugin-scanner noise or suppressed). The runtime error (Error in initializing nvenc context) does not mention missing lsmod.
Resolution / workaround
Install kmod inside the container (or bake it into the image):
apt-get update && apt-get install -y kmod
Dockerfile example:
RUN apt-get update && apt-get install -y kmod && rm -rf /var/lib/apt/lists/*
After installation, the same GStreamer pipeline exhibits the same successful initialization sequence as on the host (NvMMLiteOpen, NvVideo: NVENC, preroll, etc.).
Recommendations (for NVIDIA and integrators)
-
Documentation: Call out
kmod(or guaranteed presence oflsmod) as a dependency for Jetson multimedia / NVENC in container images, or document thelsmod | grep nvgpucheck inlibnvtvmr.soif that is intentional. -
Images: Consider adding
kmodas an explicit dependency in official L4T / DeepStream container bases where NVMM/NVENC is supported. -
Robustness: Prefer checking
/proc/modulesor an ioctl/sysfs-based probe instead of shelling tolsmod, to avoid fragile container minimal images and silent failures. -
Diagnostics: If
gst-plugin-scanneror similar logs showlsmod: not found, treat it as a likely hard failure for later NVENC/NVMM use in that image.
Reproduction checklist (for verification)
-
Jetson Orin NX, JetPack 6.2.2 (L4T R36.5.0).
-
Pull
nvcr.io/nvidia/deepstream-l4t:7.1-samples-multiarch. -
Run container:
--runtime=nvidia --privileged(plus usual device/GPU env if required by your setup). -
Confirm
which lsmodis empty orlsmodreturns 127 before fix. -
Run the
gst-launch-1.0pipeline above; expect NVENC init failure. -
apt-get install -y kmod; rerun pipeline; expect success.