Kernel panic in nvmap_handle_remove with DeepStream + Argus on Xavier NX (JP 5.1.4)

Robotics & Edge Computing Jetson & Embedded Systems Jetson Xavier NX
,
1

Hi,

I’m seeing a reproducible kernel panic on a Jetson Xavier NX with a custom carrier board, running JetPack 5.1.4 (L4T R35.6.0).

Setup:

  • One DeepStream pipeline with nvarguscamerasrc + nvinfer + nvtracker
  • One additional GStreamer pipeline using nvarguscamerasrc

Issue:
After running for a while (anywhere from ~24 minutes to ~29 hours), the system panics. Example trace:

kernel BUG at /usr/src/kernel/nvidia/drivers/video/tegra/nvmap/nvmap_handle.c:106!
Internal error: Oops - BUG: 00000000f2000800 [#1] PREEMPT SMP
...
CPU: 1 PID: 176857 Comm: argus_thread Tainted: G           OE     5.10.216-tegra #2
...
nvmap_handle_remove+0xec/0x100 [nvmap]
_nvmap_handle_free+0x9c/0x4a0 [nvmap]
nvmap_handle_put+0x10c/0x1e8 [nvmap]
nvmap_dmabuf_release+0x118/0x150 [nvmap]

Full kernel log captured over the serial debug console during the panics:
270825cap.txt (3.3 MB)

Notes:

  • Periodically restarting the DeepStream pipeline avoids the panic.
  • Nothing unusual is seen in the logs prior to the panic, except soctherm: OC ALARM 0x00000001. The module is powered with a stable 5V supply, drawing at most ~18W.

Questions:

  1. Is this a known issue on Xavier NX / JetPack 5.1.4?
  2. Has it been fixed in newer releases such as JetPack 6.2.1 for Orin NX?
  3. Are there recommended workarounds for 5.1.4 beyond periodically restarting the pipelines?

Thanks!

3

Hi,
From description, it looks to be memory leak somewhere. Please help

  1. Share the full gstreamer pipeline
  2. Try nvarguscamerasrc ! fakesink. Would like to confirm whether only nvarguscamerasrc has the issue or not
4

Hi, thanks for the quick response.

Here is the full pipeline (equivalent to our Python code):

gst-launch-1.0 -e \
  nvarguscamerasrc sensor-id=0 \
    ! 'video/x-raw(memory:NVMM), width=3840, height=2160, framerate=10/1, format=NV12' \
    ! nvvidconv flip-method=2 \
    ! 'video/x-raw(memory:NVMM), width=1280, height=736, format=RGBA' \
    ! mux.sink_0 \
  nvstreammux name=mux width=1280 height=736 batch-size=1 live-source=true \
    ! nvinfer config-file-path=config.txt \
    ! nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so \
                ll-config-file=config_tracker.yml display-tracking-id=0 \
    ! tee name=t \
    ! queue max-size-buffers=1 leaky=2 silent=true \
    ! nvmsgconv frame-interval=1 msg2p-newapi=true payload-type=PAYLOAD_DEEPSTREAM_MINIMAL \
                multiple-payloads=1 config=msgconfig.txt \
    ! nvmsgbroker proto-lib=/opt/nvidia/deepstream/deepstream-6.3/lib/libnvds_mqtt_proto.so \
                  topic=detections conn-str="127.0.0.1;1883" \
  t. \
    ! queue max-size-buffers=10 leaky=2 silent=true \
    ! nvdsosd process-mode=1 display-text=0 \
    ! nvvidconv ! 'video/x-raw,width=1280,height=720' \
    ! shmsink socket-path=/tmp/video_out sync=false wait-for-connection=false shm-size=20000000

The second pipeline in our setup is using nvarguscamerasrc, and it can run on its own for days without any issues. Would it help if I try running two parallel nvarguscamerasrc ! fakesink pipelines to see if the issue can be reproduced that way?

5

Hi,
In Deepstream you should use nvvideoconvert plugin. Please replace nvvidconv plugin with nvvideoconvert and see if the issue is present.

6

Hi,

I replaced both nvvidconv elements with nvvideoconvert and reran the test. The behavior is unchanged, the system still eventually panics with the same BUG.
010925cap.txt (3.4 MB)

7

Hi,

gst-launch-1.0 -e \
  nvarguscamerasrc sensor-id=0 \
    ! 'video/x-raw(memory:NVMM), width=3840, height=2160, framerate=10/1, format=NV12' \
    ! ! nvvideoconvert \
    ! 'video/x-raw(memory:NVMM), width=1280, height=736, format=RGBA' \
    ! mux.sink_0 \
  nvstreammux name=mux width=1280 height=736 batch-size=1 live-source=true \
    ! nvinfer config-file-path=config.txt \
    ! nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so \
                ll-config-file=config_tracker.yml display-tracking-id=0 \
    ! tee name=t \
    ! queue max-size-buffers=1 leaky=2 silent=true \
    ! fakesink
  t. \
    ! queue max-size-buffers=10 leaky=2 silent=true \
    ! nvdsosd process-mode=1 display-text=0 \
    ! nvvideoconvert ! 'video/x-raw,width=1280,height=720' \
    ! fakesink

Please try the command and see if the issue is still present. Would like to break down the pipeline by eliminating nvmsgconv, nvmsgbroker, shmsink plugins.