Memory usage spikes and GStreamer pipeline fails when nvdsosd process-mode=0

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

Hello everyone, this is my setup

• Hardware Platform (Jetson / GPU)
Jetson Orin Nx 16 GB
• DeepStream Version
DeepStream 7.0.0-1
• JetPack Version (valid for Jetson only)
JetPack 6.0
• TensorRT Version
TensorRT 8.6.2

When running the following pipeline

URI="file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4"
gst-launch-1.0 -e \
  nvstreammux name=streammux batch-size=1 sync-inputs=0 width=1920 height=1080 ! \
    queue name=preinfer_queue max-size-buffers=3 leaky=downstream ! \
    nvinfer name=primary_gie unique-id=1 config-file-path=config_infer_primary.txt batch-size=1 ! \
    nvtracker name=tracker ll-config-file=config_tracker_IOU.yml \
      ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so \
      tracking-id-reset-mode=1 ! \
    nvinfer name=secondary_gie_0 unique-id=2 config-file-path=config_infer_secondary_vehicletypes.txt \
      process-mode=2 infer-on-gie-id=1 infer-on-class-ids=0 batch-size=1 ! \
    nvinfer name=secondary_gie_1 unique-id=3 config-file-path=config_infer_secondary_vehiclemake.txt \
      process-mode=2 infer-on-gie-id=1 infer-on-class-ids=0 batch-size=1 ! \
    queue name=prerecord_queue max-size-buffers=3 leaky=downstream ! \
    nvdsosd process-mode=0 display-clock=1 display-text=1 display-bbox=1 display-mask=1 ! \
    nvvideoconvert compute-hw=0 ! \
    'video/x-raw(memory:NVMM),format=I420' ! \
    nvv4l2h264enc insert-sps-pps=true iframeinterval=30 control-rate=0 preset-level=1 maxperf-enable=true bitrate=4000000 ! \
    h264parse config-interval=5 ! \
    fakesink sync=true async=false \
  nvurisrcbin uri="$URI" file-loop=true disable-passthrough=true gpu-id=0 cudadec-memtype=0 ! \
    nvvideoconvert compute-hw=0 disable-passthrough=true ! 'video/x-raw(memory:NVMM),format=RGBA' ! perf ! \
    streammux.sink_0

Find attached the txt files needed for the nvinfer elements at the end of this message.

I am experiencing that the memory tracked by top starts growing until it reaches around 1100% (sometimes less, sometimes more) and the pipeline fails with the following error messages attached in the next file

deepstream_pipeline_error_message.txt (5.6 KB)

I noticed that this is related to the process-mode property value set to the default (CPU_MODE) nvdsosdelement. When I change the process-mode to 1, GPU_MODE, the memory percentage value reported by top is constant.

I could also reproduce the issue in an older JetPack (5.1.2, DeepStream 6.3) version on an Xavier NX.

Questions for the NVIDIA team:

  1. Is this behavior expected?
  2. Changing the process-mode to be done by the GPU is the only way to avoid the issue from happening?
  3. Is there any patch for the nvdsosd element that avoids this behavior?

NOTE: I can not change the JetPack and DeepStream versions.

Related txt files for the nvinfer elements:

config_infer_secondary_vehicletypes.txt (2.8 KB)
config_infer_primary.txt (3.5 KB)
config_infer_secondary_vehiclemake.txt (2.8 KB)
labels_sgie_make.txt (134 Bytes)
labels_sgie_types.txt (39 Bytes)
labels_pgie.txt (29 Bytes)

Thank you in advance
Regards!
Eduardo Salazar
Embedded SW Engineer at RidgeRun

Contact us: support@ridgerun.com
Developers wiki: https://developer.ridgerun.com/
Website: www.ridgerun.com

3

Thanks for the sharing! I am checking. Noticing the title is inconsistent with the content, is there any mistake?

4

Hello,

I changed the title.

Additionally, if the nvdsosd properties are set in the following way

display-clock=0 display-text=0 display-mask=0 display-bbox=1

MEM still growing; when changing display-bbox=0, it became constant.

I hope this information helps narrow down the issue.

Thanks!

5

Hello @fanzh

Is there any update on this?

Regards!

6

Sorry for the late reply! the osd issue can be reproduced on my end with the following simplified cmd.

gst-launch-1.0  filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvideoconvert ! streammux.sink_0 nvstreammux name=streammux batch-size=1 sync-inputs=0 width=1920 height=1080 ! \
    nvinfer name=primary_gie unique-id=1 config-file-path=dstest2_pgie_config.txt batch-size=1 ! \
    nvdsosd process-mode=0 display-text=0 display-bbox=1   !  \
    fakesink sync=true async=false

nvdsosd has been openosurced from DS8.0. the path is under /opt/nvidia/deepstream/deepstream/sources/libs/nvll_osd/. First you can build the code after copying it( maybe there will some compilation error ). then rebuild the code with the following workaround, which is based on DS9.0.
nvll_osd-new.cpp (14.7 KB)

7

Hello @fanzh

Thank you for the new source code provided.

However, our DeepStream version is 7.0. Could you please provide a binary or a source code version compatible with our DeepStream version?

If not, how can this be tested for older deepstream systems? Is this backwards compatible?

Thank you in advance.
Best regards!

9

Since DS7.0 nvdsosd is closedsource, we will only provide a binary with the fix.

10

libnvds_osd.zip (810.0 KB) . after unizp, please use the folllowing cmd to repalce.

cp /opt/nvidia/deepstream/deepstream/lib/libnvds_osd.so /opt/nvidia/deepstream/deepstream/lib/libnvds_osd.so_bk && cp libnvds_osd.so /opt/nvidia/deepstream/deepstream/lib/libnvds_osd.so