Extra memory usage in GPU0 when testing deepstream-segmentation-app in GPU1

Accelerated Computing Intelligent Video Analytics DeepStream SDK
#1

• Hardware Platform (GPU ) RTX3080 ×2
• DeepStream Version 6.0.1
• TensorRT Version 8.0.1.6
**• NVIDIA GPU Driver Version ** 470.82.01

• Issue Type( bugs)
We found some extra memory usage in GPU0(default) when using segmentation model in multi gpu application.

• How to reproduce the issue ?
/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-segmentation-test
image
we set all the gpu-id as 1

lQLPJxaJmP7Rf01xzQIDsC102pu-3VGTAuKBm4mAYQA_515_113

./deepstream-segmentation-app dstest_segmentation_config_industrial.txt /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_industrial.jpg

lQDPJxaJmP7Rf6M9zQJssPZb1aWs9B2xAuKBm6dAbgA_620_61

#3

I am checking

#4

  1. after checking， deepstream_segmentation_app1.c can reproduce this issue, deepstream_segmentation_app2.c can’t, the difference is nvinfer 's usage.
    deepstream_segmentation_app1.c (13.9 KB)
    deepstream_segmentation_app2.c (13.9 KB)

  2. check result with gst-launch-1.0 command:
    this command can’t reproduce.
    gst-launch-1.0 filesrc location=…/…/…/…/samples/streams/sample_720p.mjpeg ! jpegparse ! nvv4l2decoder gpu-id=1 ! mux.sink_0 nvstreammux name=mux batch-size=1 width=1280 height=720 gpu-id=1 ! fakesink
    this command can reproduce.
    gst-launch-1.0 filesrc location=…/…/…/…/samples/streams/sample_720p.mjpeg ! jpegparse ! nvv4l2decoder gpu-id=1 ! mux.sink_0 nvstreammux name=mux batch-size=1 width=1280 height=720 gpu-id=1 ! nvinfer config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.txt gpu-id=1 ! fakesink

will continue to check.

#5

it is a reproducible bug, nvinfer plugin is opensource, please use this workaround:

  1. modify /opt/nvidia/deepstream/deepstream-6.1/sources/gst-plugins/gst-nvinfer/gstnvinfer.cpp, like this:
    static gpointer gst_nvinfer_input_queue_loop (gpointer data)
    {
    GstNvInfer *nvinfer = (GstNvInfer *) data;
    cudaSetDevice (nvinfer->gpu_id);
    …
    }
  2. compile, then copy libnvdsgst_infer.so to /opt/nvidia/deepstream/deepstream/lib/, backup old libnvdsgst_infer.so first.