Deepstream-app not running when more than 12 cams or 3 models

peteryoon · January 3, 2025, 8:45am

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 7.1
• JetPack Version (valid for Jetson only)
• TensorRT Version 10.6
• NVIDIA GPU Driver Version (valid for GPU only) 560.30.35
• Issue Type( questions, new requirements, bugs) Questions
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

I running NVIDIA-AI-IOT / deepstream_reference_apps version 7.1 modified deepstream_parallel_infer_app.cpp that body-pose related codes are removed.

Q1 app is not running

I have 3 difference yolox models.
If I run 1 models with 8 videos, it’s okay
If I run 2 models with 8 videos, it’s okay
But when I run 3 models with 9 videos or 2 models with 16 videos, its’ not okay…
Sometimes, 16vidoes with 2 models works…
But mostly it stuck at first 1 or 2 frame, shows few video frames and few green frames.
PERF shows 0 fps for all channels.

HW SPEC
cpu: Intel(R) Xeon(R) W-2245 CPU @ 3.90GH
gpu: nvidia rtx A4000

f21_config.yml.txt (3.6 KB)
f21_metamux.txt (1.2 KB)
deepstream_parallel_infer_app.cpp.txt (49.5 KB)

Q2 converting to cv::Mat

I tried to convert mat in this way. but It not working properly…

for (l_frame = batch_meta->frame_meta_list; l_frame != NULL; l_frame = l_frame->next) {
    NvDsFrameMeta *frame_meta = (NvDsFrameMeta *) (l_frame->data);
    NvDsMetaList *l_next = NULL;

    if (surface && surface->surfaceList[frame_meta->batch_id].dataPtr) {
      int width = surface->surfaceList[frame_meta->batch_id].width;
      int height = surface->surfaceList[frame_meta->batch_id].height;

      unsigned char *cpu_frame = (unsigned char *) malloc(width * height * 4);
      cudaMemcpy(cpu_frame, surface->surfaceList[frame_meta->batch_id].dataPtr,
                 width * height * 4, cudaMemcpyDeviceToHost);

      cv::Mat frame(height, width, CV_8UC4, cpu_frame);         
     free(cpu_frame);
    }
  }

Also I tried to use get_converted_mat() from /opt/nvidia/deepstream/deepstream-7.1/sources/gst-plugins/gstdsexample.cpp.
However, I have no clue to find correspoding variabledsexample in parallel-infer…

Your help would be greatly appreciated… I sincerely seek your kind assistance.
Thank you

fanzh · January 6, 2025, 9:24am

sorry for the late replay! DS7.1 requires R535. please make sure the component version meets the requirement in this table.

peteryoon · January 6, 2025, 10:33am

Thank you for the reply!

I will install matched version and try again!

And how can I extract frame data from nvbuffer?

my code is below

  NvBufSurface *surface = (NvBufSurface *)in_map_info.data;
  for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
       l_frame = l_frame->next)
  {
    NvDsFrameMeta *frame_meta = (NvDsFrameMeta *)(l_frame->data);
    NvBufSurfaceParams *params = &surface->surfaceList[frame_meta->batch_id];
      uint32_t width = params->width;
      uint32_t height = params->height;
      uint32_t pitch = params->pitch;
      void *data = params->mappedAddr.addr;

      // pitch= 2048, height=1080, width=1920
      // data type is NVBUF_COLOR_FORMAT_NV12_709
      // in GPU Mem, color type YUV, buf size  Y (WH) UV (WH/2)

      size_t bufferSize = (width * height) + (width * height / 2); // Y (width x height) + UV (width x height / 2)
      uint8_t* dstCPUBuffer = new uint8_t[bufferSize];

      // use cudamemcpy2d since pitch != width
      cudaError_t err = cudaMemcpy2D(dstCPUBuffer, width, data, pitch, width, height * 3 / 2, cudaMemcpyDeviceToHost);


      cv::Mat frame(params->height, params->width, CV_8UC4, hostBuffer);
      cv::Mat bgr;
      cv::cvtColor(frame, bgr, cv::COLOR_RGBA2BGR);

}

and I got error

CUDA memcpy failed: invalid argument

Is there any correct way to get frame data on cpu?
I need cpu cv::Mat…

Thank you in advance :)

fanzh · January 7, 2025, 2:01pm

please refer to this sample for how to access nv12 by opencv.

peteryoon · January 8, 2025, 2:02am

I followed the code snippet and got this error.

Code

static GstPadProbeReturn
body_pose_gie_src_pad_buffer_probe(GstPad *pad, GstPadProbeInfo *info,
                          gpointer u_data)
{
  GstBuffer *buf = (GstBuffer *)info->data;
  NvDsMetaList *l_frame = NULL;
  NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta(buf);

  GstMapInfo in_map_info;
  if (!gst_buffer_map(buf, &in_map_info, GST_MAP_READ)) {
    g_print("Error: Failed to map gst buffer\n");
    return GST_PAD_PROBE_OK;
  }
  NvBufSurface *surface = (NvBufSurface *)in_map_info.data;
   
  for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
       l_frame = l_frame->next)
  {
    NvDsFrameMeta *frame_meta = (NvDsFrameMeta *)(l_frame->data);
    NvBufSurfaceParams *params = &surface->surfaceList[frame_meta->batch_id];
    
      if ( NvBufSurfaceMap (surface, -1, -1, NVBUF_MAP_READ) != 0 ){
        g_print("NvBufSurfaceMap failed..\n");
      }
}

Result

nvbufsurface: mapping of memory type (2) not supported
NvBufSurfaceMap failed..

I failed from the beginning with NvBufSurfaceMap.

According to the SDK API, NvBufSurfaceMap do

Maps hardware batched buffers to the HOST or CPU address space.

Valid for NVBUF_MEM_CUDA_UNIFIED type memory for dGPU and NVBUF_MEM_SURFACE_ARRAY and NVBUF_MEM_HANDLE type memory for Jetson.

How can I convert memory type of surface ?

peteryoon · January 8, 2025, 3:06am

Based on the example code from sample opt\nvidia\deepstream\deepstream-7.1\sources\gst-plugins\gst-dsexample\gstdsexample_optimized.cpp, I fixed my code as below.

static GstPadProbeReturn
body_pose_gie_src_pad_buffer_probe(GstPad *pad, GstPadProbeInfo *info,
                          gpointer u_data)
{
  gchar *msg = NULL;
  GstBuffer *buf = (GstBuffer *)info->data;
  NvDsMetaList *l_frame = NULL;
  NvDsMetaList *l_obj = NULL;
  NvDsMetaList *l_user = NULL;
  NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta(buf);

  GstMapInfo in_map_info;
  if (!gst_buffer_map(buf, &in_map_info, GST_MAP_READ)) {
    g_print("Error: Failed to map gst buffer\n");
    return GST_PAD_PROBE_OK;
  }
  NvBufSurface *surface = (NvBufSurface *)in_map_info.data;
   

    NvBufSurfaceCreateParams create_params = { 0 };
    create_params.gpuId = 0;
    create_params.width = 1920;
    create_params.height = 1080;
    create_params.size = 0;
    create_params.colorFormat = NVBUF_COLOR_FORMAT_NV12_709;
    create_params.layout = NVBUF_LAYOUT_PITCH;
    create_params.memType = NVBUF_MEM_CUDA_UNIFIED;

  if (NvBufSurfaceCreate (&surface, 16, &create_params) != 0) {
          GST_ERROR ("Error: Could not allocate internal buffer for dsexample");
        }
   // NvBufSurfaceMap is valid for NVBUF_MEM_CUDA_UNIFIED (3) type memory for dGPU
        if ( NvBufSurfaceMap (surface, -1, -1, NVBUF_MAP_READ) != 0 ){
        g_print("NvBufSurfaceMap failed..\n");
      }

  NvBufSurfaceSyncForCpu (surface, 0, 0);

  for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
       l_frame = l_frame->next)
  {
    NvDsFrameMeta *frame_meta = (NvDsFrameMeta *)(l_frame->data);
    NvBufSurfaceParams *params = &surface->surfaceList[frame_meta->batch_id];
  
    //g_print("default %d pined %d device %d, unified %d, this %d\n",  NVBUF_MEM_DEFAULT ,  NVBUF_MEM_CUDA_PINNED, \
            NVBUF_MEM_CUDA_DEVICE, NVBUF_MEM_CUDA_UNIFIED, surface->memType);
     
     // default 0 pined 1 device 2, unified 3, this 2 
     
    if (frame_meta->source_id < 4)
    {
      guint height = surface->surfaceList[frame_meta->batch_id].height;
      guint width = surface->surfaceList[frame_meta->batch_id].width;
      cv::Mat nv12_mat = cv::Mat(height * 3 / 2, width, CV_8UC1, surface->surfaceList[frame_meta->batch_id].mappedAddr.addr[0], surface->surfaceList[frame_meta->batch_id].pitch);
      cv::Mat rgba_mat;
      cv::cvtColor(nv12_mat, rgba_mat, cv::COLOR_YUV2RGBA_NV12);
      std::string filename = "/workspace/tmpjpg/src" + std::to_string(frame_meta->source_id) + "_frame" + std::to_string(frame_meta->frame_num) + ".jpg";
      cv::imwrite(filename, rgba_mat);

    }

  }
  
  NvBufSurfaceUnMap(surface, 0, 0);

  gst_buffer_unmap(buf, &in_map_info);

  return GST_PAD_PROBE_OK;
}

and code works but image files are not save right…

its just 33kb and show green screen only…

Could you please help me ?

fanzh · January 8, 2025, 6:14am

seems you create a new surface. please use the
NvBufSurface *surface = (NvBufSurface *)in_map_info.data; and make sure “sur->colorFormat” is nv12.

peteryoon · January 8, 2025, 8:17am

The color current color type is NVBUF_COLOR_FORMAT_NV12_709.

I checked color type by printing each variable.

static GstPadProbeReturn
body_pose_gie_src_pad_buffer_probe(GstPad *pad, GstPadProbeInfo *info,
                          gpointer u_data)
{
  GstBuffer *buf = (GstBuffer *)info->data;
  NvDsMetaList *l_frame = NULL;
  NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta(buf);

  GstMapInfo in_map_info;
  if (!gst_buffer_map(buf, &in_map_info, GST_MAP_READ)) {
    g_print("Error: Failed to map gst buffer\n");
    return GST_PAD_PROBE_OK;
  }
  NvBufSurface *surface = (NvBufSurface *)in_map_info.data;
  NvBufSurfaceMap(surface, -1, -1, NVBUF_MAP_READ);
  NvBufSurfaceSyncForCpu (surface, 0, 0);

  for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
       l_frame = l_frame->next)
  {
    NvDsFrameMeta *frame_meta = (NvDsFrameMeta *)(l_frame->data);
    NvBufSurfaceParams *params = &surface->surfaceList[frame_meta->batch_id];
  
    if (frame_meta->source_id == 0)
    {
      g_print("Current color format is  %d \n" , params->colorFormat);
      g_print("RGBA %d, NV12 %d NVBUF_COLOR_FORMAT_NV12_709 %d\n", NVBUF_COLOR_FORMAT_RGBA, NVBUF_COLOR_FORMAT_NV12, NVBUF_COLOR_FORMAT_NV12_709);
    }
  }
  
  NvBufSurfaceUnMap(surface, 0, 0);
  gst_buffer_unmap(buf, &in_map_info);
  return GST_PAD_PROBE_OK;
}

Current color format is  33 
RGBA 19, NV12 6 NVBUF_COLOR_FORMAT_NV12_709 33

If I use NvBufSurfaceMap I got memory type error.

nvbufsurface: mapping of memory type (2) not supported

According to the SDK API I mentiond above, NvBufSurfaceMap only work with
NVBUF_MEM_CUDA_UNIFIED which has value 3 while current mem type is 2.

Thus I tried to convert memory type by following example /opt/nvidia/deepstream/deepstream-7.1/sources/gst-plugins/gst-dsexample/gstdsexample_optimized.cpp.

`NVBUF_MEM_CUDA_DEVICE` (2) --> `NVBUF_MEM_CUDA_UNIFIED` (3) by new surface params.

As a results, it didn’t work…

Bottom line,

color type is NV12 ( NVBUF_COLOR_FORMAT_NV12_709 33)
NvBufSurfaceMap failed due to memory type

What can I do …?

fanzh · January 8, 2025, 8:26am

please add a nvideoconvert before nvdsexample to output `NVBUF_MEM_CUDA_UNIFIED type, here is a sample:

gst-launch-1.0 uridecodebin uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! nvvideoconvert nvbuf-memory-type=3 ! fakesink

peteryoon · January 8, 2025, 8:35am

I modified every nvbuf-memory-type in yml and it worked!!!

Thank you so much!!

Topic		Replies	Views
Deepstream DeepStream SDK gstreamer , jetson , deepstream	14	426	July 9, 2024
OpenCV Mat to NvBufSurface (to use in NvBufSurfTransform) DeepStream SDK	16	3504	October 12, 2021
Issues extracting images from surface object DeepStream SDK	4	445	October 19, 2023
Opencv mat clone osd frame coredump DeepStream SDK	17	464	March 15, 2024
Package the tensorRT inference results into GstBuffer and push them to the downstream DeepStream SDK	7	1296	October 17, 2022
Too Small Captured Full Frame to RAM in dsexample DeepStream SDK	8	868	October 12, 2022
Save frames extracted from Deepstream pipeline in C++ OpenCV DeepStream SDK	8	839	October 30, 2023
NvBufSurfaceSyncForDevice return -1 and how to replace frame in pipeline? DeepStream SDK	21	2047	October 12, 2021
How to switch between different video sources and zoom in to full screen on Sink DeepStream SDK deepstream	13	78	November 6, 2024
From nvBufSurface to gpu::Mat or cuda stream DeepStream SDK cuda , deepstream	6	6	January 10, 2025

Deepstream-app not running when more than 12 cams or 3 models

Q1 app is not running

Q2 converting to cv::Mat

Related topics