Fast memory increasing when using USE_NEW_NVSTREAMMUX and nvds_obj_enc_process

1002469771 · May 23, 2023, 4:15am

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
GPU
• DeepStream Version
6.1
• NVIDIA GPU Driver Version (valid for GPU only)
510.47.03
• Issue Type( questions, new requirements, bugs)
bugs
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

I ran deepstream-app in nvcr.io/nvidia/deepstream:6.1-devel container with some code changed.

./deepstream-app -c /opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app/source2_1080p_dec_infer-resnet_demux_int8.txt

Here’s some code I added to encode image data.

static GstPadProbeReturn
gie_primary_processing_done_buf_prob (GstPad * pad, GstPadProbeInfo * info,
    gpointer u_data)
{
  GstBuffer *buf = (GstBuffer *) info->data;
  GstMapInfo inmap = GST_MAP_INFO_INIT;
  if (!gst_buffer_map (buf, &inmap, GST_MAP_READ)) {
      return GST_PAD_PROBE_OK;
  }
  NvBufSurface *ip_surf = (NvBufSurface *) inmap.data;
  gst_buffer_unmap (buf, &inmap);

  AppCtx *appCtx = (AppCtx *) u_data;
  NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta (buf);
  if (!batch_meta) {
    NVGSTDS_WARN_MSG_V ("Batch meta not found for buffer %p", buf);
    return GST_PAD_PROBE_OK;
  }

  //write_kitti_output (appCtx, batch_meta);

  //encode jpg here
  int imageCount = 0;
  for (NvDsMetaList * l_frame = batch_meta->frame_meta_list; l_frame != NULL;
      l_frame = l_frame->next) {
    NvDsFrameMeta *frame_meta = l_frame->data;
    for (NvDsMetaList * l_obj = frame_meta->obj_meta_list; l_obj != NULL;
        l_obj = l_obj->next) {
      NvDsObjectMeta *obj = (NvDsObjectMeta *) l_obj->data;
      float left = obj->rect_params.left;
      float top = obj->rect_params.top;
      float right = left + obj->rect_params.width;
      float bottom = top + obj->rect_params.height;
      // Here confidence stores detection confidence, since dump gie output
      // is before tracker plugin
      float confidence = obj->confidence;
      // fprintf (bbox_params_dump_file,
      //     "%s 0.0 0 0.0 %f %f %f %f 0.0 0.0 0.0 0.0 0.0 0.0 0.0 %f\n",
      //     obj->obj_label, left, top, right, bottom, confidence);

      NvDsObjEncUsrArgs userData = { 0 };
      /* To be set by user */
      userData.saveImg = false;
      userData.attachUsrMeta = true;
      /* Set if Image scaling Required */
      userData.scaleImg = FALSE;
      userData.scaledWidth = 0;
      userData.scaledHeight = 0;
      /* Preset */
      userData.objNum = imageCount++;
      /* Quality */
      userData.quality = 30;
      /*Main Function Call */
      if(!nvds_obj_enc_process (appCtx->jpg_handle, &userData, ip_surf, obj, frame_meta)) {
        g_print("jpg encode failed!\n");    
      }
    }
  }
  nvds_obj_enc_finish (appCtx->jpg_handle);

  return GST_PAD_PROBE_OK;
}

And here’s part of my config file:

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=4
#uri=file://../../streams/sample_1080p_h264.mp4
uri=rtsp://
num-sources=1
#drop-frame-interval=2
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[source1]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=4
#uri=file://../../streams/sample_1080p_h264.mp4
uri=rtsp://
num-sources=1
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[source2]
enable=2
type=4
uri=rtsp://
num-sources=1
gpu-id=0
cudadec-memtype=0

[sink0]
#source0 output as filesink
enable=0
...

[sink1]
#source1 output as filesink
enable=0
...

[osd]
enable=0
...

[streammux]
#gpu-id=0
##Boolean property to inform muxer that sources are live
#live-source=0
batch-size=1
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
#batched-push-timeout=33000
## Set muxer output width and height
#width=1920
#height=1080
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
#nvbuf-memory-type=0
## If set to TRUE, system timestamp will be attached as ntp timestamp
## If set to FALSE, ntp timestamp from rtspsrc, if available, will be attached
attach-sys-ts-as-ntp=1

# config-file property is mandatory for any gie section.
# Other properties are optional and if set will override the properties set in
# the infer config file.
[primary-gie]
enable=1
gpu-id=0
model-engine-file=../../models/Primary_Detector/resnet10.caffemodel_b1gpu0_int8.engine
batch-size=32
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=0
gie-unique-id=1
nvbuf-memory-type=0
config-file=config_infer_primary.txt

[tests]
file-loop=0

Source1 is a rtsp stream with a resolution of 2560x1440;
Source2 is a rtsp stream with a resolution of 1920x1080;
Source3 is a rtsp stream with a resolution of 2048x1536

It ran well when I used old nvstreammumx, however, if I launched it with this:

export USE_NEW_NVSTREAMMUX="yes"

the memory would increase very fast, like from 3G to 16G in seconds, and would keep going up.

yuweiw · May 23, 2023, 6:31am

Hi @1002469771 ， will this issue occur without your changes for the source code and how do you confirm memory growth?

1002469771 · May 23, 2023, 7:23am

How could I encode image data if I don’t change the source code? Is there anything wrong about my changing? And I used top command to monitor memory and cpu change, it grew very fast.

1002469771 · May 23, 2023, 7:30am

It ran steadily when I commented nvds_obj_enc_process, and it ran well if the rtsp streams are the same resolution, so I think there maybe something wrong with different resulotion streams and that’s why I posted my stream resolutions.

yuweiw · May 23, 2023, 7:54am

This is for debug purposes to narrowing down the scope of the problem.

just use the 3 rtsp sources without code changes and new streammux, it’s works well.
use the 3 rtsp sources, code changes and nvstreammux, it’s works well
use the 3 rtsp sources and new nvstreammux, comment on the nvds_obj_enc_process, it’s works well.

Are these all right?

Also, some of your modifications are missing from the code you attached, like how to create and destroy the jpg_handle.

1002469771 · May 23, 2023, 7:57am

just use the 3 rtsp sources without code changes and new streammux, it’s works well.(Yes)
use the 3 rtsp sources, code changes and nvstreammux, it’s works well.(No)
use the 3 rtsp sources and new nvstreammux, comment on the nvds_obj_enc_process, it’s works well.(Yes)

1002469771 · May 23, 2023, 8:10am

Here’s how I create jpg_handle:
in deepstream_app_main.c, mian function

  for (i = 0; i < num_instances; i++) {
    appCtx[i] = g_malloc0 (sizeof (AppCtx));
    appCtx[i]->person_class_id = -1;
    appCtx[i]->car_class_id = -1;
    appCtx[i]->index = i;
    appCtx[i]->active_source_index = -1;
    appCtx[i]->jpg_handle= nvds_obj_enc_create_context ();
    if (show_bbox_text) {
      appCtx[i]->show_bbox_text = TRUE;
    }
    ...
  }

and defined here:

struct _AppCtx
{
  ...
  guint ota_watch_desc;
  NvDsObjEncCtxHandle jpg_handle;
};

no destruction yet.

yuweiw · May 23, 2023, 8:37am

I mean old nvstreammux. Is it works well?
Also, could you add one more sink in your config file cause you have three sources?

1002469771 · May 23, 2023, 8:41am

Yes, old nvstreammux works well.
I didn’t use any sink, I set enable=0 in the config file, but I’ll try to add one or more sinks to see if there’s any difference.

1002469771 · May 23, 2023, 8:54am

I’ve tried add 1 and 3 sinks, no use.Though it ran steadily(3G memory) for the first 1 minutes when there were 3 sinks, then it jumped to a high memory(16G) all of a sudden.

1002469771 · May 24, 2023, 3:02am

Hi there! I did some more tests.I changed sample_1080p_h264.mp4 video’s resolution with ffmpeg like this:

./ffmpeg -i /workshop/sample_1080p_h264.mp4 -filter:v scale=2560:1440 -c:a copy /workshop/sample_2560_1440.mp4
./ffmpeg -i /workshop/sample_1080p_h264.mp4 -filter:v scale=2048:1536 -c:a copy /workshop/sample_2048_1536.mp4

and I replaced the rtsp sources with these 3 offline video files like this:

[source0]
enable=1
type=2
uri=file:///sample_1080p_h264.mp4
...
[source1]
enable=1
type=2
uri=file:///sample_2560_1440.mp4
...

[source2]
enable=1
type=2
uri=file:///sample_2048_1536.mp4
...

then I ran deepstream-app, it occurred again.
Could you try this on your environment to reproduce it? It should be easier now.

yuweiw · May 24, 2023, 7:17am

I use your code and config file, mp4 file, new nvstreammux. But I use the deepstream 6.2. It works ok without memory growth. Could you use the deepstream 6.2 to run it?

1002469771 · May 27, 2023, 5:34am

I tried to use deepstream 6.2,the problem was gone. And I’ve migrated my own program on deepstream 6.2, it works well now, so I guess it has been fixed in 6.2. Thanks for your help.

system · June 10, 2023, 5:35am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.