1000s of dmabuf open file handles reported by lsof (re-posted)

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

This post takes over 1000s of dmabuf open file handles reported by lsof.

This issue reproduces in L4T 32.4.3 as well.

Here’s the test command line we used. The nvinfer configuration is derived from the sample test application of DeepSteeam.

$ gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream-5.0/samples/streams/sample_1080p_h264.mp4 ! \
        decodebin ! m.sink_0 nvstreammux name=m batch-size=1 width=1280 \
        height=720 ! nvinfer config-file-path=dstest1_pgie_config.txt ! \
        nvvideoconvert ! nvdsosd ! perf ! nvegltransform ! nveglglessink sync=false

Then, we aggregated the result of lsof while gst-launch is running as above.

About 7000 dmabuf handles are generated on the whole of the system.

$ lsof | grep dmabuf | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/120/gvfs
  Output information may be incomplete.
7075

Also the following is the breakdown of the result. The head number of each line shows the total occurrences of dmabuf handles for each command name lsof identifies. Dmabuf handles of compiz, gmain, gdbus and dconf\x20 reside regardless of the gst-launch execution. Others occur while gst-launch is running.

$ lsof | grep dmabuf | uniq -c --check-chars=9
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/120/gvfs
  Output information may be incomplete.
110 compiz     6110                 nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
110 gmain      6110  6121           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
110 gdbus      6110  6128           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
110 dconf\x20  6110  6130           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
630 gst-launc 11580                 nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
315 cuda-EvtH 11580 11582           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
1575 gst-launc 11580 11583           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
315 typefind: 11580 11600           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
315 gmain     11580 11601           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
315 qtdemux0: 11580 11602           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
630 multiqueu 11580 11603           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
315 NVMDecBuf 11580 11605           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
315 NVMDecDis 11580 11606           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
315 NVMDecFrm 11580 11607           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
315 NVMDecVPR 11580 11608           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
315 V4L2_DecT 11580 11609           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
315 gst-launc 11580 11610           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
315 m:src     11580 11611           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
315 nvv4l2dec 11580 11612           nvidia  mem       REG               0,11                 9649 anon_inode:dmabuf (stat: No such file or directory)
4

Hi,
We have tested with deepstream-app with loop=1 and did not see any FD leak.
Total number of open FDs is not an issue, however if total FDs are increasing between start & stop of pipeline then this is a memory leak.
You are running single pipeline and it is possible to have so many open FDs directly proportional to number of sources connected in the pipeline.

Do you trigger out of memory killer in running your usecase? Running deepstream-app in loop test looks fine. If you do hit out of memory, please share steps so that we can replicate the issue.

5

Hi,

If we repeat state changing of the GStreame pipeline between PLAYING and NULL, we can see a fd leak. This is a general operation of GStreamer. I modified deepstream_test1_app to restart after reaching to EOS by state changing to NULL once and retuning to PLAYING as bellow and monitored the number of dmabuf handles while it is running.

diff --git a/deepstream_test1_app.c b/deepstream_test1_app.c
index 28a3c3b..c8a6d47 100644
--- a/deepstream_test1_app.c
+++ b/deepstream_test1_app.c
@@ -44,6 +44,7 @@ gint frame_number = 0;
 gchar pgie_classes_str[4][32] = { "Vehicle", "TwoWheeler", "Person",
   "Roadsign"
 };
+GstElement *pipeline = NULL;
 
 /* osd_sink_pad_buffer_probe  will extract metadata received on OSD sink pad
  * and update params for drawing rectangle, object information etc. */
@@ -122,7 +123,9 @@ bus_call (GstBus * bus, GstMessage * msg, gpointer data)
   switch (GST_MESSAGE_TYPE (msg)) {
     case GST_MESSAGE_EOS:
       g_print ("End of stream\n");
-      g_main_loop_quit (loop);
+      gst_element_set_state (pipeline, GST_STATE_NULL);
+      gst_element_set_state (pipeline, GST_STATE_PLAYING);
+      //g_main_loop_quit (loop);
       break;
     case GST_MESSAGE_ERROR:{
       gchar *debug;
@@ -147,7 +150,7 @@ int
 main (int argc, char *argv[])
 {
   GMainLoop *loop = NULL;
-  GstElement *pipeline = NULL, *source = NULL, *h264parser = NULL,
+  GstElement *source = NULL, *h264parser = NULL,
       *decoder = NULL, *streammux = NULL, *sink = NULL, *pgie = NULL, *nvvidconv 
= NULL,
       *nvosd = NULL;
 #ifdef PLATFORM_TEGRA

I retrieved a raw H.264 stream from sample_1080p_h264.mp4 with MP4Box and feed it to deepstream-test1-app I modified for repeating PLAYING and NULL.

./deepstream-test1-app sample_1080p_h264_track1.h264

Then, I monitored the number of dmabuf handles. They progressively increased according to the result of this tial.

1st playback time

$ lsof | grep <deepstream-test1-app PID> | grep dmabuf | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/120/gvfs
      Output information may be incomplete.
5100

5th playback time

$ lsof | grep <deepstream-test1-app PID> | grep dmabuf | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/120/gvfs
      Output information may be incomplete.
5645

10th playback time

$ lsof | grep <deepstream-test1-app PID> | grep dmabuf | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/120/gvfs
      Output information may be incomplete.
5900

We observed “Too many open files” before out of memory killer occurs and out of memory could also occur if we increase the maximum number of FDs for a process to avoid the FD exhaustion.

As you said, this waste of FDs also leads to memory exhaustion. Dmabuf handles are usually associated with physical resources for the acceleration with image processors, so buffer sizes in the MByte order should be allocated for a dmabuf handle to have raw image data depending on video resolution. In that case, we will face difficulties in coexistence with other software or integration into a certain product etc…
Are those quite a lot of dmabuf handles actually needed all?

6

Hi,
We apply the patch and check socket FDs by executing:

$ sudo netstat -wxnep | grep _PID_ | wc -l

Don’t see the number increasing in 30-min test. So in your test, after applying the patch to deepstream-test1, how long does it take to hit Too many open files? We would like to set up a long run for a try.

7

Hi,

Dmabuf handles are different from socket FDs. They are not seen with netstat. Please check the dmabuf FD leak with lsof just like I showed.

The following is the command line.

$ lsof | grep <deepstream-test1-app PID> | grep dmabuf | wc -l

When “Too many files open” occurs, other proprietary plugins using FDs are used simultaneously. Only running deepstream-test1-app might take quite a lot of time to reproduce “Too many files open”. Sorry for mixing up our specific case with this simple unit test. However it’s sure that dmabuf handles progressively increase for every repeating. Please see that using lsof.

8

Hi,
We have seen issue in switch between NULL → PLAY state with short interval. In the case the decoder is not able to dequeue buffers quick enough and triggers leak. Please make sure there is at least 100ms between NULL → PLAY → NULL and check if it helps.

If the issue is still present, we will try to set up a long run with deepstream-test1. See if we can hit Too many files open. If you have a more efficient way to replicate the issue, please kindly share it.

9

Hi,

The interval you suggested seems ineffective.

I found an efficient way to reproduce “too many open files”. deepstream-app works multiple pipelines, so a lot more FDs are regularly used. I modified deepstream-app for repeating the playback and did a long duration test using it. Then, “too many open files” reproduced within 5 hours on Jetson TX2.

At first, you need to add the following modification to deepstream-app and rebuild it in order to repeat the playback.

diff --git a/deepstream_app.c b/deepstream_app.c
index 37a2f70..87a506c 100644
--- a/deepstream_app.c
+++ b/deepstream_app.c
@@ -203,7 +203,7 @@ bus_callback (GstBus * bus, GstMessage * message, gpointer 
data)
        */
       NVGSTDS_INFO_MSG_V ("Received EOS. Exiting ...\n");
       appCtx->quit = TRUE;
-      return FALSE;
+      return TRUE;
       break;
     }
     default:
diff --git a/deepstream_app_main.c b/deepstream_app_main.c
index 640b820..1fe125c 100644
--- a/deepstream_app_main.c
+++ b/deepstream_app_main.c
@@ -287,9 +287,25 @@ event_thread_func (gpointer arg)
   }
 
   if (i == num_instances) {
-    quit = TRUE;
-    g_main_loop_quit (main_loop);
-    return FALSE;
+    for (i = 0; i < num_instances; i++) {
+      if (gst_element_set_state (appCtx[i]->pipeline.pipeline,
+              GST_STATE_NULL) == GST_STATE_CHANGE_FAILURE) {
+        g_print ("\ncan't set pipeline to playing state.\n");
+        return FALSE;
+      }
+    }
+
+    g_usleep(100000);
+
+    for (i = 0; i < num_instances; i++) {
+      appCtx[i]->quit = FALSE;
+      if (gst_element_set_state (appCtx[i]->pipeline.pipeline,
+              GST_STATE_PLAYING) == GST_STATE_CHANGE_FAILURE) {
+        g_print ("\ncan't set pipeline to playing state.\n");
+        return FALSE;
+      }
+    }
+    return TRUE;
   }
   // Check for keyboard input
   if (!kbhit ()) {

Then, run the following command line with the existing configuration for deepstream-app. I used source8_1080p_dec_infer-resnet_tracker_tiled_display_fp16_nano.txt.

$ ./deepstream-app -c source8_1080p_dec_infer-resnet_tracker_tiled_display_fp16_nano.txt

After about 5 hours, you can see “nvdc: open: Too many open files” errors after many “OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error”. Please try this procedure.

gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_mot_klt.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
gstnvtracker: Past frame output is OFF
INFO: Loading Model Engine from File
ERROR: Deserialize engine failed because file path: /opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/../../models/Primary_Detector_Nano/resnet10.caffemodel_b8_gpu0_fp16.engine open error
3:44:44.081508093 31483   0x558fac4890 WARN                 nvinfer gstnvinfer.cpp:672:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1690> [UID = 1]: deserialize engine from file :/opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/../../models/Primary_Detector_Nano/resnet10.caffemodel_b8_gpu0_fp16.engine failed
3:44:44.081594365 31483   0x558fac4890 WARN                 nvinfer gstnvinfer.cpp:672:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1799> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/../../models/Primary_Detector_Nano/resnet10.caffemodel_b8_gpu0_fp16.engine failed, try rebuild
3:44:44.081641213 31483   0x558fac4890 INFO                 nvinfer gstnvinfer.cpp:675:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1715> [UID = 1]: Trying to create engine from model files
INFO: Parsing prototxt a binaryproto Caffe model from files
INFO: [TRT]: Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
INFO: [TRT]: Detected 1 inputs and 2 output network tensors.
ERROR: Serialize engine failed because of file path: /opt/nvidia/deepstream/deepstream-5.0/samples/models/Primary_Detector_Nano/resnet10.caffemodel_b8_gpu0_fp16.engine opened error
3:45:06.037381982 31483   0x558fac4890 WARN                 nvinfer gstnvinfer.cpp:672:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1743> [UID = 1]: failed to serialize cude engine to file: /opt/nvidia/deepstream/deepstream-5.0/samples/models/Primary_Detector_Nano/resnet10.caffemodel_b8_gpu0_fp16.engine
INFO: [Implicit Engine Info]: layers num: 3
0   INPUT  kFLOAT input_1         3x272x480       
1   OUTPUT kFLOAT conv2d_bbox     16x17x30        
2   OUTPUT kFLOAT conv2d_cov/Sigmoid 4x17x30         

3:45:06.053655440 31483   0x558fac4890 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/config_infer_primary_nano.txt sucessfully
Opening in BLOCKING MODE 
**PERF:  29.23 (19.92)	29.23 (19.92)	29.42 (19.92)	29.24 (19.92)	29.23 (19.92)	29.42 (19.92)	29.42 (19.92)	29.23 (19.92)	
** INFO: <bus_callback:181>: Pipeline ready

NvMMLiteOpen : Block : BlockType = 261 
Opening in BLOCKING MODE 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
Opening in BLOCKING MODE 
NvMMLiteBlockCreate : Block : BlockType = 261 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
NvMMLiteOpen : Block : BlockType = 261 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
Opening in BLOCKING MODE 
NvMMLiteBlockCreate : Block : BlockType = 261 
NvMMLiteOpen : Block : BlockType = 261 
NvMMLiteBlockCreate : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
NvMMLiteBlockCreate : Block : BlockType = 261 
NvMMLiteBlockCreate : Block : BlockType = 261 
** INFO: <bus_callback:167>: Pipeline running

KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552 
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552 
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552 
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552 
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552 
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552
        .
        .
        .
        .
nvdc: open: Too many open files
nvdc: failed to open '/dev/tegra_dc_ctrl'.
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552 
nvdc: open: Too many open files
nvdc: failed to open '/dev/tegra_dc_ctrl'.
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552 
nvdc: open: Too many open files
nvdc: failed to open '/dev/tegra_dc_ctrl'.
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552
10

Hi,
From the log, the error print is from nvoverlaysink. Have you tried the case with nvegltransform ! nveglglessink? If you have tried the case, please share the infromation for our reference.

And from the patch, you don’t terminate the pipeline but switching to NULL and then back to PLAY state. Is this the same as your usecase? We have the property in config file:

[tests]
file-loop=0

But it looks different from your usecase. Would like to confirm this.

11

Hi,

Have you tried the case with nvegltransform ! nveglglessink ?

No, I haven’t, but I’ll try the case with nvegltransform ! nveglglessink.

And from the patch, you don’t terminate the pipeline but switching to NULL and then back to PLAY state. Is this the same as your usecase?

This is very close to the usecase we hope to successfully work. Our usecase continuously receives an input video stream and pass it to DeepStream, then the state of the pipeline can be changed to PLAYING or NULL at any time depending on user operation. My patch uses EOS as a trigger for changing the state for a simple test, but actually changing the state may occur at any time during the playback.

But it looks different from your usecase. Would like to confirm this.

Exactly. file-loop seems to affect only a source plugin. Does this repeat reading an input file in the source plugin without changing the state of the pipeline?
We would like to test the repeat of changing the state between NULL and PLAYING for the whole of the pipeline.

12

Hi,

I tried the case with nvegltransform ! nveglglessink, then a memory allocation error from ddkvic was seen and finally the process crashed instead of “Too many files open”.

I just changed the sink type from NV_DS_SINK_RENDER_OVERLAY to NV_DS_SINK_RENDER_EGL in the config file for this test.

diff --git a/source8_1080p_dec_infer-resnet_tracker_tiled_display_fp16_nano.txt 
b/source8_1080p_dec_infer-resnet_tracker_tiled_display_fp16_nano.txt
index aa84271..9921c07 100644
--- a/source8_1080p_dec_infer-resnet_tracker_tiled_display_fp16_nano.txt
+++ b/source8_1080p_dec_infer-resnet_tracker_tiled_display_fp16_nano.txt
@@ -55,7 +55,7 @@ cudadec-memtype=0
 [sink0]
 enable=1
 #Type - 1=FakeSink 2=EglSink 3=File
-type=5
+type=2
 sync=1
 source-id=0
 gpu-id=0

Here’s the log of the last two session. In the second form the last, an internal data stream error occurred in tiled_display_queue at the end. Then, “Couldn’t create ddkvic Session: Cannot allocate memory” finally showed and crashed.

Using winsys: x11 
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_mot_klt.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
gstnvtracker: Past frame output is OFF
INFO: Loading Model Engine from File
ERROR: Deserialize engine failed because file path: /opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/../../models/Primary_Detector_Nano/resnet10.caffemodel_b8_gpu0_fp16.engine open error
10:11:11.781946751 11710   0x558b48d520 WARN                 nvinfer gstnvinfer.cpp:672:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1690> [UID = 1]: deserialize engine from file :/opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/../../models/Primary_Detector_Nano/resnet10.caffemodel_b8_gpu0_fp16.engine failed
10:11:11.782033694 11710   0x558b48d520 WARN                 nvinfer gstnvinfer.cpp:672:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1799> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/../../models/Primary_Detector_Nano/resnet10.caffemodel_b8_gpu0_fp16.engine failed, try rebuild
10:11:11.782080190 11710   0x558b48d520 INFO                 nvinfer gstnvinfer.cpp:675:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1715> [UID = 1]: Trying to create engine from model files
INFO: Parsing prototxt a binaryproto Caffe model from files
INFO: [TRT]: Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
INFO: [TRT]: Detected 1 inputs and 2 output network tensors.
ERROR: Serialize engine failed because of file path: /opt/nvidia/deepstream/deepstream-5.0/samples/models/Primary_Detector_Nano/resnet10.caffemodel_b8_gpu0_fp16.engine opened error
10:11:34.435200188 11710   0x558b48d520 WARN                 nvinfer gstnvinfer.cpp:672:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1743> [UID = 1]: failed to serialize cude engine to file: /opt/nvidia/deepstream/deepstream-5.0/samples/models/Primary_Detector_Nano/resnet10.caffemodel_b8_gpu0_fp16.engine
INFO: [Implicit Engine Info]: layers num: 3
0   INPUT  kFLOAT input_1         3x272x480       
1   OUTPUT kFLOAT conv2d_bbox     16x17x30        
2   OUTPUT kFLOAT conv2d_cov/Sigmoid 4x17x30         

10:11:34.451428778 11710   0x558b48d520 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/config_infer_primary_nano.txt sucessfully
**PERF:  29.07 (19.44)	29.20 (19.44)	29.10 (19.44)	29.21 (19.44)	30.07 (19.44)	29.07 (19.44)	28.50 (19.44)	28.50 (19.44)	
** INFO: <bus_callback:181>: Pipeline ready

Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NvMMLiteOpen : Block : BlockType = 261 
NvMMLiteOpen : Block : BlockType = 261 
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
NvMMLiteBlockCreate : Block : BlockType = 261 
NvMMLiteBlockCreate : Block : BlockType = 261 
Opening in BLOCKING MODE 
NvMMLiteBlockCreate : Block : BlockType = 261 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
NvMMLiteBlockCreate : Block : BlockType = 261 
NvMMLiteBlockCreate : Block : BlockType = 261 
** INFO: <bus_callback:167>: Pipeline running

KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
error in Blit. Error: 4 
**PERF:  3.98 (19.44)	4.04 (19.44)	4.06 (19.44)	4.00 (19.44)	3.98 (19.44)	3.98 (19.44)	3.99 (19.44)	3.99 (19.44)	
**PERF:  29.12 (19.45)	29.12 (19.45)	29.12 (19.45)	29.12 (19.45)	29.12 (19.45)	29.12 (19.45)	29.12 (19.45)	29.12 (19.45)	
**PERF:  30.18 (19.45)	30.18 (19.45)	30.18 (19.45)	30.18 (19.45)	30.18 (19.45)	30.18 (19.45)	30.18 (19.45)	30.18 (19.45)	
**PERF:  31.07 (19.45)	31.07 (19.45)	31.07 (19.45)	31.07 (19.45)	31.07 (19.45)	31.07 (19.45)	31.07 (19.45)	31.07 (19.45)	
**PERF:  30.71 (19.45)	30.71 (19.45)	30.71 (19.45)	30.71 (19.45)	30.71 (19.45)	30.71 (19.45)	30.71 (19.45)	30.71 (19.45)	

**PERF:  FPS 0 (Avg)	FPS 1 (Avg)	FPS 2 (Avg)	FPS 3 (Avg)	FPS 4 (Avg)	FPS 5 (Avg)	FPS 6 (Avg)	FPS 7 (Avg)	
**PERF:  31.12 (19.45)	31.12 (19.45)	31.12 (19.45)	31.12 (19.45)	31.12 (19.45)	31.12 (19.45)	31.12 (19.45)	31.12 (19.45)	
**PERF:  30.00 (19.45)	30.00 (19.45)	30.00 (19.45)	30.00 (19.45)	30.00 (19.45)	30.00 (19.45)	30.00 (19.45)	30.00 (19.45)	
**PERF:  28.96 (19.45)	28.96 (19.45)	28.96 (19.45)	28.96 (19.45)	28.96 (19.45)	28.96 (19.45)	28.96 (19.45)	28.96 (19.45)	
**PERF:  30.52 (19.46)	30.52 (19.46)	30.52 (19.46)	30.52 (19.46)	30.52 (19.46)	30.52 (19.46)	30.52 (19.46)	30.52 (19.46)	
ERROR from tiled_display_queue: Internal data stream error.
Debug info: gstqueue.c(988): gst_queue_handle_sink_event (): /GstPipeline:pipeline/GstBin:tiled_display_bin/GstQueue:tiled_display_queue:
streaming stopped, reason error (-5)

Using winsys: x11 
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_mot_klt.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
gstnvtracker: Past frame output is OFF
INFO: Loading Model Engine from File
ERROR: Deserialize engine failed because file path: /opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/../../models/Primary_Detector_Nano/resnet10.caffemodel_b8_gpu0_fp16.engine open error
10:12:23.748229598 11710   0x558b48d520 WARN                 nvinfer gstnvinfer.cpp:672:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1690> [UID = 1]: deserialize engine from file :/opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/../../models/Primary_Detector_Nano/resnet10.caffemodel_b8_gpu0_fp16.engine failed
10:12:23.748314941 11710   0x558b48d520 WARN                 nvinfer gstnvinfer.cpp:672:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1799> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/../../models/Primary_Detector_Nano/resnet10.caffemodel_b8_gpu0_fp16.engine failed, try rebuild
10:12:23.748361597 11710   0x558b48d520 INFO                 nvinfer gstnvinfer.cpp:675:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1715> [UID = 1]: Trying to create engine from model files
INFO: Parsing prototxt a binaryproto Caffe model from files
INFO: [TRT]: Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
INFO: [TRT]: Detected 1 inputs and 2 output network tensors.
ERROR: Serialize engine failed because of file path: /opt/nvidia/deepstream/deepstream-5.0/samples/models/Primary_Detector_Nano/resnet10.caffemodel_b8_gpu0_fp16.engine opened error
10:12:46.418800449 11710   0x558b48d520 WARN                 nvinfer gstnvinfer.cpp:672:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1743> [UID = 1]: failed to serialize cude engine to file: /opt/nvidia/deepstream/deepstream-5.0/samples/models/Primary_Detector_Nano/resnet10.caffemodel_b8_gpu0_fp16.engine
INFO: [Implicit Engine Info]: layers num: 3
0   INPUT  kFLOAT input_1         3x272x480       
1   OUTPUT kFLOAT conv2d_bbox     16x17x30        
2   OUTPUT kFLOAT conv2d_cov/Sigmoid 4x17x30         

10:12:46.438397409 11710   0x558b48d520 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/config_infer_primary_nano.txt sucessfully
**PERF:  32.51 (19.44)	32.45 (19.44)	32.45 (19.44)	32.41 (19.44)	32.68 (19.44)	32.51 (19.44)	32.51 (19.44)	32.68 (19.44)	
** INFO: <bus_callback:181>: Pipeline ready

Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
Opening in BLOCKING MODE 
NvMMLiteBlockCreate : Block : BlockType = 261 
NvMMLiteBlockCreate : Block : BlockType = 261 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NvMMLiteOpen : Block : BlockType = 261 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
Opening in BLOCKING MODE 
NvMMLiteBlockCreate : Block : BlockType = 261 
NvMMLiteOpen : Block : BlockType = 261 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NvMMLiteBlockCreate : Block : BlockType = 261 
NvMMLiteBlockCreate : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
NvMMLiteBlockCreate : Block : BlockType = 261 
** INFO: <bus_callback:167>: Pipeline running

Couldn't create ddkvic Session: Cannot allocate memory
Segmentation fault (core dumped)
13

Hi,
Could you disable nvdsosd for a try? We see fd increasing in running default deepstream-test3. After disabling the plugin, it doesn’t increase. Please help give it a try. Check if you also see improvement by disabling it.

14

Hi,

Thank you for the investigation.
As you suggested, I removed nvdsosd from deepstream-test3. Also I added the implementation for repating the playback by changing the state between PLAYING and NULL. Then, I can see no fd increasing when the playback repeats. The result is as I expected.

I close this issue for the present because it’s getting away from the subject. The things we would like to discuss are complicated and we realize those can’t be sorted out enough yet to discuss here. I’ll create a new issue after we sort out them enough.

It’s great progress for us that we have found out nvdsosd has a fd leak. Thank you.