# DeepStream SDK FAQ

**URL:** <https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236>\
**Category:** DeepStream SDK\
**Created:** [August 20, 2019, 7:54am UTC](https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236 "2019-08-20T07:54:41Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![ChrisDing](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/chrisding/32/14043_2.png) [@ChrisDing](https://forums.developer.nvidia.com/u/ChrisDing)\
**Post date:** [August 20, 2019, 7:54am UTC](https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/1 "2019-08-20T07:54:41Z")

</div>

**1. Reading xxxx code is the best way to get the answer**  
**2. The user of 4.0 needs C/C++ background**   
**3. More resource:**  
[https://docs.nvidia.com/metropolis/](https://docs.nvidia.com/metropolis/)

**nvinfer / nvpreprocess / streamMux / DeMux**

1. Gst-Nvinfer source code diagram  
 ![description](https://global.discourse-cdn.com/nvidia/original/4X/8/e/7/8e7d2d1c39c7a467032729182768ac10412da2eb.svg)

2. Gst-Nvdspreprocess source code diagram  
 ![description](https://global.discourse-cdn.com/nvidia/original/4X/6/6/a/66a8eb4cd6737dbc649fe9dffa6ceabbc878daeb.svg)

3. How to get original NV12 frame buffer  
[https://devtalk.nvidia.com/default/topic/1060956/deepstream-sdk/access-frame-pointer-in-deepstream-app/post/5375214/#5375214](https://devtalk.nvidia.com/default/topic/1060956/deepstream-sdk/access-frame-pointer-in-deepstream-app/post/5375214/#5375214)

4. How to get detection confidence  
[https://devtalk.nvidia.com/default/topic/1060849/deepstream-sdk/deepstream-v4-zero-confidence-problem/?offset=2#5372609](https://devtalk.nvidia.com/default/topic/1060849/deepstream-sdk/deepstream-v4-zero-confidence-problem/?offset=2#5372609)  
[https://devtalk.nvidia.com/default/topic/1058661/deepstream-sdk/nvinfer-is-not-populating-confidence-field-in-nvdsobjectmeta-ds-4-0-/post/5373361/#5373361](https://devtalk.nvidia.com/default/topic/1058661/deepstream-sdk/nvinfer-is-not-populating-confidence-field-in-nvdsobjectmeta-ds-4-0-/post/5373361/#5373361)

5. nvinfer config “model-color-format” is defined in nvdsinfer\_context.h and parsed in gstnvinfer\_property\_parser.cpp  
nvinfer supports not only bgr/rgb, but also gray and other formats.

```auto
/**
 * Enum for color formats.
 */
typedef enum
{
    /** 24-bit interleaved R-G-B */
    NvDsInferFormat_RGB,
    /** 24-bit interleaved B-G-R */
    NvDsInferFormat_BGR,
    /** 8-bit Luma */
    NvDsInferFormat_GRAY,
    /** 32-bit interleaved R-G-B-A */
    NvDsInferFormat_RGBA,
    /** 32-bit interleaved B-G-R-x */
    NvDsInferFormat_BGRx,
    NvDsInferFormat_Unknown = 0xFFFFFFFF,
} NvDsInferFormat;

```

1. How to support each stream to deploy different aglorithm

```auto
diff --git a/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp b/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp
old mode 100644
new mode 100755
index c6867c87..cc70840c
--- a/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp
+++ b/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp
@@ -601,8 +601,10 @@ gst_nvinfer_sink_event (GstBaseTransform * trans, GstEvent * event)
     /* New source added in the pipeline. Create a source info instance for it. */
     guint source_id;
     gst_nvevent_parse_pad_added (event, &source_id);
- nvinfer->source_info->emplace (source_id, GstNvInferSourceInfo ());
- }
+ if (!nvinfer->process_full_frame && /* source_id is what your want for this SGIE */) {
+ nvinfer->source_info->emplace (source_id, GstNvInferSourceInfo ());
+ }
+ }
 
   if ((GstNvEventType) GST_EVENT_TYPE (event) == GST_NVEVENT_PAD_DELETED) {
     /* Source removed from the pipeline. Remove the related structure. */
@@ -1409,6 +1411,8 @@ gst_nvinfer_process_objects (GstNvInfer * nvinfer, GstBuffer * inbuf,
 
     /* Find the source info instance. */
     auto iter = nvinfer->source_info->find (frame_meta->pad_index);
+
+ /* If the source_id is not found, the object will be ignored */
     if (iter == nvinfer->source_info->end ()) {
       GST_WARNING_OBJECT
           (nvinfer,

```

1. How to get/update source\_id  
[https://devtalk.nvidia.com/default/topic/1062520/deepstream-sdk/getting-source-stream-id-from-nvosd-plugin/post/5380861/#5380861](https://devtalk.nvidia.com/default/topic/1062520/deepstream-sdk/getting-source-stream-id-from-nvosd-plugin/post/5380861/#5380861)  
[How to set source\_id in streammux?](https://forums.developer.nvidia.com/t/how-to-set-source-id-in-streammux/120797)

2. FP16 model issue  
If the weights in the model is outside of fp16 range, there will be uff parser issue as the below print:

NvDsInferContext[UID 1]:initialize(): Trying to create engine from model files  
ERROR nvinfer gstnvinfer.cpp:511:gst\_nvinfer\_logger: NvDsInferContext[UID 1]:log(): UffParser: Parser error: bn\_conv1/moving\_variance: Weight 110542.968750 at index 8 is outside of [-65504.000000, 65504.000000]. Please try running the parser in a higher precision mode and setting the builder to fp16 mode instead.  
NvDsInferCudaEngineGetFromTltModel: Failed to parse UFF model

In order to fix this issue, we can apply this patch to the nvinfer source code and build a new libnvds\_infer.so to replace

```auto
--- a/src/utils/nvdsinfer/nvdsinfer_context_impl.cpp
+++ b/src/utils/nvdsinfer/nvdsinfer_context_impl.cpp
@@ -1851,7 +1851,7 @@ NvDsInferContextImpl::generateTRTModel(
         }
 
         if (!uffParser->parse(initParams.uffFilePath,
- *network, modelDataType))
+ *network,DataType::kFLOAT))

```

9.Here’s a simple example CUDA kernel of cropping image: [https://github.com/dusty-nv/jetson-video/blob/master/cuda/cudaCrop.cu](https://github.com/dusty-nv/jetson-video/blob/master/cuda/cudaCrop.cu)

1. How to deploy mrcnn model ([GitHub - matterport/Mask\_RCNN: Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow](https://github.com/matterport/Mask_RCNN)) with resnet50 backbone and classNum change in h5 model ?  
[https://devtalk.nvidia.com/default/topic/1031938/deepstream-sdk/converting-mask-rcnn-to-tensor-rt/post/5416100/#5416100](https://devtalk.nvidia.com/default/topic/1031938/deepstream-sdk/converting-mask-rcnn-to-tensor-rt/post/5416100/#5416100)

2. How to disable object detecdtion for different sources?  
[https://devtalk.nvidia.com/default/topic/1068016/deepstream-sdk/can-deepstream-select-to-enable-or-disable-object-detection-for-different-sources/post/5410165/#5410165](https://devtalk.nvidia.com/default/topic/1068016/deepstream-sdk/can-deepstream-select-to-enable-or-disable-object-detection-for-different-sources/post/5410165/#5410165)

---

<div class="post-metadata">

**Author:** ![ChrisDing](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/chrisding/32/14043_2.png) [@ChrisDing](https://forums.developer.nvidia.com/u/ChrisDing)\
**Post date:** [August 20, 2019, 7:56am UTC](https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/2 "2019-08-20T07:56:37Z")

</div>

**metadata / msgconv / msgbroker / Codec/ ds-app**  
**1.** Sample of adding metadata  
[https://devtalk.nvidia.com/default/topic/1061083/deepstream-sdk/attaching-custom-type-metadata-to-gstreamer-buffer-on-src-pad-causing-sudden-crash/post/5374690/#5374690](https://devtalk.nvidia.com/default/topic/1061083/deepstream-sdk/attaching-custom-type-metadata-to-gstreamer-buffer-on-src-pad-causing-sudden-crash/post/5374690/#5374690)

**2.** Sample of customizing gst-dsexample:  
[https://devtalk.nvidia.com/default/topic/1061422/deepstream-sdk/how-to-crop-the-image-and-save/post/5375174/#5375174](https://devtalk.nvidia.com/default/topic/1061422/deepstream-sdk/how-to-crop-the-image-and-save/post/5375174/#5375174)

**3.** Sample config file of running single RTSP source:  
[https://devtalk.nvidia.com/default/topic/1058086/deepstream-sdk/how-to-run-rtp-camera-in-deepstream-on-nano/post/5366807/#5366807](https://devtalk.nvidia.com/default/topic/1058086/deepstream-sdk/how-to-run-rtp-camera-in-deepstream-on-nano/post/5366807/#5366807)

**5.** Sample of accessing NvBufSurface  
[https://devtalk.nvidia.com/default/topic/1061205/deepstream-sdk/rtsp-camera-access-frame-issue/post/5377678/#5377678](https://devtalk.nvidia.com/default/topic/1061205/deepstream-sdk/rtsp-camera-access-frame-issue/post/5377678/#5377678)

**6.** Use GST\_PAD\_PROBE\_DROP macro to drop the buffer in the attached probe.  
Refer to [Pipeline manipulation](https://gstreamer.freedesktop.org/documentation/application-development/advanced/pipeline-manipulation.html?gi-language=c) for the example

```auto
static GstPadProbeReturn
event_probe_cb (GstPad * pad, GstPadProbeInfo * info, gpointer user_data)
{
    return GST_PAD_PROBE_DROP;
}

```

**7.** Add dsexample in ds-test1 app  
[https://devtalk.nvidia.com/default/topic/1065406/deepstream-sdk/enable-dsexample-in-test-app/?offset=3#5398407](https://devtalk.nvidia.com/default/topic/1065406/deepstream-sdk/enable-dsexample-in-test-app/?offset=3#5398407)

**8.** Optical flow  
Optical flow functionality is supported only on Jetson AGX Xavier and Turing GPUs T4 / RTX 2080 etc. It won’t work on Jetson Nano and GTX

**9.** How can we set “drop-frame-interval” more than 30 ?  
a. Find and download “L4t source” from [https://developer.nvidia.com/embedded/downloads#?search=source](https://developer.nvidia.com/embedded/downloads#?search=source)  
gst-nvvideo4linux2\_src.tbz2 is in public\_sources.tbz2  
b. Apply the patches “0001-gstv4l2dec-Fix-high-CPU-usage-in-drop-frame.patch” and " 0002-gst-v4l2dec-Increase-Drop-Frame-Interval.patch"  
c. Build a new libgstnvvideo4linux2.so and replace /usr/lib/$(ARCH)/gstreamer-1.0/libgstnvvideo4linux2.so

0001-gstv4l2dec-Fix-high-CPU-usage-in-drop-frame.patch

```auto
From 5d8d5a0977473eae89c0f310171d2c7060e24eb6 Mon Sep 17 00:00:00 2001
From: vpagar <vpagar@nvidia.com>
Date: Thu, 5 Dec 2019 16:04:02 +0530
Subject: [PATCH 1/2] gstv4l2dec: Fix high CPU usage in drop-frame

In case of drop-frame-interval, in LL v4l2 implementation a
thread in low level v4l2 lib which sends buffer to block and
a callback thread spins between themselves causing high CPU percentage
usage over the perid.
This CL drops frame at the gstreamer level and LL v4l2 does not handle
dropping frames.

Unit-Test:
gst-launch-1.0 multifilesrc location= sample_720p.h264 \
! h264parse ! nvv4l2decoder drop-frame-interval=3 ! fakesink
and check CPU percentage usage in htop, it should stay stable.

Bug 200562189

Change-Id: I9af22745501d6a9892c341cb640dac16f8641763
---
 gst-v4l2/gstv4l2videodec.c | 23 ++++++++++++++++++++++-
 gst-v4l2/gstv4l2videodec.h | 1 +
 2 files changed, 23 insertions(+), 1 deletion(-)

diff --git a/gst-v4l2/gstv4l2videodec.c b/gst-v4l2/gstv4l2videodec.c
index 5531f9d..f8c62f2 100644
--- a/gst-v4l2/gstv4l2videodec.c
+++ b/gst-v4l2/gstv4l2videodec.c
@@ -593,6 +593,9 @@ gst_v4l2_video_dec_start (GstVideoDecoder * decoder)
   gst_v4l2_object_unlock (self->v4l2output);
   g_atomic_int_set (&self->active, TRUE);
   self->output_flow = GST_FLOW_OK;
+#if USE_V4L2_TARGET_NV
+ self->decoded_picture_cnt = 0;
+#endif
 
   return TRUE;
 }
@@ -704,6 +707,11 @@ gst_v4l2_video_dec_set_format (GstVideoDecoder * decoder,
     }
   }
 
+#if 0
+ /* *
+ * TODO: From low level library remove support of drop frame interval after
+ * analyzing high CPU utilization in initial implementation.
+ * */
   if (self->drop_frame_interval != 0) {
     if (!set_v4l2_video_mpeg_class (self->v4l2output,
         V4L2_CID_MPEG_VIDEODEC_DROP_FRAME_INTERVAL,
@@ -712,6 +720,7 @@ gst_v4l2_video_dec_set_format (GstVideoDecoder * decoder,
       return FALSE;
     }
   }
+#endif
 #ifndef USE_V4L2_TARGET_NV_CODECSDK
   if (self->disable_dpb != DEFAULT_DISABLE_DPB) {
     if (!set_v4l2_video_mpeg_class (self->v4l2output,
@@ -1141,10 +1150,21 @@ gst_v4l2_video_dec_loop (GstVideoDecoder * decoder)
       gst_caps_unref(reference);
     }
 
- ret = gst_video_decoder_finish_frame (decoder, frame);
+#if USE_V4L2_TARGET_NV
+ if ((self->drop_frame_interval == 0) ||
+ (self->decoded_picture_cnt % self->drop_frame_interval == 0))
+ ret = gst_video_decoder_finish_frame (decoder, frame);
+ else
+ ret = gst_video_decoder_drop_frame (GST_VIDEO_DECODER (self), frame);
 
     if (ret != GST_FLOW_OK)
       goto beach;
+
+ self->decoded_picture_cnt += 1;
+#else
+ ret = gst_video_decoder_finish_frame (decoder, frame);
+#endif
+
   } else {
     GST_WARNING_OBJECT (decoder, "Decoder is producing too many buffers");
     gst_buffer_unref (buffer);
@@ -1696,6 +1716,7 @@ gst_v4l2_video_dec_init (GstV4l2VideoDec * self)
   self->skip_frames = DEFAULT_SKIP_FRAME_TYPE;
   self->nvbuf_api_version_new = DEFAULT_NVBUF_API_VERSION_NEW;
   self->drop_frame_interval = 0;
+ self->decoded_picture_cnt = 0;
   self->num_extra_surfaces = DEFAULT_NUM_EXTRA_SURFACES;
 #ifndef USE_V4L2_TARGET_NV_CODECSDK
   self->disable_dpb = DEFAULT_DISABLE_DPB;
diff --git a/gst-v4l2/gstv4l2videodec.h b/gst-v4l2/gstv4l2videodec.h
index 50d07c5..5015c30 100644
--- a/gst-v4l2/gstv4l2videodec.h
+++ b/gst-v4l2/gstv4l2videodec.h
@@ -71,6 +71,7 @@ struct _GstV4l2VideoDec
   GstFlowReturn output_flow;
   guint64 frame_num;
 #ifdef USE_V4L2_TARGET_NV
+ guint64 decoded_picture_cnt;
   guint32 skip_frames;
   guint32 drop_frame_interval;
   gboolean nvbuf_api_version_new;
-- 
2.17.1

```

0002-gst-v4l2dec-Increase-Drop-Frame-Interval.patch

```auto
From 52665605036144ac20628c95e52fdd82edae71b9 Mon Sep 17 00:00:00 2001
From: vpagar <vpagar@nvidia.com>
Date: Wed, 11 Dec 2019 11:56:25 +0530
Subject: [PATCH 2/2] gst-v4l2dec: Increase Drop Frame Interval

Bug 200575866

Change-Id: If5576683c0fad95832595838d032d3145b88ea36
---
 gst-v4l2/gstv4l2videodec.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gst-v4l2/gstv4l2videodec.c b/gst-v4l2/gstv4l2videodec.c
index f8c62f2..00d7740 100644
--- a/gst-v4l2/gstv4l2videodec.c
+++ b/gst-v4l2/gstv4l2videodec.c
@@ -1807,7 +1807,7 @@ gst_v4l2_video_dec_class_init (GstV4l2VideoDecClass * klass)
           "Drop frames interval",
           "Interval to drop the frames,ex: value of 5 means every 5th frame will be given by decoder, rest all dropped",
           0,
- 30, 30,
+ G_MAXUINT, G_MAXUINT,
           G_PARAM_READWRITE | G_PARAM_STATIC_STRINGS | GST_PARAM_MUTABLE_READY));
 
   g_object_class_install_property (gobject_class, PROP_NUM_EXTRA_SURFACES,
-- 
2.17.1

```

10.use deepstream-app option  
refer [https://devtalk.nvidia.com/default/topic/1069070/deepstream-sdk/labels-disappear-with-multiple-sources/post/5415523/#5415523](https://devtalk.nvidia.com/default/topic/1069070/deepstream-sdk/labels-disappear-with-multiple-sources/post/5415523/#5415523)

---

<div class="post-metadata">

**Author:** ![anassamar8](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@anassamar8](https://forums.developer.nvidia.com/u/anassamar8)\
**Post date:** [August 21, 2019, 6:32am UTC](https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/3 "2019-08-21T06:32:42Z")

</div>

Thank you very much ChrisDing. This is really helpful.

---

<div class="post-metadata">

**Author:** ![ChrisDing](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/chrisding/32/14043_2.png) [@ChrisDing](https://forums.developer.nvidia.com/u/ChrisDing)\
**Post date:** [December 16, 2019, 2:48am UTC](https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/4 "2019-12-16T02:48:59Z")

</div>

**nvtracker standalone user sample**

[https://devtalk.nvidia.com/default/topic/1066252/deepstream-sdk/klt-nvmot-usage/](https://devtalk.nvidia.com/default/topic/1066252/deepstream-sdk/klt-nvmot-usage/)

**get nvtracker history**

[https://devtalk.nvidia.com/default/topic/1061798/deepstream-sdk/how-to-obtain-previous-states-of-tracked-object-/](https://devtalk.nvidia.com/default/topic/1061798/deepstream-sdk/how-to-obtain-previous-states-of-tracked-object-/)

---

<div class="post-metadata">

**Author:** ![bcao](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/bcao/32/14043_2.png) [@bcao](https://forums.developer.nvidia.com/u/bcao)\
**Post date:** [March 30, 2020, 2:49pm UTC](https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/5 "2020-03-30T14:49:50Z")

</div>

**Fix for a memory accumulation bug in GstBaseParse**  
A memory accumulation bug was found in GStreamer’s Base Parse class which potentially affects all codec parsers provided by GStreamer. This bug is seen only with long duration seekable streams (mostly containerized files e.g. mp4). This does not affect live sources like RTSP. We have filed an issue on GStreamer’s gitlab project ([gstbaseparse: High memory usage in association index for long duration files (#468) · Issues · GStreamer / gstreamer · GitLab](https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/468)).

**Temporary fix**

1. Check the exact gstreamer version installed on the system.

> $ gst-inspect-1.0 --version

gst-inspect-1.0 version 1.14.5

GStreamer 1.14.5

[https://launchpad.net/distros/ubuntu/+source/gstreamer1.0](https://launchpad.net/distros/ubuntu/+source/gstreamer1.0)

1. Clone the Gstreamer repo and checkout the tag corresponding to the installed version

> $ git clone [git@gitlab.freedesktop.org](mailto:git@gitlab.freedesktop.org):gstreamer/gstreamer.git

> $ cd gstreamer

> $ git checkout 1.14.5

1. Make sure build dependencies are installed

> $ sudo apt install libbison-dev build-essential flex debhelper

1. Run autogen.sh and configure script

> $ ./autogen.sh –noconfigure

> $ ./configure –prefix=$(pwd)/out # Don’t want to overwrite system libs

1. Save the following patch to a file

```auto
diff --git a/libs/gst/base/gstbaseparse.c b/libs/gst/base/gstbaseparse.c
index 41adf130e..ffc662a45 100644
--- a/libs/gst/base/gstbaseparse.c
+++ b/libs/gst/base/gstbaseparse.c
@@ -1906,6 +1906,9 @@ gst_base_parse_add_index_entry (GstBaseParse * parse, guint64 offset,
   GST_LOG_OBJECT (parse, "Adding key=%d index entry %" GST_TIME_FORMAT
       " @ offset 0x%08" G_GINT64_MODIFIER "x", key, GST_TIME_ARGS (ts), offset);
 
+ if (!key)
+ goto exit;
+
   if (G_LIKELY (!force)) {
 
     if (!parse->priv->upstream_seekable) {

```

1. Apply the patch

> $ cat patch.txt | patch -p1

1. Build the sources

> $ make -j$(nproc) && make install

1. Backup the distribution provided library and copy the newly built library. Adjust the library name for version. For jetson replace x86\_64-linux-gnu with aarch64-linux-gnu

> $ sudo cp /usr/lib/x86\_64-linux-gnu/libgstbase-1.0.so.0.1405.0 ${HOME}/libgstbase-1.0.so.0.1405.0.backup

> $ sudo cp out/lib/libgstbase-1.0.so.0.1405.0 /usr/lib/x86\_64-linux-gnu/

---

<div class="post-metadata">

**Author:** ![mchi](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/mchi/32/14043_2.png) [@mchi](https://forums.developer.nvidia.com/u/mchi)\
**Post date:** [April 27, 2020, 1:07am UTC](https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/6 "2020-04-27T01:07:09Z")

</div>



---

<div class="post-metadata">

**Author:** ![bcao](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/bcao/32/14043_2.png) [@bcao](https://forums.developer.nvidia.com/u/bcao)\
**Post date:** [July 9, 2020, 7:34am UTC](https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/7 "2020-07-09T07:34:47Z")

</div>

**[DS5.0 xx\_All\_App] For DS 5.0 DP: how to integrate nvdsanalytics plugin in C deepstream-app**

1. User need to create analytics bin in /opt/nvidia/deepstream/deepstream-5.0/sources/apps/apps-common/src
2. Refer deepstream\_dsexample.c and similarly create deepstream\_nvdsanalytics.c
3. deepstream\_app.h should be modified to add the instance of nvdsanalytics bin and config in the structures
4. deepstream\_config\_file\_parser.c needs to updated for parsing of nvdsanalytics config from configuration file
5. deepstream\_app.c should be updated for adding the nvdsanalytics bin in the pipeline, ideally location is after the tracker
6. Create a new cpp file with process\_meta function declared with extern “C”, this will parse the meta for nvdsanalytics, refer sample nvdanalytics test app probe call for creation of the function
7. Add the probe in deepstream\_app\_main.c after nvdsanalytics bin
8. Modify Makefile to compile the cpp and deepstream\_app\_main.c using g++ with -fpermisive flag and link deepstream-app using g++

These are rough steps, but additional modifications in header files required

For DS 5.0 GA we would be adding the support for meta access

---

<div class="post-metadata">

**Author:** ![ersheng](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/ersheng/32/15508_2.png) [@ersheng](https://forums.developer.nvidia.com/u/ersheng)\
**Post date:** [July 10, 2020, 3:50am UTC](https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/8 "2020-07-10T03:50:12Z")

</div>

**DeepStream 5.0 Manual for YoloV4**

- The original Yolo implementation via CUDA kernel in DeepStream is based on old Yolo models (v2, v3) so it may not suit new Yolo models like YoloV4. Location: `/opt/nvidia/deepstream/deepstream-5.0/sources/objectDetector_Yolo/nvdsinfer_custom_impl_Yolo/kernels.cu`

- We are trying to embed Yolo layer into tensorRT engine **while converting darknet or pytorch into engine** , this is before deploying to DeepStream. This new solution would cause the old Yolo cuda kernel in DeepStream no longer to be used.

You can try following steps to make DeepStream working for YoloV4:

1. go to [https://github.com/Tianxiaomo/pytorch-YOLOv4](https://github.com/Tianxiaomo/pytorch-YOLOv4) to generate a TensorRT engine according to this workflow: **DarkNet or Pytorch → ONNX → TensorRT**.
2. Add following C++ functions into **`objectDetector_Yolo/nvdsinfer_custom_impl_Yolo/nvdsparsebbox_Yolo.cpp`** and rebuild **`libnvdsinfer_custom_impl_Yolo.so`**
3. Here are configuration files for you as references (You have to update a little to suit your environment):  
[config\_infer\_primary\_yoloV4.txt](https://forums.developer.nvidia.com/uploads/short-url/pgQwGTK26qwPZTmFK7JXPilwjwn.txt) (3.4 KB)  
[deepstream\_app\_config\_yoloV4.txt](https://forums.developer.nvidia.com/uploads/short-url/ynkafGDu8PVD5Fmczt3Zm7LYrTn.txt) (3.8 KB)

```cpp
static NvDsInferParseObjectInfo convertBBoxYoloV4(const float& bx1, const float& by1, const float& bx2,
                                     const float& by2, const uint& netW, const uint& netH)
{
    NvDsInferParseObjectInfo b;
    // Restore coordinates to network input resolution

    float x1 = bx1 * netW;
    float y1 = by1 * netH;
    float x2 = bx2 * netW;
    float y2 = by2 * netH;

    x1 = clamp(x1, 0, netW);
    y1 = clamp(y1, 0, netH);
    x2 = clamp(x2, 0, netW);
    y2 = clamp(y2, 0, netH);

    b.left = x1;
    b.width = clamp(x2 - x1, 0, netW);
    b.top = y1;
    b.height = clamp(y2 - y1, 0, netH);

    return b;
}

static void addBBoxProposalYoloV4(const float bx, const float by, const float bw, const float bh,
                     const uint& netW, const uint& netH, const int maxIndex,
                     const float maxProb, std::vector<NvDsInferParseObjectInfo>& binfo)
{
    NvDsInferParseObjectInfo bbi = convertBBoxYoloV4(bx, by, bw, bh, netW, netH);
    if (bbi.width < 1 || bbi.height < 1) return;

    bbi.detectionConfidence = maxProb;
    bbi.classId = maxIndex;
    binfo.push_back(bbi);
}

static std::vector<NvDsInferParseObjectInfo>
decodeYoloV4Tensor(
    const float* boxes, const float* scores,
    const uint num_bboxes, NvDsInferParseDetectionParams const& detectionParams,
    const uint& netW, const uint& netH)
{
    std::vector<NvDsInferParseObjectInfo> binfo;

    uint bbox_location = 0;
    uint score_location = 0;
    for (uint b = 0; b < num_bboxes; ++b)
    {
        float bx1 = boxes[bbox_location];
        float by1 = boxes[bbox_location + 1];
        float bx2 = boxes[bbox_location + 2];
        float by2 = boxes[bbox_location + 3];

        float maxProb = 0.0f;
        int maxIndex = -1;

        for (uint c = 0; c < detectionParams.numClassesConfigured; ++c)
        {
            float prob = scores[score_location + c];
            if (prob > maxProb)
            {
                maxProb = prob;
                maxIndex = c;
            }
        }

        if (maxProb > detectionParams.perClassPreclusterThreshold[maxIndex])
        {
            addBBoxProposalYoloV4(bx1, by1, bx2, by2, netW, netH, maxIndex, maxProb, binfo);
        }

        bbox_location += 4;
        score_location += detectionParams.numClassesConfigured;
    }

    return binfo;
}

extern "C" bool NvDsInferParseCustomYoloV4(
    std::vector<NvDsInferLayerInfo> const& outputLayersInfo,
    NvDsInferNetworkInfo const& networkInfo,
    NvDsInferParseDetectionParams const& detectionParams,
    std::vector<NvDsInferParseObjectInfo>& objectList)
{
    if (NUM_CLASSES_YOLO != detectionParams.numClassesConfigured)
    {
        std::cerr << "WARNING: Num classes mismatch. Configured:"
                  << detectionParams.numClassesConfigured
                  << ", detected by network: " << NUM_CLASSES_YOLO << std::endl;
    }

    std::vector<NvDsInferParseObjectInfo> objects;

    const NvDsInferLayerInfo &boxes = outputLayersInfo[0]; // num_boxes x 4
    const NvDsInferLayerInfo &scores = outputLayersInfo[1]; // num_boxes x num_classes

    // 3 dimensional: [num_boxes, 1, 4]
    assert(boxes.inferDims.numDims == 3);
    // 2 dimensional: [num_boxes, num_classes]
    assert(scores.inferDims.numDims == 2);

    // The second dimension should be num_classes
    assert(detectionParams.numClassesConfigured == scores.inferDims.d[1]);
    
    uint num_bboxes = boxes.inferDims.d[0];

    // std::cout << "Network Info: " << networkInfo.height << " " << networkInfo.width << std::endl;

    std::vector<NvDsInferParseObjectInfo> outObjs =
        decodeYoloV4Tensor(
            (const float*)(boxes.buffer), (const float*)(scores.buffer), num_bboxes, detectionParams,
            networkInfo.width, networkInfo.height);

    objects.insert(objects.end(), outObjs.begin(), outObjs.end());

    objectList = objects;

    return true;
}

```

---

<div class="post-metadata">

**Author:** ![mchi](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/mchi/32/14043_2.png) [@mchi](https://forums.developer.nvidia.com/u/mchi)\
**Post date:** [October 13, 2020, 5:34am UTC](https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/9 "2020-10-13T05:34:49Z")

</div>

**1. [DS5.0GA\_Jetson\_dGPU\_Plugin] Measure of the FPS of pipeline**

> [@Slow real-time performance when running custom YOLOv3 app](https://forums.developer.nvidia.com/t/slow-real-time-performance-when-running-custom-yolov3-app/155994/7):
>
> You can refer to below change to measure the fps diff --git a/deepstream\_test2\_app.c b/deepstream\_test2\_app.c index 949219f..e8ed8f7 100644 --- a/deepstream\_test2\_app.c +++ b/deepstream\_test2\_app.c @@ -80,6 +80,12 @@ guint sgie1\_unique\_id = 2; guint sgie2\_unique\_id = 3; guint sgie3\_unique\_id = 4; +typedef struct \_perf\_measure{ + GstClockTime pre\_time; + GstClockTime total\_time; + guint count; +}perf\_measure; + /\* This is the buffer probe function that we have registered on the sin…

**2. [DS5.0GA\_Jetson\_dGPU\_Plugin] Dump the Inference Input**

> [@YOLO SGIE problem with resize](https://forums.developer.nvidia.com/t/yolo-sgie-problem-with-resize/155977/13):
>
> [dump\_infer\_input\_to\_file.patch.txt](https://forums.developer.nvidia.com/uploads/short-url/byXzsrCQMrPDx5qTGtfLiZhCUBO.txt) (8.8 KB) Could you try attached change to dump the input just before calling TRT inference API - enqueue() ?

**3. [DS5\_Jetson\_dGPU\_Plugin] Dump the Inference outputs**

- apply Attached [dump\_dsinfer\_raw\_TRT\_infer\_outputs.txt](https://forums.developer.nvidia.com/uploads/short-url/tEGZIkZvCJeNMNb2Sn6EskYFd24.txt) (1.8 KB) into /opt/nvidia/deepstream/deepstream/sources/libs/nvdsinfer/
- build libnvds\_infer.so and replace /opt/nvidia/deepstream/deepstream-6.0/lib/libnvds\_infer.so

**4. [DS5.0GA\_Jetson\_App] Rotate camera input image with NvBufSurfTransform() API**

> [@Jetson Nano CSI Raspberry Pi Camera V2 upside down video when run an example with deepstream-app](https://forums.developer.nvidia.com/t/jetson-nano-csi-raspberry-pi-camera-v2-upside-down-video-when-run-an-example-with-deepstream-app/82077/7):
>
> Hi, Please use NvBufSurface APIs. Below is a patch for reference: sources\apps\apps-common\src\deepstream\_source\_bin.c: @@ -22,6 +22,8 @@ #include \<string.h\> +#include "nvbufsurface.h" +#include "nvbufsurftransform.h" #include "gstnvdsmeta.h" #include "deepstream\_common.h" #include "deepstream\_sources.h" @@ -62,6 +64,64 @@ set\_camera\_v4l2\_params (NvDsSourceConfig \* config, NvDsSrcBin \* bin) return TRUE; } +static GstPadProbeReturn +nvargus\_src\_pad\_buffer\_probe (GstPad \* pad, Gst…

---

<div class="post-metadata">

**Author:** ![mchi](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/mchi/32/14043_2.png) [@mchi](https://forums.developer.nvidia.com/u/mchi)\
**Post date:** [November 8, 2020, 1:58pm UTC](https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/10 "2020-11-08T13:58:35Z")

</div>

**5. [DS5.0GA\_App] Generate GStreamer Pipeline Graph**  
Use one of below method according to your application type to generate the GST pipeline graph.

```
4.1 deepstream-app
 run "export GST_DEBUG_DUMP_DOT_DIR=/tmp/" before deepstream-app command, e.g.
 $ sudo apt-get install graphviz
 $ export GST_DEBUG_DUMP_DOT_DIR=/tmp/
 $ deepstream-app -c deepstream_app_config_yoloV2.txt
 $ cd /tmp/
 $ dot -Tpng 0.03.47.898178403-ds-app-playing.dot >~/0.03.47.898178403-ds-app-playing.png // png file includes the graph

4.2 gstreamer command line
for exmaple,
  $ run "export GST_DEBUG_DUMP_DOT_DIR=/tmp/" before deepstream-app command, e.g.
  $ sudo apt-get install graphviz
  $ export GST_DEBUG_DUMP_DOT_DIR=/tmp/
  $ gst-launch-1.0 ....
  $ cd /tmp/
  $ dot -Tpng 0.03.47.898178403-ds-app-playing.dot >~/0.03.47.898178403-ds-app-playing.png // png file includes the graph

 4.3 DeepStream application
  for exmaple
  4.3.1 add "g_setenv("GST_DEBUG_DUMP_DOT_DIR", "/tmp", TRUE);" before gst_init()
  4.3.2 add "GST_DEBUG_BIN_TO_DOT_FILE_WITH_TS(GST_BIN(gst_objs.pipeline), GST_DEBUG_GRAPH_SHOW_ALL, "demo-app-pipeline");" at the point where want to export the dot file, e.g. when switching to PLAYING
   BTW, need to include header file - #include <gio/gio.h>

 4.4 Python DeepStream
  Refer to https://forums.developer.nvidia.com/t/python-deepstream-program-not-generating-dot-file/163837/8?u=mchi

```

---

<div class="post-metadata">

**Author:** ![bcao](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/bcao/32/14043_2.png) [@bcao](https://forums.developer.nvidia.com/u/bcao)\
**Post date:** [November 20, 2020, 3:35am UTC](https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/11 "2020-11-20T03:35:39Z")

</div>

**6. [DS 5.0.1\_All\_Plugin] Tracker FAQ topic**

[Deepstream Tracker FAQ](https://forums.developer.nvidia.com/t/deepstream-tracker-faq/160014)

---

<div class="post-metadata">

**Author:** ![bcao](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/bcao/32/14043_2.png) [@bcao](https://forums.developer.nvidia.com/u/bcao)\
**Post date:** [November 25, 2020, 9:25am UTC](https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/12 "2020-11-25T09:25:50Z")

</div>

**7. [DS 5.0GA\_All\_App] Enable Latency measurement for deepstream sample apps**

1. If you are using deepstream-app, to check the component latency directly, you need to set the env

2. If you are using other deepstream sample apps such as deepstream-test3, you need to apply the following patch and set the env

```patch
diff --git a/apps/deepstream/sample_apps/deepstream-test3/deepstream_test3_app.c b/apps/deepstream/sample_apps/deepstream-test3/deepstream_test3_app.c
index 426bd69..c7c2472 100644
--- a/apps/deepstream/sample_apps/deepstream-test3/deepstream_test3_app.c
+++ b/apps/deepstream/sample_apps/deepstream-test3/deepstream_test3_app.c
@@ -26,6 +26,7 @@
 #include <math.h>
 #include <string.h>
 #include <sys/time.h>
+#include <stdlib.h>

 #include "gstnvdsmeta.h"
 //#include "gstnvstreammeta.h"
@@ -73,6 +74,41 @@ gchar pgie_classes_str[4][32] = { "Vehicle", "TwoWheeler", "Person",

 //static guint probe_counter = 0;

+typedef struct {
+ GMutex *lock;
+ int num_sources;
+}LatencyCtx;
+
+static GstPadProbeReturn
+latency_measurement_buf_prob(GstPad * pad, GstPadProbeInfo * info, gpointer u_data)
+{
+ LatencyCtx *ctx = (LatencyCtx *) u_data;
+ static int batch_num = 0;
+ guint i = 0, num_sources_in_batch = 0;
+ if(nvds_enable_latency_measurement)
+ {
+ GstBuffer *buf = (GstBuffer *) info->data;
+ NvDsFrameLatencyInfo *latency_info = NULL;
+ g_mutex_lock (ctx->lock);
+ latency_info = (NvDsFrameLatencyInfo *)
+ calloc(1, ctx->num_sources * sizeof(NvDsFrameLatencyInfo));;
+ g_print("\n ************BATCH-NUM = %d************** \n",batch_num);
+ num_sources_in_batch = nvds_measure_buffer_latency(buf, latency_info);
+
+ for(i = 0; i < num_sources_in_batch; i++)
+ {
+ g_print("Source id = %d Frame_num = %d Frame latency = %lf (ms) \n",
+ latency_info[i].source_id,
+ latency_info[i].frame_num,
+ latency_info[i].latency);
+ }
+ g_mutex_unlock (ctx->lock);
+ batch_num++;
+ }
+
+ return GST_PAD_PROBE_OK;
+}
+
 /* tiler_sink_pad_buffer_probe will extract metadata received on OSD sink pad
  * and update params for drawing rectangle, object information etc. */

@@ -107,9 +143,9 @@ tiler_src_pad_buffer_probe (GstPad * pad, GstPadProbeInfo * info,
                 num_rects++;
             }
         }
- g_print ("Frame Number = %d Number of objects = %d "
- "Vehicle Count = %d Person Count = %d\n",
- frame_meta->frame_num, num_rects, vehicle_count, person_count);
+ // g_print ("Frame Number = %d Number of objects = %d "
+ // "Vehicle Count = %d Person Count = %d\n",
+ // frame_meta->frame_num, num_rects, vehicle_count, person_count);
 #if 0
         display_meta = nvds_acquire_display_meta_from_pool(batch_meta);
         NvOSD_TextParams *txt_params = &display_meta->text_params;
@@ -383,7 +419,7 @@ main (int argc, char *argv[])
 #ifdef PLATFORM_TEGRA
   transform = gst_element_factory_make ("nvegltransform", "nvegl-transform");
 #endif
- sink = gst_element_factory_make ("nveglglessink", "nvvideo-renderer");
+ sink = gst_element_factory_make ("fakesink", "nvvideo-renderer");

   if (!pgie || !tiler || !nvvidconv || !nvosd || !sink) {
     g_printerr ("One element could not be created. Exiting.\n");
@@ -467,6 +503,18 @@ gst_bin_add_many (GST_BIN (pipeline), queue1, pgie, queue2, tiler, queue3,
         tiler_src_pad_buffer_probe, NULL, NULL);
   gst_object_unref (tiler_src_pad);

+ GstPad *sink_pad = gst_element_get_static_pad (nvosd, "src");
+ if (!sink_pad)
+ g_print ("Unable to get src pad\n");
+ else {
+ LatencyCtx *ctx = (LatencyCtx *)g_malloc0(sizeof(LatencyCtx));
+ ctx->lock = (GMutex *)g_malloc0(sizeof(GMutex));
+ ctx->num_sources = num_sources;
+ gst_pad_add_probe (sink_pad, GST_PAD_PROBE_TYPE_BUFFER,
+ latency_measurement_buf_prob, ctx, NULL);
+ }
+ gst_object_unref (sink_pad);
+
   /* Set the pipeline to "playing" state */
   g_print ("Now playing:");
   for (i = 0; i < num_sources; i++) {

```

1. Before DeepStream7.0(deepstream\_python\_apps v1.1.11), if you use a python app such as deepstream\_test\_3.py, you need to apply the following patch and set the env

export NVDS\_ENABLE\_COMPONENT\_LATENCY\_MEASUREMENT=1  
export NVDS\_ENABLE\_LATENCY\_MEASUREMENT=1

`pip install cffi`

Then execute like below

```auto
python3 deepstream_test_3.py --no-display -i rtsp://"your_rtsp_uri_0" uri1

```

```patch
diff --git a/apps/deepstream-test3/deepstream_test_3.py b/apps/deepstream-test3/deepstream_test_3.py
index d81ec92..21d2f3b 100755
--- a/apps/deepstream-test3/deepstream_test_3.py
+++ b/apps/deepstream-test3/deepstream_test_3.py
@@ -36,6 +36,28 @@ from common.FPS import PERF_DATA
 
 import pyds
 
+from cffi import FFI
+
+ffi = FFI()
+
+clib = None
+
+ffi.cdef("""
+typedef struct
+{
+ uint32_t source_id;
+ uint32_t frame_num;
+ double comp_in_timestamp;
+ double latency;
+} NvDsFrameLatencyInfo;
+
+uint32_t nvds_measure_buffer_latency(void *buf, NvDsFrameLatencyInfo *latency_info);
+bool nvds_get_enable_latency_measurement();
+""")
+
+# Compile the C sources to produce the following .dll (or .so under *nix)
+clib = ffi.dlopen("/opt/nvidia/deepstream/deepstream/lib/libnvdsgst_meta.so")
+
 no_display = False
 silent = False
 file_loop = False
@@ -56,6 +78,27 @@ OSD_PROCESS_MODE= 0
 OSD_DISPLAY_TEXT= 1
 pgie_classes_str= ["Vehicle", "TwoWheeler", "Person","RoadSign"]
 
+batch_num = 0
+
+def osd_src_pad_buffer_probe(pad, info, u_data):
+ number_source = u_data
+ gst_buffer = info.get_buffer()
+ if not gst_buffer:
+ print("Unable to get GstBuffer ")
+ return
+ global batch_num
+ if clib.nvds_get_enable_latency_measurement:
+ print(f" ************BATCH-NUM = {batch_num}**************")
+ c_gst_buf = ffi.cast("void *", hash(gst_buffer))
+ cNvDsFrameLatencyInfo = ffi.new(f"NvDsFrameLatencyInfo[{number_source}]")
+ sources = clib.nvds_measure_buffer_latency(c_gst_buf, cNvDsFrameLatencyInfo)
+ for i in range(sources):
+ print(f"Source id = {cNvDsFrameLatencyInfo[i].source_id} "
+ f"Frame_num = {cNvDsFrameLatencyInfo[i].frame_num} "
+ f"Frame latency = {cNvDsFrameLatencyInfo[i].latency} (ms) ")
+ batch_num += 1
+ return Gst.PadProbeReturn.OK
+
 # pgie_src_pad_buffer_probe will extract metadata received on tiler sink pad
 # and update params for drawing rectangle, object information etc.
 def pgie_src_pad_buffer_probe(pad,info,u_data):
@@ -199,7 +242,7 @@ def create_source_bin(index,uri):
         return None
     return nbin
 
-def main(args, requested_pgie=None, config=None, disable_probe=False):
+def main(args, requested_pgie=None, config=None, disable_probe=True):
     global perf_data
     perf_data = PERF_DATA(len(args))
 
@@ -380,6 +423,12 @@ def main(args, requested_pgie=None, config=None, disable_probe=False):
             # perf callback function to print fps every 5 sec
             GLib.timeout_add(5000, perf_data.perf_print_callback)
 
+ osd_src_pad=nvosd.get_static_pad("src")
+ if not osd_src_pad:
+ sys.stderr.write(" Unable to get src pad \n")
+ else:
+ osd_src_pad.add_probe(Gst.PadProbeType.BUFFER, osd_src_pad_buffer_probe, number_sources)
+
     # List the sources
     print("Now playing...")
     for i, source in enumerate(args):

```

1. Since DeepStream7.0(deepstream\_python\_apps v1.1.11), you only need to apply the following patch and set the env

export NVDS\_ENABLE\_COMPONENT\_LATENCY\_MEASUREMENT=1  
export NVDS\_ENABLE\_LATENCY\_MEASUREMENT=1

```auto
measure_latency = False #global variable
def buffer_probe(pad,info,u_data):
......
    global measure_latency
    if measure_latency:
        num_sources_in_batch = pyds.nvds_measure_buffer_latency(hash(gst_buffer))
        if num_sources_in_batch == 0:
            print("Unable to get number of sources in GstBuffer for latency measurement")
......

```

---

<div class="post-metadata">

**Author:** ![bcao](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/bcao/32/14043_2.png) [@bcao](https://forums.developer.nvidia.com/u/bcao)\
**Post date:** [November 25, 2020, 9:33am UTC](https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/13 "2020-11-25T09:33:52Z")

</div>

**8. [DS 5.0GA\_All\_App] Enable Perf measurement(FPS) for deepstream sample apps**

1. If you are using deepstream-app, you can add `enable-perf-measurement=1` under Application Group in the config file
2. If you are using other deepstream sample apps such as deepstream-test2, you can apply following patch to enable it

```patch
diff --git a/sources/apps/sample_apps/deepstream-test2/deepstream_test2_app.c b/sources/apps/sample_apps/deepstream-test2/deepstream_test2_app.c
index a2231acf535b4826adb766ed28f3aa80294c7f82..e37d7504ed07c9db77e5d3cdac2c4943fd0d1010 100755
--- a/sources/apps/sample_apps/deepstream-test2/deepstream_test2_app.c
+++ b/sources/apps/sample_apps/deepstream-test2/deepstream_test2_app.c
@@ -28,6 +28,7 @@
 #include <string.h>
 
 #include "gstnvdsmeta.h"
+#include "deepstream_perf.h"
 
 #define PGIE_CONFIG_FILE "dstest2_pgie_config.txt"
 #define SGIE1_CONFIG_FILE "dstest2_sgie1_config.txt"
@@ -51,6 +52,29 @@
  * based on the fastest source's framerate. */
 #define MUXER_BATCH_TIMEOUT_USEC 40000
 
+#define MAX_STREAMS 64
+
+typedef struct
+{
+ /** identifies the stream ID */
+ guint32 stream_index;
+ gdouble fps[MAX_STREAMS];
+ gdouble fps_avg[MAX_STREAMS];
+ guint32 num_instances;
+ guint header_print_cnt;
+ GMutex fps_lock;
+ gpointer context;
+
+ /** Test specific info */
+ guint32 set_batch_size;
+}DemoPerfCtx;
+
+
+typedef struct {
+ GMutex *lock;
+ int num_sources;
+}LatencyCtx;
+
 gint frame_number = 0;
 /* These are the strings of the labels for the respective models */
 gchar sgie1_classes_str[12][32] = { "black", "blue", "brown", "gold", "green",
@@ -80,6 +104,66 @@ guint sgie1_unique_id = 2;
 guint sgie2_unique_id = 3;
 guint sgie3_unique_id = 4;
 
+/**
+ * callback function to print the performance numbers of each stream.
+ */
+static void
+perf_cb (gpointer context, NvDsAppPerfStruct * str)
+{
+ DemoPerfCtx *thCtx = (DemoPerfCtx *) context;
+
+ g_mutex_lock(&thCtx->fps_lock);
+ /** str->num_instances is == num_sources */
+ guint32 numf = str->num_instances;
+ guint32 i;
+
+ for (i = 0; i < numf; i++) {
+ thCtx->fps[i] = str->fps[i];
+ thCtx->fps_avg[i] = str->fps_avg[i];
+ }
+ thCtx->context = thCtx;
+ g_print ("**PERF: ");
+ for (i = 0; i < numf; i++) {
+ g_print ("%.2f (%.2f)\t", thCtx->fps[i], thCtx->fps_avg[i]);
+ }
+ g_print ("\n");
+ g_mutex_unlock(&thCtx->fps_lock);
+}
+
+/**
+ * callback function to print the latency of each component in the pipeline.
+ */
+
+static GstPadProbeReturn
+latency_measurement_buf_prob(GstPad * pad, GstPadProbeInfo * info, gpointer u_data)
+{
+ LatencyCtx *ctx = (LatencyCtx *) u_data;
+ static int batch_num = 0;
+ guint i = 0, num_sources_in_batch = 0;
+ if(nvds_enable_latency_measurement)
+ {
+ GstBuffer *buf = (GstBuffer *) info->data;
+ NvDsFrameLatencyInfo *latency_info = NULL;
+ g_mutex_lock (ctx->lock);
+ latency_info = (NvDsFrameLatencyInfo *)
+ calloc(1, ctx->num_sources * sizeof(NvDsFrameLatencyInfo));;
+ g_print("\n ************BATCH-NUM = %d************** \n",batch_num);
+ num_sources_in_batch = nvds_measure_buffer_latency(buf, latency_info);
+
+ for(i = 0; i < num_sources_in_batch; i++)
+ {
+ g_print("Source id = %d Frame_num = %d Frame latency = %lf (ms) \n",
+ latency_info[i].source_id,
+ latency_info[i].frame_num,
+ latency_info[i].latency);
+ }
+ g_mutex_unlock (ctx->lock);
+ batch_num++;
+ }
+
+ return GST_PAD_PROBE_OK;
+}
+
 /* This is the buffer probe function that we have registered on the sink pad
  * of the OSD element. All the infer elements in the pipeline shall attach
  * their metadata to the GstBuffer, here we will iterate & process the metadata
@@ -144,9 +228,9 @@ osd_sink_pad_buffer_probe (GstPad * pad, GstPadProbeInfo * info,
         nvds_add_display_meta_to_frame(frame_meta, display_meta);
     }
 
- g_print ("Frame Number = %d Number of objects = %d "
- "Vehicle Count = %d Person Count = %d\n",
- frame_number, num_rects, vehicle_count, person_count);
+ // g_print ("Frame Number = %d Number of objects = %d "
+ // "Vehicle Count = %d Person Count = %d\n",
+ // frame_number, num_rects, vehicle_count, person_count);
     frame_number++;
     return GST_PAD_PROBE_OK;
 }
@@ -586,6 +670,30 @@ main (int argc, char *argv[])
     gst_pad_add_probe (osd_sink_pad, GST_PAD_PROBE_TYPE_BUFFER,
         osd_sink_pad_buffer_probe, NULL, NULL);
 
+ GstPad *sink_pad = gst_element_get_static_pad (nvvidconv1, "src");
+ if (!sink_pad)
+ g_print ("Unable to get sink pad\n");
+ else {
+ LatencyCtx *ctx = (LatencyCtx *)g_malloc0(sizeof(LatencyCtx));
+ ctx->lock = (GMutex *)g_malloc0(sizeof(GMutex));
+ ctx->num_sources = argc - 2;
+ gst_pad_add_probe (sink_pad, GST_PAD_PROBE_TYPE_BUFFER,
+ latency_measurement_buf_prob, ctx, NULL);
+ }
+ gst_object_unref (sink_pad);
+
+ GstPad *tiler_pad = gst_element_get_static_pad (nvtiler, "sink");
+ if (!tiler_pad)
+ g_print ("Unable to get tiler_pad pad\n");
+ else {
+ NvDsAppPerfStructInt *str = (NvDsAppPerfStructInt *)g_malloc0(sizeof(NvDsAppPerfStructInt));
+ DemoPerfCtx *perf_ctx = (DemoPerfCtx *)g_malloc0(sizeof(DemoPerfCtx));
+ g_mutex_init(&perf_ctx->fps_lock);
+ str->context = perf_ctx;
+ enable_perf_measurement (str, tiler_pad, argc-2, 1, 0, perf_cb);
+ }
+ gst_object_unref (tiler_pad);
+
   /* Set the pipeline to "playing" state */
   g_print ("Now playing: %s\n", argv[1]);
   gst_element_set_state (pipeline, GST_STATE_PLAYING);

```

---

<div class="post-metadata">

**Author:** ![mchi](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/mchi/32/14043_2.png) [@mchi](https://forums.developer.nvidia.com/u/mchi)\
**Post date:** [November 25, 2020, 1:59pm UTC](https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/14 "2020-11-25T13:59:31Z")

</div>

**9. [DS 5.0GA\_Jetson\_App] Capture HW & SW Memory Leak log**

1.Download attachment on to Jetson device and rename to nvmemstat.py  
2.Install “lsof” tool

```bash
   $ sudo apt-get install lsof

```

3.Run your application on Jetson in one terminal or background  
4.Run this script with command :

```bash
 # replace PROGRAM_NAME to application name in step 2 
 # this script will monitor the hardware memory, SW memory, etc.
 $ sudo python3 nvmemstat.py -p PROGRAM_NAME   

```

5.Share the log on the topic for further triage

[nvmemstat.py.txt](https://forums.developer.nvidia.com/uploads/short-url/fSFkRXc8hUyqOBEutCbDFmgmKND.txt) (4.7 KB)

---

<div class="post-metadata">

**Author:** ![Fiona.Chen](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/fiona.chen/32/15508_2.png) [@Fiona.Chen](https://forums.developer.nvidia.com/u/Fiona.Chen)\
**Post date:** [December 23, 2020, 8:05am UTC](https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/15 "2020-12-23T08:05:29Z")

</div>

**10. [ALL\_Jetson\_plugin] Jetson GStreamer Plugins Using with DeepStream**

For the user of Jetson DeepStream (JetPack), there are some accelerated gstreamer plugins which is hardware accelerated by Jetson but are not listed in DeepStream plugin list [GStreamer Plugin Overview — DeepStream documentation](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_Intro.html#gstreamer-plugin-overview).

Some of these plugins can be used in the DeepStream pipeline to extend the DeepStream functions while some of them are not compatible to DeepStreamSDK.

The basic document for the Gstreamer accelerated plugins is [Multimedia — Jetson Linux  
Developer Guide 34.1 documentation (nvidia.com)](https://docs.nvidia.com/jetson/archives/r35.1/DeveloperGuide/text/SD/Multimedia.html)

_ **DeepStream compatible plugins:** _

- nvegltransform: NvEGLTransform

Typical usage:

`gst-launch-1.0 uridecodebin uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! m.sink_0 nvstreammux name=m batch-size=1 width=1280 height=720 ! nvinfer config-file-path=/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt ! nvtracker tracker-width=640 tracker-height=480 ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so ll-config-file=config_tracker_NvDCF_perf.yml enable-batch-process=1 ! nvvideoconvert ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvmultistreamtiler ! nvdsosd ! nvvideoconvert ! nvegltransform ! nveglglessink`

- nvarguscamerasrc: nvarguscamerasrc: NvArgusCameraSrc

Typical usage:

`gst-launch-1.0 nvarguscamerasrc bufapi-version=true sensor-id=0 ! 'video/x-raw(memory:NVMM),width=640,height=480,framerate=30/1,format=NV12' ! m.sink_0 nvstreammux name=m batch-size=1 width=1280 height=720 ! nvinfer config-file-path=/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt ! nvtracker tracker-width=640 tracker-height=480 ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so ll-config-file=config_tracker_NvDCF_perf.yml enable-batch-process=1 ! nvvideoconvert ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvmultistreamtiler ! nvdsosd ! nvvideoconvert ! nvegltransform ! nveglglessink`

For DeepStream 6.2 GA, the pipeline should be

`gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=640,height=480,framerate=30/1,format=NV12' ! m.sink_0 nvstreammux name=m batch-size=1 width=1280 height=720 ! nvinfer config-file-path=/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt ! nvtracker tracker-width=640 tracker-height=480 ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so ll-config-file=config_tracker_NvDCF_perf.yml enable-batch-process=1 ! nvvideoconvert ! 'video/x-raw(memory:NVMM),format=RGBA' ! nvmultistreamtiler ! nvdsosd ! nvvideoconvert ! nvegltransform ! nveglglessink`

The related topic in forum:

[Segfault when `nvvideoconvert` and `nvv4l2h265enc` are used together - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums](https://forums.developer.nvidia.com/t/segfault-when-nvvideoconvert-and-nvv4l2h265enc-are-used-together/157107)

- nvv4l2camerasrc: nvv4l2camerasrc: NvV4l2CameraSrc

Typical usage:

`gst-launch-1.0 nvv4l2camerasrc device=/dev/video0 bufapi-version=1 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=60/1' ! nvvideoconvert ! 'video/x-raw(memory:NVMM),format=NV12' ! mx.sink_0 nvv4l2camerasrc device=/dev/video1 bufapi-version=1 ! 'video/x-raw(memory:NVMM),width=1920,height=1080,framerate=60/1' ! nvvideoconvert ! 'video/x-raw(memory:NVMM),format=NV12' ! mx.sink_1 nvstreammux width=1920 height=1080 batch-size=2 live-source=1 name=mx ! nvinfer config-file-path=/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt batch-size=2 ! nvvideoconvert ! nvmultistreamtiler width=1920 height=1080 rows=1 columns=2 ! nvvideoconvert ! nvdsosd ! nvegltransform ! nveglglessink sync=0`

The related topic in forum:  
[Low camera frame rate - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums](https://forums.developer.nvidia.com/t/low-camera-frame-rate/155306)

- nvdrmvideosink: Nvidia Drm Video Sink

Typical pipeline:  
`gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvideoconvert ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt ! nvdrmvideosink conn_id=0 plane_id=1 set_mode=0 -e`

The related topic in forum:  
[Which videosink for Jetson TX2 in EGLFS? - Jetson & Embedded Systems / Jetson TX2 - NVIDIA Developer Forums](https://forums.developer.nvidia.com/t/which-videosink-for-jetson-tx2-in-eglfs/160490)

- nv3dsink: Nvidia 3D sink

Typical pipeline:  
`gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvideoconvert ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt ! nv3dsink sync=false`

**Note: The nv3dsink plugin is a window-based rendering sink, and based on X11.**

- nvoverlaysink: OpenMax Video Sink

Typical pipeline:

`gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder bufapi-version=1 ! nvvideoconvert ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt ! nvoverlaysink sync=0`

**Note:The nvoverlaysink plugin is deprecated in L4T release 32.1. Please use nvdrmvideosink or nv3dsink for rendering gst-v4l2 decoder output.**

- nvjpegenc: JPEG image encoder

_ **DeepStream Incompatible Plugins** _

- nvvidconv: NvVidConv Plugin
- nvv4l2vp8enc: V4L2 VP8 Encoder
- nvv4l2vp9enc: V4L2 VP9 Encoder
- nveglstreamsrc: nVidia EGL Stream
- nvvideosink: nVidia Video Sink
- nvivafilter: nvivafilter: NvIVAFilter Plugin  
The related topic in forum:  
[Missing plugins from deepstream-l4t docker image - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums](https://forums.developer.nvidia.com/t/missing-plugins-from-deepstream-l4t-docker-image/147257)
- nvtee: NvTee
- videocuda: CUDA Post processor
- nvcompositor: NvCompositor

The DeepStream incompatible plugins are **NOT** guaranteed to be used with DeepStream pipeline.

Typical pipeline:  
`gst-launch-1.0 nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=960 sink_0::height=540 sink_1::xpos=960 sink_1::ypos=0 sink_1::width=960 sink_1::height=540 sink_2::xpos=0 sink_2::ypos=540 sink_2::width=1920 sink_2::height=540 ! nvegltransform ! nveglglessink \ filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! comp. \ filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! comp. \ filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! comp. -e `

The related topic in forum:  
[How to Customize layout from Nvmultistream-tiler module from DeepStream - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums](https://forums.developer.nvidia.com/t/how-to-customize-layout-from-nvmultistream-tiler-module-from-deepstream/227056)

---

<div class="post-metadata">

**Author:** ![mchi](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/mchi/32/14043_2.png) [@mchi](https://forums.developer.nvidia.com/u/mchi)\
**Post date:** [April 27, 2021, 4:43am UTC](https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/17 "2021-04-27T04:43:32Z")

</div>

**12. [DSx\_Jetson\_App]: Dump NV12 NvBufSurface into a YUV file**  
Each NV12 NvBufSurface includes two semi-planes which are not continuous in memory.  
[gstnvinfer\_dump\_NV12\_NvBufSurface.patch](https://forums.developer.nvidia.com/uploads/short-url/balmnC0fvgKhWs2dJq4d0h3hsxo.patch) (4.9 KB)

This is a sample change to /opt/nvidia/deepstream/deepstream-5.1/sources/gst-plugins/gst-nvinfer/gstnvinfer.cpp to dump the NV12 NvBufSurface before transforming to RGB data.  
After getting the YUV file, we can view it with [GitHub - IENT/YUView: The Free and Open Source Cross Platform YUV Viewer with an advanced analytics toolset](https://github.com/IENT/YUView)

Note: Also verified with DeepStream 6.2 on Jetson

---

<div class="post-metadata">

**Author:** ![bcao](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/bcao/32/14043_2.png) [@bcao](https://forums.developer.nvidia.com/u/bcao)\
**Post date:** [April 30, 2021, 2:01am UTC](https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/18 "2021-04-30T02:01:32Z")

</div>

**13. [DS 5.x\_All\_App] How to access and modify the NvBufSurface**

Refer [Deepstream sample code snippet - #3 by bcao](https://forums.developer.nvidia.com/t/deepstream-sample-code-snippet/142683/3)

---

<div class="post-metadata">

**Author:** ![kesong](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/kesong/32/101255_2.png) [@kesong](https://forums.developer.nvidia.com/u/kesong)\
**Post date:** [September 1, 2021, 6:53am UTC](https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/19 "2021-09-01T06:53:38Z")

</div>

**14. [All\_Jetson\_App] Check memory leakage with valgrind**

1. Install valgrind with below command  
$ sudo apt-get install valgrind valgrind-dbg
2. Run application with below command  
$ valgrind --tool=memcheck --leak-check=full --num-callers=100 --show-leak-kinds=definite,indirect --track-origins=yes ./app

---

<div class="post-metadata">

**Author:** ![mchi](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/mchi/32/14043_2.png) [@mchi](https://forums.developer.nvidia.com/u/mchi)\
**Post date:** [December 4, 2021, 3:37pm UTC](https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/21 "2021-12-04T15:37:39Z")

</div>

**15.[DSx\_All\_App] Debug Tips for DeepStream Accuracy Issue**

**Ensure the image pre-process before inference aligns with the training pre-process.**  
**15.1** Confirm your model has got good accuracy in training and inference outside DeepStream  
**15.2**  **nvinfer**  
When deploying a **ONNX** model to DeepStream with nvinfer plugin, confirm below nvinfer parameters are set correctly _ **to align with the corresponding settings in Training** _  
**15.2.1 Input scale & offset**  
1). net-scale-factor =  
2). offsets  
The usage of these two parameters are as below (from [doc](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html#gst-nvinfer))

```
   ![image|690x139](upload://3OdGSGK4Lniqp4VqHVy92yPVPSY.png)

```

**15.2.2 Input Order**  
1). network-input-order= // 0:NCHW 1:NHWC  
2). infer-dims= // if network-input-order=1, i.e. NHWC, infer-dims must be specified, otherwise, nvinfer can’t detect input dims automatically  
3). model-color-format= // 0: RGB 1: BGR 2: GRAY  
**15.2.3 scale and padding**  
1). maintain-aspect-ratio= // whether to maintain aspect ratio while scaling input  
2). symmetric-padding= // whether to pad image symmetrically while scaling input. By defaulut, it’s asymmetrical padding and the image will be scaled to top left corner.  
**15.2.4 inference precision**  
1). network-mode= // 0: FP32 1: INT8 2: FP16. If INT8 accuracy is not good, try FP16 or FP32  
**15.2.5 threshold**  
1). threshold=  
2). pre-cluster-threshold=  
3). Post-cluster-threshold=  
Above are some highlighted parameters for a quick check for accuracy. For more detailed informantion, please refer to nvinfer doc - [Gst-nvinfer — DeepStream documentation](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html#gst-nvinfer)  
**15.2.6** avoid missing the object that close to the border of the image(Version 6.2 and above)  
1). crop-objects-to-roi-boundary=1  
**15.3 Dump the input or output of the nvinfer**  
Below two items in [DeepStream SDK FAQ - #9 by mchi](https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/9)  
2. [DS5.0GA\_Jetson\_dGPU\_Plugin] Dump the Inference Input ==\> compare the input between DS and your own standalone inference/training app  
3. [DS5\_Jetson\_dGPU\_Plugin] Dump the Inference outputs ==\> then apply your own parser offline check this output data  
**15.4 Try to remove/replace the plugin with extra conversion.**  
**15.4.1** The following two pipelines have the same result before Gst-nvstreammux plugin, you can choose the second one to reduce extra conversions caused by videoconvert and Gst-nvvideoconvert.  
` multifilesrc->jpegdec->videoconvert->nvvideoconvert->nvstreammux->nvinfer… multifilesrc->nvjpegdec->nvstreammux->nvinfer…… `  
**15.4.2** If you use Gst-nvstreammux, please set the width and height paremeters to align with the video.  
We suggest you try to use the Gst-nvstreammux New instead of Gst-nvstreammux especially when the width or height of video is not multiples of 4. If you need to use Gst-nvstreammux New, please ensure that all sources have the same resolution.

---

<div class="post-metadata">

**Author:** ![mchi](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/mchi/32/14043_2.png) [@mchi](https://forums.developer.nvidia.com/u/mchi)\
**Post date:** [December 6, 2021, 10:40am UTC](https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/22 "2021-12-06T10:40:51Z")

</div>

**16. [DeepStream 6.0 GA] python binding installation**

Download the wheel files directly from [Releases · NVIDIA-AI-IOT/deepstream\_python\_apps · GitHub](https://github.com/NVIDIA-AI-IOT/deepstream_python_apps/releases)

[Next page](https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236.md?page=2)