How to improve the performance of pipelines in deepstream

[LFYTMLY]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)：Jetson
• DeepStream Version：DeepStream-6.2
• JetPack Version (valid for Jetson only)：5.1.4

I am currently using the YOLOv5S-INT8 model, and the throughput of the trtexec test model is approximately 120, the approximate pure inference time of deepstream is shown below through the Nsight system：

ExecutionContext::enqueue
Begins: 12.6954s
Ends: 12.6991s (+3.643 ms)
Thread: 1811957

It took 3.643ms, which also indicates that the model should not have any major problems.

But I called the performance analysis code in the example in my code, and then output：

************BATCH-NUM = 346**************
Comp name = nvv4l2decoder0 in_system_timestamp = 1758077341101.964111 out_system_timestamp = 1758077341104.131104               component latency= 2.166992
Comp name = nvstreammux-streammux_0 source_id = 0 pad_index = 0 frame_num = 346               in_system_timestamp = 1758077341104.302002 out_system_timestamp = 1758077341104.437012               component_latency = 0.135010
Comp name = nvinfer0 in_system_timestamp = 1758077341104.500977 out_system_timestamp = 1758077341120.623047               component latency= 16.122070
Comp name = nvvideoconvert0 in_system_timestamp = 1758077341120.761963 out_system_timestamp = 1758077341125.799072               component latency= 5.037109
Source id = 0 Frame_num = 346 Frame latency = 24.025879 (ms)

At frame 346, the delay of the entire pipeline is 24ms, of which nvinfer took 16ms.

When I view the pure inference time of the current frame in the Nsight system:

When I viewed the pure inference time of the current frame in the Nsight system as 3.699ms, the post-processing I used was GitHub - marcoslucianops/DeepStream-Yolo: NVIDIA DeepStream SDK 7.1 / 7.0 / 6.4 / 6.3 / 6.2 / 6.1.1 / 6.1 / 6.0.1 / 6.0 / 5.1 implementation for YOLO models Post processing, and I printed it for about 0.8ms

I want to ask what Nvinfer is doing at other times? How can I improve the performance of the pipeline?

My Pipeline is nvurisrcbin - nvstreammux - nvinfer - nvvideoconvert -fakesink

[LFYTMLY]

my nsys_report

nsys_report.nsys-rep

[Fiona.Chen]

Did you get the nsys report with the DeepStream-Yolo/deepstream_app_config.txt at master · marcoslucianops/DeepStream-Yolo (Nothing changed?)?

[LFYTMLY]

@Fiona.Chen

I just used his post-processing code, and the rep report was generated in my custom code. I did not modify his code.

[LFYTMLY]

@Fiona.Chen

I am not sure where my performance bottleneck is now. If it is my YOLOv5S INT model, the performance can only be like this. I will consider replacing the model. If there is a high latency in the component, could you please help me analyze where the problem lies?

[Fiona.Chen]

So what is the pipeline and configurations? The element latency will be impacted by the upstream and downstream elements in the pipeline.

[Fiona.Chen]

@LFYTMLY

I am not sure where my performance bottleneck is now. If it is my YOLOv5S INT model, the performance can only be like this.

Why do you think the performance is not it should be?

[LFYTMLY]

@Fiona.Chen

Because I looked at a single model with an RTSP stream, and the total time per frame was about 25 milliseconds, which means that my model can only detect about 40 frames per second. In this case, I don’t think it has fully utilized its performance. My device is a Jetson Orin Nano 4GB

[Fiona.Chen]

@LFYTMLY

Because I looked at a single model with an RTSP stream, and the total time per frame was about 25 milliseconds, which means that my model can only detect about 40 frames per second. In this case, I don’t think it has fully utilized its performance.

Do you mean your RTSP stream has 40 FPS framerate and you want the pipeline can inference the frame in time?

[Fiona.Chen]

Please pay attention to the difference between “interval” and “latency”. DeepStream pipeline is based on GStreamer which is an asynchronous architecture, the elements work asynchronously. In such a system, the frame latency just means the time used for a frame to pass through all the elements in the system. But the interval means the time between two succeeded frames, it has nothing to do with frame latency, the succeeded frame can goes into the pipeline at any time and it does not have to wait for the previous frame to reach to the sink element(the end consumer of the pipeline). The framerate is calculated by the interval but not by latency.

So what is your concern about the trtexec result of yolov5s model and the latency data you’ve got from your pipeline with RTSP input?

[LFYTMLY]

@Fiona.Chen

I may not have expressed myself clearly, but my current problem is that my model’s inference time is already within a relatively good range (stable, pure inference takes more than 10 milliseconds, and post-processing time is also at the level of 1 millisecond), but my pipeline frames still take a considerable amount of time from nvurisrcbin to fakesink, and it takes almost 30 milliseconds for nvinfer. I don’t know who has wasted the remaining time except for my post-processing and inference time? Synchronize? Or wait for the result?

[Fiona.Chen]

@LFYTMLY

my pipeline frames still take a considerable amount of time from nvurisrcbin to fakesink, and it takes almost 30 milliseconds for nvinfer.

How did you get such data?

[Fiona.Chen]

@LFYTMLY

my model’s inference time is already within a relatively good range (stable, pure inference takes more than 10 milliseconds, and post-processing time is also at the level of 1 millisecond)

How did you get these data?

[LFYTMLY]

According to the DeepStream SDK FAQ

DeepStream SDK FAQ

7. [DS 5.0GA_All_App] Enable Latency measurement for deepstream sample apps If you are using deepstream-app, to check the component latency directly, you need to set the env export NVDS_ENABLE_COMPONENT_LATENCY_MEASUREMENT=1 export NVDS_ENABLE_LATENCY_MEASUREMENT=1 If you are using other deepstream sample apps such as deepstream-test3, you need to apply the following patch and set the env export NVDS_ENABLE_COMPONENT_LATENCY_MEASUREMENT=1 export NVDS_ENABLE_LATENCY_MEASUREMENT=1 …

I have selected the outputs of three of the stages

************BATCH-NUM = 2**************
Comp name = nvv4l2decoder0 in_system_timestamp = 1758077330807.289062 out_system_timestamp = 1758077330808.569092               component latency= 1.280029
Comp name = nvstreammux-streammux_0 source_id = 0 pad_index = 0 frame_num = 2               in_system_timestamp = 1758077330808.654053 out_system_timestamp = 1758077330808.729980               component_latency = 0.075928
Comp name = nvinfer0 in_system_timestamp = 1758077330811.878906 out_system_timestamp = 1758077330873.504883               component latency= 61.625977
Comp name = nvvideoconvert0 in_system_timestamp = 1758077331097.657959 out_system_timestamp = 1758077331107.289062               component latency= 9.631104
Source id = 0 Frame_num = 2 Frame latency = 335.302979 (ms)

************BATCH-NUM = 97**************
Comp name = nvv4l2decoder0 in_system_timestamp = 1758077331856.710938 out_system_timestamp = 1758077331858.309082               component latency= 1.598145
Comp name = nvstreammux-streammux_0 source_id = 0 pad_index = 0 frame_num = 97               in_system_timestamp = 1758077331858.448975 out_system_timestamp = 1758077331858.531006               component_latency = 0.082031
Comp name = nvinfer0 in_system_timestamp = 1758077331871.187012 out_system_timestamp = 1758077331923.315918               component latency= 52.128906
Comp name = nvvideoconvert0 in_system_timestamp = 1758077331923.454102 out_system_timestamp = 1758077331926.813965               component latency= 3.359863
Source id = 0 Frame_num = 97 Frame latency = 70.300049 (ms)

************BATCH-NUM = 237**************
Comp name = nvv4l2decoder0 in_system_timestamp = 1758077336763.907959 out_system_timestamp = 1758077336766.124023               component latency= 2.216064
Comp name = nvstreammux-streammux_0 source_id = 0 pad_index = 0 frame_num = 237               in_system_timestamp = 1758077336766.280029 out_system_timestamp = 1758077336766.437988               component_latency = 0.157959
Comp name = nvinfer0 in_system_timestamp = 1758077336766.500977 out_system_timestamp = 1758077336781.346924               component latency= 14.845947
Comp name = nvvideoconvert0 in_system_timestamp = 1758077336781.513916 out_system_timestamp = 1758077336786.608887               component latency= 5.094971
Source id = 0 Frame_num = 237 Frame latency = 22.900146 (ms)

After stabilizing to over 100 frames, the overall pipeline takes about 25ms, but in the Nsight system, the average duration of ExecutionContext:: enqueue after 120 frames is 8.2ms.
Now explain how I obtained this post-processing time.

I used the DeepStream Yulo plugin from Marcellucianops in nvdsparsebbox_Yolo.cpp

static bool
NvDsInferParseCustomYolo(std::vector<NvDsInferLayerInfo> const &outputLayersInfo,
                         NvDsInferNetworkInfo const &networkInfo, NvDsInferParseDetectionParams const &detectionParams,
                         std::vector<NvDsInferParseObjectInfo> &objectList)
{
  auto t0 = std::chrono::high_resolution_clock::now();

  if (outputLayersInfo.empty())
  {
    std::cerr << "ERROR: Could not find output layer in bbox parsing" << std::endl;
    return false;
  }

  std::vector<NvDsInferParseObjectInfo> objects;

  const NvDsInferLayerInfo &output = outputLayersInfo[0];
  const uint outputSize = output.inferDims.d[0];

  std::vector<NvDsInferParseObjectInfo> outObjs = decodeTensorYolo((const float *)(output.buffer), outputSize,
                                                                   networkInfo.width, networkInfo.height, detectionParams.perClassPreclusterThreshold);

  objects.insert(objects.end(), outObjs.begin(), outObjs.end());

  objectList = objects;
  auto now2 = std::chrono::steady_clock::now();

  auto t1 = std::chrono::high_resolution_clock::now();

  double elapsed = std::chrono::duration<double, std::milli>(t1 - t0).count();

  std::cout << "[NvDsInferParseCustomYolo] Postprocessing time = " << elapsed << " ms" << std::endl;

  return true;
}

This is what I consider post-processing time

[Fiona.Chen]

Thank you for sharing the information. SO your only concern is about the single frame latency in the pipeline.

Are you using live stream (RTSP stream) to do the latency test?

[Fiona.Chen]

1. Can you tell us the pipeline and configurations of the elements when you enable the latency measurement? Is there any probe function you have applied to your pipeline? Are you testing with live streams but not local video files?

@LFYTMLY

I used the DeepStream Yulo plugin from Marcellucianops in nvdsparsebbox_Yolo.cpp

static bool
NvDsInferParseCustomYolo(std::vector<NvDsInferLayerInfo> const &outputLayersInfo,
                         NvDsInferNetworkInfo const &networkInfo, NvDsInferParseDetectionParams const &detectionParams,
                         std::vector<NvDsInferParseObjectInfo> &objectList)

This is not the whole post-processing time, this is just one frame postprocessing but not the batch postprocessing. Please refer to gst_nvinfer_output_loop() function inside /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvinfer/gstnvinfer.cpp, the nvtx has been added to measure postprocessing with Nsight. The correct way to get Nsight report is introduced in DeepStream SDK FAQ - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums

[Fiona.Chen]

From your previous nsight report, I saw 4 input live streams(maybe rtsp streams) and there are opencv operations inside the report.

[LFYTMLY]

@Fiona.Chen

I am now re using the deepstream-test3 demo to validate my model.

Why is there a long period of neutral between my two frames of inference? The main OS runtime libraries are ioctl operations and futex operations.

futex
Begins: 7.65511s
Ends: 7.70074s (+45.628 ms)

Call stack at 7.65511s:
libc-2.31.so!syscall
libglib-2.0.so.0.6400.6!g_cond_wait
libnvdsgst_infer.so!0xffff8814904c
libglib-2.0.so.0.6400.6!0xffff9469747c
Nsight Systems frames
libpthread-2.31.so!start_thread
libc-2.31.so!thread_start

ioctl
Begins: 7.64094s
Ends: 7.64559s (+4.648 ms)

Call stack at 7.64094s:
libc-2.31.so!__GI___ioctl
libnvrm_host1x.so!0xffff901b6cf0
libnvbufsurftransform.so.1.0.0!NvBufSurfTransformSyncObjWait
libnvdsgst_infer.so!0xffff8814928c
libglib-2.0.so.0.6400.6!0xffff9469747c
Nsight Systems frames
libpthread-2.31.so!start_thread
libc-2.31.so!thread_start

[Fiona.Chen]

@LFYTMLY

I am now re using the deepstream-test3 demo to validate my model.

Can you answer my questions directly? Are you using RTSP source when you got the nsight report? What is your configurations with deepsteam-test3 when you got the nsight report? Can you provide the complete nsys report but not pieces? Have you enabled the nvtx log in your nsight report?

I’ve mentioned in previous post that the element latency will be impacted by the upstream and downstream elements in the pipeline. So it is important to provide the details of the pipeline.

[LFYTMLY]

@Fiona.Chen

Thank you very much for your patient reply.

nvurisrcbin config:

g_object_set(G_OBJECT(uri_decode_bin), “file-loop”, TRUE, NULL);
g_object_set(G_OBJECT(uri_decode_bin), “cudadec-memtype”, 0, NULL);
g_object_set(G_OBJECT(uri_decode_bin), “uri”, uri, NULL);

streammux config

g_object_set(G_OBJECT(streammux), “batch-size”, num_sources, NULL);
g_object_set(G_OBJECT(streammux), “width”, 1920, “height”, 1080, NULL);
g_object_set(G_OBJECT(streammux), “batched-push-timeout”, 40000, NULL);
g_object_set(G_OBJECT(streammux), “nvbuf-memory-type”, 4, NULL);
g_object_set(G_OBJECT(streammux), “live-source”, TRUE, NULL);

nvinfer config

g_object_set(G_OBJECT(pgie), “config-file-path”,“/code/infer.txt”, NULL);
g_object_set(G_OBJECT(pgie), “batch-size”, num_sources, NULL);
g_object_set(G_OBJECT(pgie), “interval”, 0, NULL);

fakesink config

g_object_set(G_OBJECT(sink), “qos”, 0, NULL);

Component Link：

nvurisrcbin-nvstreammux-queue1-nvinfer-queue2-fakesink

Specific demo code:

test.c

The reps generated based on the knowledge in the FAQ are as follows:

nsys_my1.nsys-rep

As a newcomer to Deepstream, thank you for your timely response and help me solve the problem