How do you change the open souce code? And could you attach the log with GST_DEBUG=3?
I tried changing the open source code ( deepstream_source_bin.cpp) same as done in this postā> How to change source pixel format from YUYV to MJPG - #7 by jhchris713
However, I got the below log again when I use GST_DEBUG=3. Camera starts after 1 minute and between each new frame, nearly 3000 frames are getting lost!!! (Please see the messages in the log). Is there a buffer problem?
I have the same problem when running all the TAO Models with the 2 USB cameras I have (One of them is Logitech Brio, the other one is Microsoft Kinect). I am using the latest jetpack version with Jetson AGX Orin.
Also note that this problem does not occur if I run the deepstream-app directly with the usb cameras : (deepstream-app -c source1_usb_dec_infer_resnet_int8.txt)
Were there any patches to solve this issue?
GST_DEBUG=3 ./deepstream-gaze-app 3 ../../../configs/facial_tao/sample_faciallandmarks_config.txt v4l2:///dev/video0 ./gazenet
Request sink_0 pad from streammux
Now playing: v4l2:///dev/video0
Using winsys: x11
Inside Custom Lib : Setting Prop Key=config-file Value=../../../configs/gaze_tao/sample_gazenet_model_config.txt
0:00:02.127890021 648337 0xaaaabb1bd200 INFO nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<second-infer-engine1> NvDsInferContext[UID 2]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1909> [UID = 2]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.2/sources/deepstream_tao_apps/models/faciallandmark/faciallandmarks.etlt_b32_gpu0_int8.engine
INFO: [FullDims Engine Info]: layers num: 4
0 INPUT kFLOAT input_face_images 1x80x80 min: 1x1x80x80 opt: 32x1x80x80 Max: 32x1x80x80
1 OUTPUT kFLOAT conv_keypoints_m80 80x80x80 min: 0 opt: 0 Max: 0
2 OUTPUT kFLOAT softargmax 80x2 min: 0 opt: 0 Max: 0
3 OUTPUT kFLOAT softargmax:1 80 min: 0 opt: 0 Max: 0
ERROR: [TRT]: 3: Cannot find binding of given name: softargmax,softargmax:1,conv_keypoints_m80
0:00:02.280710142 648337 0xaaaabb1bd200 WARN nvinfer gstnvinfer.cpp:677:gst_nvinfer_logger:<second-infer-engine1> NvDsInferContext[UID 2]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:1876> [UID = 2]: Could not find output layer 'softargmax,softargmax:1,conv_keypoints_m80' in engine
0:00:02.280762337 648337 0xaaaabb1bd200 INFO nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<second-infer-engine1> NvDsInferContext[UID 2]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2012> [UID = 2]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.2/sources/deepstream_tao_apps/models/faciallandmark/faciallandmarks.etlt_b32_gpu0_int8.engine
0:00:02.495921030 648337 0xaaaabb1bd200 INFO nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<second-infer-engine1> [UID 2]: Load new model:../../../configs/facial_tao/faciallandmark_sgie_config.txt sucessfully
0:00:02.496122580 648337 0xaaaabb1bd200 WARN nvinfer gstnvinfer.cpp:677:gst_nvinfer_logger:<primary-infer-engine1> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1170> [UID = 1]: Warning, OpenCV has been deprecated. Using NMS for clustering instead of cv::groupRectangles with topK = 20 and NMS Threshold = 0.5
0:00:03.951602405 648337 0xaaaabb1bd200 INFO nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-infer-engine1> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1909> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.2/sources/deepstream_tao_apps/models/faciallandmark/facenet.etlt_b1_gpu0_int8.engine
INFO: [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT input_1 3x416x736
1 OUTPUT kFLOAT output_bbox/BiasAdd 4x26x46
2 OUTPUT kFLOAT output_cov/Sigmoid 1x26x46
0:00:04.107970860 648337 0xaaaabb1bd200 INFO nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-infer-engine1> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2012> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.2/sources/deepstream_tao_apps/models/faciallandmark/facenet.etlt_b1_gpu0_int8.engine
0:00:04.111334697 648337 0xaaaabb1bd200 INFO nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary-infer-engine1> [UID 1]: Load new model:../../../configs/facial_tao/config_infer_primary_facenet.txt sucessfully
Decodebin child added: source
Decodebin child added: decodebin0
Running...
Decodebin child added: nvjpegdec0
0:00:04.668504186 648337 0xaaaad4b34a40 FIXME videodecoder gstvideodecoder.c:946:gst_video_decoder_drain_out:<nvjpegdec0> Sub-class should implement drain()
0:00:04.668636771 648337 0xaaaad4b34a40 FIXME videodecoder gstvideodecoder.c:946:gst_video_decoder_drain_out:<nvjpegdec0> Sub-class should implement drain()
0:00:04.688270444 648337 0xaaaad4b34a40 WARN v4l2bufferpool gstv4l2bufferpool.c:809:gst_v4l2_buffer_pool_start:<source:pool:src> Uncertain or not enough buffers, enabling copy threshold
In cb_newpad
###Decodebin pick nvidia decoder plugin.
Deserializing engine from: ./gazeinfer_impl/../../../../models/gazenet/gazenet_facegrid.etlt_b8_gpu0_fp16.engineThe logger passed into createInferRuntime differs from one already provided for an existing builder, runtime, or refitter. Uses of the global logger, returned by nvinfer1::getLogger(), will return the existing value.
Loaded engine size: 9 MiB
Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
Deserialization required 16830 microseconds.
[MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +9, now: CPU 0, GPU 153 (MiB)
Total per-runner device persistent memory is 56832
Total per-runner host persistent memory is 109056
Allocated activation device memory of size 22133248
[MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +21, now: CPU 0, GPU 174 (MiB)
0:01:49.222172741 648337 0xaaaad4b34a40 WARN v4l2src gstv4l2src.c:914:gst_v4l2src_create:<source> Timestamp does not correlate with any clock, ignoring driver timestamps
Frame Number = 0 Face Count = 0
Frame Number = 1 Face Count = 0
Frame Number = 2 Face Count = 0
0:06:46.434095088 648337 0xaaaad4b34a40 WARN v4l2src gstv4l2src.c:978:gst_v4l2src_create:<source> lost frames detected: count = 3013 - ts: 0:06:41.802925254
Frame Number = 3 Face Count = 0
0:08:28.993810703 648337 0xaaaad4b34a40 WARN v4l2src gstv4l2src.c:978:gst_v4l2src_create:<source> lost frames detected: count = 2960 - ts: 0:08:24.362643141
Frame Number = 4 Face Count = 0
0:10:09.927904656 648337 0xaaaad4b34a40 WARN v4l2src gstv4l2src.c:978:gst_v4l2src_create:<source> lost frames detected: count = 2906 - ts: 0:10:05.296740006
Frame Number = 5 Face Count = 0
0:11:49.180885204 648337 0xaaaad4b34a40 WARN v4l2src gstv4l2src.c:978:gst_v4l2src_create:<source> lost frames detected: count = 2875 - ts: 0:11:44.549713226
Frame Number = 6 Face Count = 0
0:13:27.290887861 648337 0xaaaad4b34a40 WARN v4l2src gstv4l2src.c:978:gst_v4l2src_create:<source> lost frames detected: count = 3029 - ts: 0:13:22.659718347
Frame Number = 7 Face Count = 0
0:15:10.057218118 648337 0xaaaad4b34a40 WARN v4l2src gstv4l2src.c:978:gst_v4l2src_create:<source> lost frames detected: count = 2988 - ts: 0:15:05.426038588
Frame Number = 8 Face Count = 0
0:16:51.594950636 648337 0xaaaad4b34a40 WARN v4l2src gstv4l2src.c:978:gst_v4l2src_create:<source> lost frames detected: count = 2940 - ts: 0:16:46.963780322
Frame Number = 9 Face Count = 0
0:18:30.829688863 648337 0xaaaad4b34a40 WARN v4l2src gstv4l2src.c:978:gst_v4l2src_create:<source> lost frames detected: count = 2909 - ts: 0:18:26.198518421
Frame Number = 10 Face Count = 0
Itās only for deepstream-app, not for other apps,like deepstream-gaze-app.You should add patches to the source code, like: deepstream_gaze_app.cpp
2. Could you test the latency in your env?
https://forums.developer.nvidia.com/t/deepstream-sdk-faq/80236/12
This latency test is for deepstream-app and in my case I have no problem running the deesptream-app . Both of my USB cameras work perfectly with deesptream-app and the latency is very low. See the log below:
************BATCH-NUM = 169**************
Comp name = nvstreammux-src_bin_muxer source_id = 0 pad_index = 0 frame_num = 0 in_system_timestamp = 1679142637834.819092 out_system_timestamp = 1679142637834.877930 component_latency = 0.058838
Comp name = primary_gie in_system_timestamp = 1679142637834.927002 out_system_timestamp = 1679142637838.049072 component latency= 3.122070
Comp name = tiled_display_tiler in_system_timestamp = 1679142637838.095947 out_system_timestamp = 1679142637844.960938 component latency= 6.864990
Comp name = osd_conv in_system_timestamp = 1679142637845.044922 out_system_timestamp = 1679142637847.385010 component latency= 2.340088
Comp name = nvosd0 in_system_timestamp = 1679142637847.434082 out_system_timestamp = 1679142637848.361084 component latency= 0.927002
Source id = 0 Frame_num = 0 Frame latency = 1679142637848.440918 (ms)
I tested all the "deepstream tao apps "with the 2 USB cameras and itās the same problem as before. How do I do latency test with the deepstream-tao apps? Is there a way to replicate the problem on someoneās board? (Jetson AGX Orin+ Logitech Brio with the latest jetpack)
I donāt know the root cause of this problem is so I donāt know what to patch.
I tried the below pipeline as an example and it worked for my USB cameras. How do I incorporate a similar pipeline?
GST_DEBUG=3 gst-launch-1.0 v4l2src device=/dev/video0 ! image/jpeg,width=1920,height=1080,framerate=30/1 ! jpegparse ! jpegdec ! autovideosink sync=false
The second item from the link is how to add latency to a separate app.
2.If you are using other deepstream sample apps such as deepstream-test3, you need to apply the following patch and set the env
You can refer to our open source demo: sources\apps\sample_apps\deepstream-test1\deepstream_test1_app.c
You can customize it according to your needs, like change the source to v4l2 sourceā¦:
Just for example:
source = gst_element_factory_make ("v4l2src", "file-source");
caps_filter = gst_element_factory_make ("capsfilter", NULL);
caps = gst_caps_new_simple ("image/jpeg",
"width", G_TYPE_INT, 1920, "height", G_TYPE_INT,
1080, "framerate", GST_TYPE_FRACTION,
30, 1, NULL);
g_object_set (G_OBJECT (caps_filter ), "caps", caps, NULL);
jpegparser = gst_element_factory_make ("jpegparse", "jpeg-parser");
decoder= gst_element_factory_make ("nvjpegdec", "jpeg-decoder");
The latency patch worked with deepstream-test3 as below, however the latency patch does not work for deepstream-gaze-app (I applied the same patch but it gave errors). There is obviously latency per the previous logs.
************BATCH-NUM = 89**************
Comp name = nvv4l2decoder0 in_system_timestamp = 1679314790955.870117 out_system_timestamp = 1679314790989.819092 component latency= 33.948975
Comp name = nvstreammux-stream-muxer source_id = 0 pad_index = 0 frame_num = 89 in_system_timestamp = 1679314790989.862061 out_system_timestamp = 1679314791088.554932 component_latency = 98.692871
Comp name = primary-nvinference-engine in_system_timestamp = 1679314791088.610107 out_system_timestamp = 1679314791094.482910 component latency= 5.872803
Comp name = nvtiler in_system_timestamp = 1679314791212.158936 out_system_timestamp = 1679314791221.554932 component latency= 9.395996
Comp name = nvvideo-converter in_system_timestamp = 1679314791342.839111 out_system_timestamp = 1679314791345.239990 component latency= 2.400879
Comp name = nv-onscreendisplay in_system_timestamp = 1679314791345.354004 out_system_timestamp = 1679314791345.887939 component latency= 0.533936
Source id = 0 Frame_num = 89 Frame latency = 390.077881 (ms)
I did some customization for the caps as suggested in the previous post, and I got the below. Again the display shows on the screen but it is extremely slow with~3000 frames lost in between each display frame update. There is a lot of latency. The log is slightly different than the previous logs (with GST-DEBUG=3) however still could not draw any conclusion.
I tested logitech 720p and it works well, but my application requires a high resolution usb cam. Is there any plan to create a patch to solve this problem since I believe other people will run into the same problem.
Decodebin child added: source
Decodebin child added: decodebin0
Running...
Decodebin child added: nvjpegdec0
0:00:04.995885101 16221 0xaaab2aa3ef00 FIXME videodecoder gstvideodecoder.c:946:gst_video_decoder_drain_out:<nvjpegdec0> Sub-class should implement drain()
0:00:04.995998319 16221 0xaaab2aa3ef00 FIXME videodecoder gstvideodecoder.c:946:gst_video_decoder_drain_out:<nvjpegdec0> Sub-class should implement drain()
0:00:05.016596539 16221 0xaaab2aa3ef00 WARN v4l2bufferpool gstv4l2bufferpool.c:809:gst_v4l2_buffer_pool_start:<source:pool:src> Uncertain or not enough buffers, enabling copy threshold
In cb_newpad
###Decodebin pick nvidia decoder plugin.
0:01:41.616084110 16221 0xaaab2aa3ef00 WARN v4l2src gstv4l2src.c:914:gst_v4l2src_create:<source> Timestamp does not correlate with any clock, ignoring driver timestamps
Deserializing engine from: ./gazeinfer_impl/../../../../models/gazenet/gazenet_facegrid.etlt_b8_gpu0_fp16.engineThe logger passed into createInferRuntime differs from one already provided for an existing builder, runtime, or refitter. Uses of the global logger, returned by nvinfer1::getLogger(), will return the existing value.
Loaded engine size: 9 MiB
Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
Keep the original bbox
Deserialization required 464917 microseconds.
[MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +9, now: CPU 0, GPU 153 (MiB)
Total per-runner device persistent memory is 56832
Total per-runner host persistent memory is 109056
Allocated activation device memory of size 22133248
[MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +21, now: CPU 0, GPU 174 (MiB)
Gaze: 258.006195 -143.412491 -74.006882 0.045197 -0.100402
Frame Number = 0 Face Count = 1
Keep the original bbox
Gaze: 271.582672 -125.255020 -79.581390 0.019716 -0.123271
Frame Number = 1 Face Count = 1
Keep the original bbox
Gaze: 281.816040 -115.070984 -172.622025 0.030466 -0.111039
Frame Number = 2 Face Count = 1
0:06:19.170011289 16221 0xaaab2aa3ef00 WARN v4l2src gstv4l2src.c:978:gst_v4l2src_create:<source> lost frames detected: count = 2793 - ts: 0:06:14.156461051
Keep the original bbox
Gaze: 278.370544 -124.234871 -180.674515 0.034349 -0.092809
Frame Number = 3 Face Count = 1
Could you just use file source, like 1080P video, to see the frame rateļ¼ We can confirm whether it is your cameraās problem firstly.