I was running the pre-trained models using deepstream, but the input is video file by default . Is there a way to run the model and detect objects using usb cam in real-time, instead of outputting to the video file out.mp4?
/opt/nvidia/deepstream/deepstream-6.2/sources/deepstream_tao_apps$ sudo ./apps/tao_detection/ds-tao-detection -c configs/peopleSegNet_tao/pgie_peopleSegNet_tao_config.txt -i v4l2:///dev/video2
Request sink_0 pad from streammux
batchSize 1…
Now playing: configs/peopleSegNet_tao/pgie_peopleSegNet_tao_config.txt
Opening in BLOCKING MODE
WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
0:00:04.814394104 2498088 0xaaaabfdb5730 INFO nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1909> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.2/sources/deepstream_tao_apps/models/peopleSegNet/peopleSegNet_resnet50.etlt_b1_gpu0_fp16.engine
INFO: [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT Input 3x576x960
1 OUTPUT kFLOAT generate_detections 100x6
2 OUTPUT kFLOAT mask_head/mask_fcn_logits/BiasAdd 100x2x28x28
0:00:05.032843361 2498088 0xaaaabfdb5730 INFO nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2012> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.2/sources/deepstream_tao_apps/models/peopleSegNet/peopleSegNet_resnet50.etlt_b1_gpu0_fp16.engine
0:00:05.049528802 2498088 0xaaaabfdb5730 INFO nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus: [UID 1]: Load new model:configs/peopleSegNet_tao/pgie_peopleSegNet_tao_config.txt sucessfully
Decodebin child added: source
Decodebin child added: decodebin0
Opening in BLOCKING MODE
Running…
**PERF: FPS 0 (Avg)
Mon Mar 6 20:36:09 2023
**PERF: 0.00(0.00)
In cb_newpad
###Decodebin did not pick nvidia decoder plugin.
NvMMLiteOpen : Block : BlockType = 4
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
H264: Profile = 66, Level = 0
NVMEDIA: Need to set EMC bandwidth : 197400
NVMEDIA_ENC: bBlitMode is set to TRUE
Mon Mar 6 20:36:14 2023
**PERF: 7.74(7.50)
Mon Mar 6 20:36:19 2023
**PERF: 7.25(7.30)
…
/opt/nvidia/deepstream/deepstream-6.2/sources/deepstream_tao_apps$ sudo GST_DEBUG=3 ./apps/tao_detection/ds-tao-detection -c configs/peopleSegNet_tao/pgie_peopleSegNet_tao_config.txt -i v4l2:///dev/video0Request sink_0 pad from streammux
batchSize 1…
Now playing: configs/peopleSegNet_tao/pgie_peopleSegNet_tao_config.txt
Opening in BLOCKING MODE
0:00:00.346134246 2805895 0xaaaaf77bf040 WARN v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0xaaaaf779d710 Failed to determine interlace mode
0:00:00.346205543 2805895 0xaaaaf77bf040 WARN v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0xaaaaf779d710 Failed to determine interlace mode
0:00:00.346230439 2805895 0xaaaaf77bf040 WARN v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0xaaaaf779d710 Failed to determine interlace mode
0:00:00.346250600 2805895 0xaaaaf77bf040 WARN v4l2 gstv4l2object.c:2395:gst_v4l2_object_add_interlace_mode:0xaaaaf779d710 Failed to determine interlace mode
0:00:00.346329193 2805895 0xaaaaf77bf040 WARN v4l2 gstv4l2object.c:4497:gst_v4l2_object_probe_caps:h264-enc:src Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Unknown error -1
WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
0:00:05.079757081 2805895 0xaaaaf77bf040 INFO nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1909> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.2/sources/deepstream_tao_apps/models/peopleSegNet/peopleSegNet_resnet50.etlt_b1_gpu0_fp16.engine
INFO: [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT Input 3x576x960
1 OUTPUT kFLOAT generate_detections 100x6
2 OUTPUT kFLOAT mask_head/mask_fcn_logits/BiasAdd 100x2x28x28
0:00:05.291795226 2805895 0xaaaaf77bf040 INFO nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2012> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.2/sources/deepstream_tao_apps/models/peopleSegNet/peopleSegNet_resnet50.etlt_b1_gpu0_fp16.engine
0:00:05.308946453 2805895 0xaaaaf77bf040 INFO nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus: [UID 1]: Load new model:configs/peopleSegNet_tao/pgie_peopleSegNet_tao_config.txt sucessfully
Decodebin child added: source
Decodebin child added: decodebin0
Opening in BLOCKING MODE
Running…
**PERF: FPS 0 (Avg)
Mon Mar 6 20:56:39 2023
**PERF: 0.00(0.00)
Decodebin child added: nvjpegdec0
0:00:06.140427495 2805895 0xaaab14b17860 FIXME videodecoder gstvideodecoder.c:946:gst_video_decoder_drain_out: Sub-class should implement drain()
0:00:06.140573737 2805895 0xaaab14b17860 FIXME videodecoder gstvideodecoder.c:946:gst_video_decoder_drain_out: Sub-class should implement drain()
0:00:06.202098046 2805895 0xaaab14b17860 WARN v4l2bufferpool gstv4l2bufferpool.c:809:gst_v4l2_buffer_pool_start:source:pool:src Uncertain or not enough buffers, enabling copy threshold
Mon Mar 6 20:56:44 2023
**PERF: 0.00(0.00)
In cb_newpad
###Decodebin pick nvidia decoder plugin.
NvMMLiteOpen : Block : BlockType = 4
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
0:00:12.126468971 2805895 0xaaab14b17860 WARN v4l2bufferpool gstv4l2bufferpool.c:1082:gst_v4l2_buffer_pool_start:h264-enc:pool:src Uncertain or not enough buffers, enabling copy threshold
H264: Profile = 66, Level = 0
0:00:12.495085492 2805895 0xaaaaf6f07240 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:h264-enc:pool:sink Buffer 0xffff2405c7e0 without the memory tag has maxsize (192) that is smaller than the configured buffer pool size (3133440). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
NVMEDIA: Need to set EMC bandwidth : 846000
NVMEDIA_ENC: bBlitMode is set to TRUE
0:00:12.535112806 2805895 0xffff2405a0c0 WARN v4l2bufferpool gstv4l2bufferpool.c:1533:gst_v4l2_buffer_pool_dqbuf:h264-enc:pool:src Driver should never set v4l2_buffer.field to ANY
0:00:12.540184569 2805895 0xffff2405a0c0 FIXME basesink gstbasesink.c:3246:gst_base_sink_default_event: stream-start event without group-id. Consider implementing group-id handling in the upstream elements
0:00:12.540288347 2805895 0xffff2405a0c0 WARN qtmux gstqtmux.c:2981:gst_qt_mux_start_file: Robust muxing requires reserved-moov-update-period to be set
Mon Mar 6 20:56:49 2023
**PERF: 0.00(0.00)
0:00:17.579355883 2805895 0xaaab14b17860 WARN v4l2src gstv4l2src.c:914:gst_v4l2src_create: Timestamp does not correlate with any clock, ignoring driver timestamps
0:00:17.794318548 2805895 0xaaaaf6f07240 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:h264-enc:pool:sink Buffer 0xffff102fe480 without the memory tag has maxsize (192) that is smaller than the configured buffer pool size (3133440). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
Mon Mar 6 20:56:54 2023
**PERF: 0.19(0.11)
0:00:23.137407754 2805895 0xaaaaf6f07240 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:h264-enc:pool:sink Buffer 0xffff102fe5a0 without the memory tag has maxsize (192) that is smaller than the configured buffer pool size (3133440). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
Mon Mar 6 20:56:59 2023
**PERF: 0.19(0.14)
0:00:28.308630503 2805895 0xaaab14b17860 WARN v4l2src gstv4l2src.c:978:gst_v4l2src_create: lost frames detected: count = 162 - ts: 0:00:22.151430011
0:00:28.468794854 2805895 0xaaaaf6f07240 WARN bufferpool gstbufferpool.c:1235:default_reset_buffer:h264-enc:pool:sink Buffer 0xffff102fe6c0 without the memory tag has maxsize (192) that is smaller than the configured buffer pool size (3133440). The buffer will be not be reused. This is most likely a bug in this GstBufferPool subclass
Yuweiw: I did that and the live display showed up on the screen however this time the frames are getting lost and getting delayed. I am seeing a frame update every 8 seconds or so. The camera is like frozen.
v4l2src gstv4l2src.c:978:gst_v4l2src_create: lost frames detected:
/opt/nvidia/deepstream/deepstream-6.2/sources/deepstream_tao_apps$ sudo GST_DEBUG=3 ./apps/tao_detection/ds-tao-detection -c configs/peopleSegNet_tao/pgie_peopleSegNet_tao_config.txt -i v4l2:///dev/video1 -d
Request sink_0 pad from streammux
batchSize 1...
Now playing: configs/peopleSegNet_tao/pgie_peopleSegNet_tao_config.txt
Using winsys: x11
WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
0:00:04.273412878 2592614 0xaaaae65a8f30 INFO nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1909> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.2/sources/deepstream_tao_apps/models/peopleSegNet/peopleSegNet_resnet50.etlt_b1_gpu0_fp16.engine
INFO: [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT Input 3x576x960
1 OUTPUT kFLOAT generate_detections 100x6
2 OUTPUT kFLOAT mask_head/mask_fcn_logits/BiasAdd 100x2x28x28
0:00:04.450522706 2592614 0xaaaae65a8f30 INFO nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2012> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.2/sources/deepstream_tao_apps/models/peopleSegNet/peopleSegNet_resnet50.etlt_b1_gpu0_fp16.engine
0:00:04.464711689 2592614 0xaaaae65a8f30 INFO nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary-nvinference-engine> [UID 1]: Load new model:configs/peopleSegNet_tao/pgie_peopleSegNet_tao_config.txt sucessfully
Decodebin child added: source
Decodebin child added: decodebin0
Running...
**PERF: FPS 0 (Avg)
Tue Mar 7 19:57:37 2023
**PERF: 0.00(0.00)
Decodebin child added: nvjpegdec0
0:00:05.278370020 2592614 0xaaaae5d0c180 FIXME videodecoder gstvideodecoder.c:946:gst_video_decoder_drain_out:<nvjpegdec0> Sub-class should implement drain()
0:00:05.278500838 2592614 0xaaaae5d0c180 FIXME videodecoder gstvideodecoder.c:946:gst_video_decoder_drain_out:<nvjpegdec0> Sub-class should implement drain()
0:00:05.314582358 2592614 0xaaaae5d0c180 WARN v4l2bufferpool gstv4l2bufferpool.c:809:gst_v4l2_buffer_pool_start:<source:pool:src> Uncertain or not enough buffers, enabling copy threshold
In cb_newpad
###Decodebin pick nvidia decoder plugin.
Tue Mar 7 19:57:42 2023
**PERF: 0.00(0.00)
0:00:13.758332568 2592614 0xaaaae5d0c460 WARN basesink gstbasesink.c:3003:gst_base_sink_is_too_late:<nvvideo-renderer> warning: A lot of buffers are being dropped.
0:00:13.758362265 2592614 0xaaaae5d0c460 WARN basesink gstbasesink.c:3003:gst_base_sink_is_too_late:<nvvideo-renderer> warning: There may be a timestamping problem, or this computer is too slow.
Tue Mar 7 19:57:47 2023
**PERF: 0.27(0.18)
0:00:17.081108624 2592614 0xaaaae5d0c180 WARN v4l2src gstv4l2src.c:914:gst_v4l2src_create:<source> Timestamp does not correlate with any clock, ignoring driver timestamps
Tue Mar 7 19:57:52 2023
**PERF: 0.27(0.19)
0:00:20.800280382 2592614 0xaaaae5d0c180 WARN v4l2src gstv4l2src.c:978:gst_v4l2src_create:<source> lost frames detected: count = 110 - ts: 0:00:15.508610309
0:00:24.440950532 2592614 0xaaaae5d0c180 WARN v4l2src gstv4l2src.c:978:gst_v4l2src_create:<source> lost frames detected: count = 113 - ts: 0:00:19.149278092
Tue Mar 7 19:57:57 2023
**PERF: 0.27(0.26)
0:00:28.275298419 2592614 0xaaaae5d0c180 WARN v4l2src gstv4l2src.c:978:gst_v4l2src_create:<source> lost frames detected: count = 111 - ts: 0:00:22.983636410
Tue Mar 7 19:58:02 2023
**PERF: 0.26(0.24)
0:00:32.050112686 2592614 0xaaaae5d0c180 WARN v4l2src gstv4l2src.c:978:gst_v4l2src_create:<source> lost frames detected: count = 111 - ts: 0:00:26.758430070
From the log attached, it may be the clock problems of your camera. You can try to adjust the camera parameters. Or you can try to use the rtsp source from your camera.
Using the same USB camera, I can run deepstream python apps with no problems. I have run “deepstream_test_1_usb.py” to test my USB camera and it can reach to 30FPS with no frames lost:
python3 deepstream_test_1_usb.py /dev/video0
However with deesptream tao apps, the frames are lost and the camera gets frozen.
My logitech USB camera (logitech brio) is still freezing (extremely low frame rate) on the display when I try to run deepstream tao apps. How can I adjust camera parameters? From the source code? If so which one?
I can set the camera parameters as the below image shows, but when I run the deesptream tao apps, the parameters are overwritten. So I have to revise something in the code, but I don’t know where.
Decodebin child added: source
Decodebin child added: decodebin0
Running…
Decodebin child added: nvjpegdec0
0:00:04.173117035 135940 0xaaab21424e40 FIXME videodecoder gstvideodecoder.c:946:gst_video_decoder_drain_out: Sub-class should implement drain()
0:00:04.173233805 135940 0xaaab21424e40 FIXME videodecoder gstvideodecoder.c:946:gst_video_decoder_drain_out: Sub-class should implement drain()
0:00:04.192036007 135940 0xaaab21424e40 WARN v4l2bufferpool gstv4l2bufferpool.c:809:gst_v4l2_buffer_pool_start:source:pool:src Uncertain or not enough buffers, enabling copy threshold
In cb_newpad
###Decodebin pick nvidia decoder plugin.
Deserializing engine from: ./gazeinfer_impl/…/…/…/…/models/gazenet/gazenet_facegrid.etlt_b8_gpu0_fp16.engineThe logger passed into createInferRuntime differs from one already provided for an existing builder, runtime, or refitter. Uses of the global logger, returned by nvinfer1::getLogger(), will return the existing value.
Loaded engine size: 9 MiB
Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
Deserialization required 14318 microseconds.
[MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +9, now: CPU 0, GPU 153 (MiB)
Total per-runner device persistent memory is 56832
Total per-runner host persistent memory is 109056
Allocated activation device memory of size 22133248
[MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +21, now: CPU 0, GPU 174 (MiB)
0:01:40.655586937 135940 0xaaab21424e40 WARN v4l2src gstv4l2src.c:914:gst_v4l2src_create: Timestamp does not correlate with any clock, ignoring driver timestamps
Frame Number = 0 Face Count = 0
Frame Number = 1 Face Count = 0
1.You can get the real camera parameters through v4l2-ctl -d /dev/video0 --list-formats-ext and attach it.
2.You can change the gstreamer pipeline to: v4l2src->capsfilter(set your own format)->....
Your camera has 3 formats output from the list: YUYV, MJPEG and NV12. You can try to set the format to YUYV or NV12 (from the log and the image you attached before, although you set YUYV format, the actual setting is MJPEG). You can ask your camera vendor how to set it to YUYV or NV12 successfully.
I can control the camera parameters of my Logitech Brio directly with the below command. However, how can I incorporate this into my code to initialize camera parameters ?
After the above command, I run the below deesptream gaze app, however it takes very long for the camera display show up on the screen and when it does the camera frame rate drops substantially (nearly freezes). After running the deepstream gaze app, I check my camera parameters again and see that they go back to MJPEG format (Although I set it as YUYV with gst-launch-1.0)
I can now permanently set the camera parameters using guvcview. See the below attached image for the parameters (and they do not change once I set them). I have also revised the caps in deepstream-gaze-app.cpp to match the camera parameters and built it using make.
