What's the difference between process-mode 1 and 2 of sgie

Please provide complete information as applicable to your setup.

• Hardware Platform GPU
• DeepStream Version 6.0.1
• TensorRT Version 8
• NVIDIA GPU Driver Version (valid for GPU only) 470
• Issue Type questions
Hi there
I build a simple pipeline on the top of deepstream-test3. I use yolov5 as pgie and retinaface as sgie. while I set the process-mode of sgie, the difference confused me.
first of all, I use fakesink instead of eglglessink because I use 4k video as input.
then I add a probe at sinkpad of osd element to monitor the fps.
if I set process-mode to 1 (full frame as doc said), the fps is

**********************FPS*****************************************
Fps of stream 0 is  62.4
Got EOS from stream 0
End-of-stream
Exiting app

but if I set process-mode to 2, the fps is:

**********************FPS*****************************************
Fps of stream 0 is  19.2
**********************FPS*****************************************
Fps of stream 0 is  18.4
**********************FPS*****************************************
Fps of stream 0 is  20.0
**********************FPS*****************************************
Fps of stream 0 is  20.6
Got EOS from stream 0
End-of-stream
Exiting app

then i try to use eglglessink, and the video could play normally if the process-mode is 1. it couldn’t play normally if process-mode is 2 (the fps is almost 0):

Warning: gst-core-error-quark: A lot of buffers are being dropped. (13): gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstEglGlesSink:nvvideo-renderer:
There may be a timestamping problem, or this computer is too slow.
Warning: gst-core-error-quark: A lot of buffers are being dropped. (13): gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstEglGlesSink:nvvideo-renderer:
There may be a timestamping problem, or this computer is too slow.

I can’t find more information about process-mode, appropriate if you can give me some advice.
thanks a lot

You can check the comment in sampe code “deepstream_test3_app.c” regarding to process-mode: 2 is for Jetson product, that is embedded product, not for dGPU that you are using.

/* By default, OSD process-mode is set to CPU_MODE. To change mode, set as:
 * 1: GPU mode (for Tesla only)
 * 2: HW mode (For Jetson only)
 */
#define OSD_PROCESS_MODE 0

thanks for the quick reply
I guess there is two process-mode?
one is for nvinfer, another is for osd. I guess they are not the same?

Can you have a try with sync=false for eglglessink?

@kesong Yes, I tried everything I know, none worked. Even worse, now the whole pipeline freeze if I set process-mode of sgie to 2. I also try to debug, but there is no information:

0:00:04.295270671   184      0x25b61e0 LOG               bufferpool gstbufferpool.c:1133:default_acquire_buffer:<bufferpool1> no buffer, trying to allocate
0:00:04.295279418   184      0x25b61e0 DEBUG             bufferpool gstbufferpool.c:304:do_alloc_buffer:<bufferpool1> max buffers reached
0:00:04.295287448   184      0x25b61e0 LOG                 GST_POLL gstpoll.c:314:release_wakeup: 0x25b60f0: release
0:00:04.295299974   184      0x25b61e0 LOG               bufferpool gstbufferpool.c:1173:default_acquire_buffer:<bufferpool1> waiting for free buffers or flushing
0:00:04.295310985   184      0x25b61e0 DEBUG               GST_POLL gstpoll.c:1317:gst_poll_wait: 0x25b60f0: timeout :99:99:99.999999999
0:00:04.298340812   184 0x7fc21c0024d0 LOG               GST_BUFFER gstbuffer.c:710:_gst_buffer_dispose: release 0x51e735e0 to pool 0x3378750
0:00:04.298376735   184 0x7fc21c0024d0 LOG               bufferpool gstbufferpool.c:1284:default_release_buffer:<bufferpool1> released buffer 0x51e735e0 0
0:00:04.298386385   184 0x7fc21c0024d0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:04.298395101   184 0x7fc21c0024d0 LOG                 GST_POLL gstpoll.c:290:raise_wakeup: 0x25b60f0: raise
0:00:04.298432316   184      0x25b61e0 LOG               bufferpool gstbufferpool.c:1128:default_acquire_buffer:<bufferpool1> acquired buffer 0x51e735e0
0:00:04.298547273   184      0x25b61e0 DEBUG         GST_SCHEDULING gstpad.c:4326:gst_pad_chain_data_unchecked:<Secondary-inference:sink> called chainfunction &gst_base_transform_chain with buffer 0x7fc248282c70, returned ok
0:00:04.298563180   184      0x25b61e0 DEBUG         queue_dataflow gstqueue.c:1516:gst_queue_loop:<queue3> queue is empty
0:00:04.298579930   184      0x25b61e0 LOG           queue_dataflow gstqueue.c:1525:gst_queue_loop:<queue3> (queue3:src) wait for ADD: 0 of 0-200 buffers, 0 of 0-10485760 bytes, 0 of 0-1000000000 ns, 0 items
0:00:04.302565075   184 0x7fc21c0024d0 LOG               GST_BUFFER gstbuffer.c:710:_gst_buffer_dispose: release 0x51e733c0 to pool 0x3378750
0:00:04.302588173   184 0x7fc21c0024d0 LOG               bufferpool gstbufferpool.c:1284:default_release_buffer:<bufferpool1> released buffer 0x51e733c0 0
0:00:04.302603097   184 0x7fc21c0024d0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:04.306983446   184 0x7fc21c0024d0 LOG               GST_BUFFER gstbuffer.c:710:_gst_buffer_dispose: release 0x51e734d0 to pool 0x3378750
0:00:04.307006581   184 0x7fc21c0024d0 LOG               bufferpool gstbufferpool.c:1284:default_release_buffer:<bufferpool1> released buffer 0x51e734d0 0
0:00:04.307025451   184 0x7fc21c0024d0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:04.312559729   184 0x7fc21c0024d0 LOG               GST_BUFFER gstbuffer.c:710:_gst_buffer_dispose: release 0x51e735e0 to pool 0x3378750
0:00:04.312573160   184 0x7fc21c0024d0 LOG               bufferpool gstbufferpool.c:1284:default_release_buffer:<bufferpool1> released buffer 0x51e735e0 0
0:00:04.312578532   184 0x7fc21c0024d0 DEBUG             GST_BUFFER gstbuffer.c:1375:gst_buffer_is_memory_range_writable: idx 0, length -1
0:00:04.318640952   184      0x25b6190 DEBUG         GST_SCHEDULING gstpad.c:4320:gst_pad_chain_data_unchecked:<queue4:sink> calling chainfunction &gst_queue_chain with buffer buffer: 0x7fc248282c70, pts 0:00:00.199999998, dts 99:99:99.999999999, dur 99:99:99.999999999, size 64, offset none, offset_end none, flags 0x0
0:00:04.318660843   184      0x25b6190 LOG           queue_dataflow gstqueue.c:1203:gst_queue_chain_buffer_or_list:<queue4> received buffer 0x7fc248282c70 of size 64, time 0:00:00.199999998, duration 99:99:99.999999999
0:00:04.318671993   184      0x25b6190 LOG                    queue gstqueue.c:635:apply_buffer:<queue4> sink position updated to 0:00:00.199999998
0:00:04.318678721   184      0x25b6190 LOG                    queue gstqueue.c:530:update_time_level:<queue4> update sink time
0:00:04.318690260   184      0x25b6190 LOG                    queue gstqueue.c:548:update_time_level:<queue4> sink +0:00:00.199999998, src +0:00:00.133333332
0:00:04.318699835   184      0x25b6190 DEBUG         GST_SCHEDULING gstpad.c:4326:gst_pad_chain_data_unchecked:<queue4:sink> called chainfunction &gst_queue_chain with buffer 0x7fc248282c70, returned ok

I have no clue why this happened. I don’t understand why it works fine with process-mode 1?
You can find the codes here.

@kesong Hi, I don’t know why but pipeline suddenly could work though its fps is quite low. anyway, there is the debug information.

Warning: gst-core-error-quark: A lot of buffers are being dropped. (13): gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstEglGlesSink:nvvideo-renderer:
There may be a timestamping problem, or this computer is too slow.
0:00:22.196681374  1001      0x1b94ad0 WARN                basesink gstbasesink.c:2902:gst_base_sink_is_too_late:<nvvideo-renderer> warning: A lot of buffers are being dropped.
0:00:22.196692397  1001      0x1b94ad0 WARN                basesink gstbasesink.c:2902:gst_base_sink_is_too_late:<nvvideo-renderer> warning: There may be a timestamping problem, or this computer is too slow.
0:00:22.196708504  1001      0x1b94ad0 INFO        GST_ERROR_SYSTEM gstelement.c:2145:gst_element_message_full_with_details:<nvvideo-renderer> posting message: A lot of buffers are being dropped.
0:00:22.196730352  1001      0x1b94ad0 INFO        GST_ERROR_SYSTEM gstelement.c:2172:gst_element_message_full_with_details:<nvvideo-renderer> posted warning message: A lot of buffers are being dropped.
Warning: gst-core-error-quark: A lot of buffers are being dropped. (13): gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstEglGlesSink:nvvideo-renderer:
There may be a timestamping problem, or this computer is too slow.
0:00:24.186299313  1001      0x1b94ad0 WARN                basesink gstbasesink.c:2902:gst_base_sink_is_too_late:<nvvideo-renderer> warning: A lot of buffers are being dropped.
0:00:24.186317843  1001      0x1b94ad0 WARN                basesink gstbasesink.c:2902:gst_base_sink_is_too_late:<nvvideo-renderer> warning: There may be a timestamping problem, or this computer is too slow.
0:00:24.186331577  1001      0x1b94ad0 INFO        GST_ERROR_SYSTEM gstelement.c:2145:gst_element_message_full_with_details:<nvvideo-renderer> posting message: A lot of buffers are being dropped.
0:00:24.186357218  1001      0x1b94ad0 INFO        GST_ERROR_SYSTEM gstelement.c:2172:gst_element_message_full_with_details:<nvvideo-renderer> posted warning message: A lot of buffers are being dropped.
Warning: gst-core-error-quark: A lot of buffers are being dropped. (13): gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstEglGlesSink:nvvideo-renderer:
There may be a timestamping problem, or this computer is too slow.
0:00:24.186503343  1001      0x1b94ad0 WARN                basesink gstbasesink.c:2902:gst_base_sink_is_too_late:<nvvideo-renderer> warning: A lot of buffers are being dropped.
0:00:24.186521337  1001      0x1b94ad0 WARN                basesink gstbasesink.c:2902:gst_base_sink_is_too_late:<nvvideo-renderer> warning: There may be a timestamping problem, or this computer is too slow.
0:00:24.186534716  1001      0x1b94ad0 INFO        GST_ERROR_SYSTEM gstelement.c:2145:gst_element_message_full_with_details:<nvvideo-renderer> posting message: A lot of buffers are being dropped.
0:00:24.186559583  1001      0x1b94ad0 INFO        GST_ERROR_SYSTEM gstelement.c:2172:gst_element_message_full_with_details:<nvvideo-renderer> posted warning message: A lot of buffers are being dropped.
Warning: gst-core-error-quark: A lot of buffers are being dropped. (13): gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstEglGlesSink:nvvideo-renderer:
There may be a timestamping problem, or this computer is too slow.
0:00:26.156528584  1001      0x1b94ad0 WARN                basesink gstbasesink.c:2902:gst_base_sink_is_too_late:<nvvideo-renderer> warning: A lot of buffers are being dropped.
0:00:26.156547384  1001      0x1b94ad0 WARN                basesink gstbasesink.c:2902:gst_base_sink_is_too_late:<nvvideo-renderer> warning: There may be a timestamping problem, or this computer is too slow.
0:00:26.156565786  1001      0x1b94ad0 INFO        GST_ERROR_SYSTEM gstelement.c:2145:gst_element_message_full_with_details:<nvvideo-renderer> posting message: A lot of buffers are being dropped.
0:00:26.156599059  1001      0x1b94ad0 INFO        GST_ERROR_SYSTEM gstelement.c:2172:gst_element_message_full_with_details:<nvvideo-renderer> posted warning message: A lot of buffers are being dropped.
Warning: gst-core-error-quark: A lot of buffers are being dropped. (13): gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstEglGlesSink:nvvideo-renderer:
There may be a timestamping problem, or this computer is too slow.

I’m using 2080ti with one single .mp4 source, I don’t think it is because or this computer is too slow.
I followed your suggestion to set

    sink.set_property("qos",0)
    sink.set_property("sync","False")

but it doesn’t work.
Do you have any ideas? thanks a lot

I think I find the reason. That’s because if I detect faces on the top of yolo, assume the source is 30fps and there is 10 people in a frame, so there is 300 images feed into retinaface, that’s a lot of to deal with.
thanks for you guys’ help :-). I’ll find a way to solve this.

Glad to know you find the root cause.