Please provide complete information as applicable to your setup.
• Hardware Platform: GTX 1080 • DeepStream Version: 5.0 • NVIDIA GPU Driver Version (valid for GPU only): 440.33.01
Hi,
I have a pipeline with primary face detector and a secondary face recognizer. The secondary recognizer operates on the faces detected by the primary detector. I am using nvinfer for primary inference and nvinferserver for secondary inference.
The issue happens when the number of faces detected by the by primary detector is more than the batch-size of secondary inference(secondary_batch_size < total face detected of all channels). In this case it freezes, gpu-util % goes to zero and cpu also goes to zero but memory is still occupied on gpu. The number of faces can be of any number and number of streams can also upto 15. Is there a way for the pipeline to not freeze if num face detected goes beyond the secondary batch size and also to loop through the remaining faces in next batch?
There is another issue, the primary detecter is very fast and secondary is slow to catch up to the detected faces(will get stuck for few seconds in some cases).
Is there a way to control the fps of stream in the pipeline for each source other than dropping frame? The source fps can be controlled for rtp source but if a file source is given, the pipeline seems to be running at highest fps possible. I have tried videorate plugin with capsfilter but that doesn’t seem to work.
Hi @mchi,
I have captured the log for two cases. I have set the secondary batch size to 12. I have used 3 streams for the test case.
The first case I have used 3 streams with 4 faces-per-frame of each stream.
Here total faces detected == batch_size(no freezing) - temp1.log
I have stopped inference after 20 frames temp1.log (287.3 KB)
For the second case, I have used 2 streams with 4 faces-per-frame and 1 stream with 5 face-per-frame. Here the total faces detected > batch size(freezing happens)- temp2.log
Here the pipeline stops after 1 frame and CPU% and GPU util goes to 0 temp2.log (263.9 KB)
The pipeline freezes after displaying following logs:
Src id 1 timediff: 1.000000
Frame Number = 0 Number of objects = 4
Src id 0 timediff: 1.000000
Frame Number = 0 Number of objects = 4
Src id 2 timediff: 1.000000
Frame Number = 0 Number of objects = 5
0:00:05.927800013 26246 0x56242a6181e0 INFO GST_STATES gstbin.c:3424:bin_handle_async_done:<dstest1-pipeline> committing state from READY to PAUSED, old pending PLAYING
0:00:05.927809970 26246 0x56242a6181e0 INFO GST_STATES gstbin.c:3453:bin_handle_async_done:<dstest1-pipeline> continue state change, pending PLAYING
0:00:05.927816105 26246 0x56242a6181e0 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<dstest1-pipeline> notifying about state-changed READY to PAUSED (PLAYING pending)
0:00:05.927922320 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:3250:gst_bin_continue_func:<dstest1-pipeline> continue state change PAUSED to PLAYING, final PLAYING
0:00:05.928073966 26246 0x7fe244003e80 INFO GST_EVENT gstevent.c:1388:gst_event_new_latency: creating latency event 0:00:00.000000000
0:00:05.928117019 26246 0x7fe244003e80 INFO bin gstbin.c:2783:gst_bin_do_latency_func:<dstest1-pipeline> configured latency of 0:00:00.000000000
0:00:05.928133721 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<nvvideo-renderer> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928141157 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<nvvideo-renderer> completed state change to PLAYING
0:00:05.928148856 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<nvvideo-renderer> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928158543 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<dstest1-pipeline> child 'nvvideo-renderer' changed state to 4(PLAYING) successfully
0:00:05.928165846 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<q2> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928169793 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<q2> completed state change to PLAYING
0:00:05.928175456 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<q2> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928197511 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<dstest1-pipeline> child 'q2' changed state to 4(PLAYING) successfully
0:00:05.928205592 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<secondary-nvinference-engine> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928210648 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<secondary-nvinference-engine> completed state change to PLAYING
0:00:05.928214992 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<secondary-nvinference-engine> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928224281 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<dstest1-pipeline> child 'secondary-nvinference-engine' changed state to 4(PLAYING) successfully
0:00:05.928232205 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<q1> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928238456 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<q1> completed state change to PLAYING
0:00:05.928243916 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<q1> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928252838 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<dstest1-pipeline> child 'q1' changed state to 4(PLAYING) successfully
0:00:05.928258539 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<primary-nvinference-engine> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928262707 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<primary-nvinference-engine> completed state change to PLAYING
0:00:05.928266170 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<primary-nvinference-engine> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928273715 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<dstest1-pipeline> child 'primary-nvinference-engine' changed state to 4(PLAYING) successfully
0:00:05.928280012 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<stream-muxer> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928289310 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<stream-muxer> completed state change to PLAYING
0:00:05.928301134 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<stream-muxer> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928313681 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<dstest1-pipeline> child 'stream-muxer' changed state to 4(PLAYING) successfully
0:00:05.928320227 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<source-bin-01> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928333524 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<decodebin1> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928344241 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<nvv4l2decoder0> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928351928 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<nvv4l2decoder0> completed state change to PLAYING
0:00:05.928358277 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<nvv4l2decoder0> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928372965 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<decodebin1> child 'nvv4l2decoder0' changed state to 4(PLAYING) successfully
0:00:05.928381471 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<capsfilter2> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928387373 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<capsfilter2> completed state change to PLAYING
0:00:05.928392310 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<capsfilter2> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928414441 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<decodebin1> child 'capsfilter2' changed state to 4(PLAYING) successfully
0:00:05.928423551 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<h264parse2> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928430097 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<h264parse2> completed state change to PLAYING
0:00:05.928436458 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<h264parse2> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928447970 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<decodebin1> child 'h264parse2' changed state to 4(PLAYING) successfully
0:00:05.928455837 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<typefind> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928463254 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<typefind> completed state change to PLAYING
0:00:05.928468168 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<typefind> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928477384 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<decodebin1> child 'typefind' changed state to 4(PLAYING) successfully
0:00:05.928484393 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<decodebin1> completed state change to PLAYING
0:00:05.928488670 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<decodebin1> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928496936 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<source-bin-01> child 'decodebin1' changed state to 4(PLAYING) successfully
0:00:05.928503408 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<source> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928507412 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<source> completed state change to PLAYING
0:00:05.928510760 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<source> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928517769 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<source-bin-01> child 'source' changed state to 4(PLAYING) successfully
0:00:05.928523692 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<source-bin-01> completed state change to PLAYING
0:00:05.928528349 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<source-bin-01> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928539183 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<dstest1-pipeline> child 'source-bin-01' changed state to 4(PLAYING) successfully
0:00:05.928544670 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<source-bin-00> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928555814 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<decodebin2> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928566310 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<nvv4l2decoder2> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928571213 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<nvv4l2decoder2> completed state change to PLAYING
0:00:05.928577079 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<nvv4l2decoder2> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928596249 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<decodebin2> child 'nvv4l2decoder2' changed state to 4(PLAYING) successfully
0:00:05.928604188 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<capsfilter1> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928610768 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<capsfilter1> completed state change to PLAYING
0:00:05.928615832 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<capsfilter1> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928626132 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<decodebin2> child 'capsfilter1' changed state to 4(PLAYING) successfully
0:00:05.928633566 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<h264parse1> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928637737 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<h264parse1> completed state change to PLAYING
0:00:05.928641466 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<h264parse1> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928649566 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<decodebin2> child 'h264parse1' changed state to 4(PLAYING) successfully
0:00:05.928656720 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<typefind> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928661679 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<typefind> completed state change to PLAYING
0:00:05.928664750 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<typefind> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928671984 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<decodebin2> child 'typefind' changed state to 4(PLAYING) successfully
0:00:05.928677282 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<decodebin2> completed state change to PLAYING
0:00:05.928681452 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<decodebin2> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928688866 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<source-bin-00> child 'decodebin2' changed state to 4(PLAYING) successfully
0:00:05.928694819 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<source> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928698803 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<source> completed state change to PLAYING
0:00:05.928701843 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<source> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928708697 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<source-bin-00> child 'source' changed state to 4(PLAYING) successfully
0:00:05.928713735 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<source-bin-00> completed state change to PLAYING
0:00:05.928716844 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<source-bin-00> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928727725 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<dstest1-pipeline> child 'source-bin-00' changed state to 4(PLAYING) successfully
0:00:05.928733170 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<source-bin-02> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928738760 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<decodebin0> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928744762 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<nvv4l2decoder1> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928749190 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<nvv4l2decoder1> completed state change to PLAYING
0:00:05.928752977 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<nvv4l2decoder1> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928760924 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<decodebin0> child 'nvv4l2decoder1' changed state to 4(PLAYING) successfully
0:00:05.928767535 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<capsfilter0> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928771712 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<capsfilter0> completed state change to PLAYING
0:00:05.928774712 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<capsfilter0> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928782923 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<decodebin0> child 'capsfilter0' changed state to 4(PLAYING) successfully
0:00:05.928789986 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<h264parse0> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928794456 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<h264parse0> completed state change to PLAYING
0:00:05.928798123 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<h264parse0> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928805306 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<decodebin0> child 'h264parse0' changed state to 4(PLAYING) successfully
0:00:05.928811840 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<typefind> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928816044 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<typefind> completed state change to PLAYING
0:00:05.928819852 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<typefind> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928827482 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<decodebin0> child 'typefind' changed state to 4(PLAYING) successfully
0:00:05.928832801 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<decodebin0> completed state change to PLAYING
0:00:05.928836323 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<decodebin0> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928843319 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<source-bin-02> child 'decodebin0' changed state to 4(PLAYING) successfully
0:00:05.928848963 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2506:gst_bin_element_set_state:<source> current PAUSED pending VOID_PENDING, desired next PLAYING
0:00:05.928854756 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<source> completed state change to PLAYING
0:00:05.928860505 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<source> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928871967 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<source-bin-02> child 'source' changed state to 4(PLAYING) successfully
0:00:05.928878557 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<source-bin-02> completed state change to PLAYING
0:00:05.928883532 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<source-bin-02> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:05.928903763 26246 0x7fe244003e80 INFO GST_STATES gstbin.c:2954:gst_bin_change_state_func:<dstest1-pipeline> child 'source-bin-02' changed state to 4(PLAYING) successfully
0:00:05.928910382 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2676:gst_element_continue_state:<dstest1-pipeline> completed state change to PLAYING
0:00:05.928915910 26246 0x7fe244003e80 INFO GST_STATES gstelement.c:2579:_priv_gst_element_state_changed:<dstest1-pipeline> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
Hi,
Here I have set primary batch size as 5 and secondary batch size as 25. I use 4 videos with 5 faces in them and the fifth video having 6 faces in them. It freezes for all cases where total num faces > sgie batch_size
the pgie config -
[property]
gpu-id=0
net-scale-factor=1
#0=RGB, 1=BGR
model-color-format=0
#custom-network-config=primary.cfg
#model-file=primary.weights #for different batch size different engine so while running new batch size use custom-network and model-file
#model-engine-file=model_b16_gpu0_fp32.engine
#model-engine-file=model_b15_gpu0_fp32.engine
#model-engine-file=model_b10_gpu0_fp32.engine
model-engine-file=model_b5_gpu0_fp32.engine
#model-engine-file=model_b4_gpu0_fp32.engine
#model-engine-file=model_b3_gpu0_fp32.engine
#model-engine-file=model_b2_gpu0_fp32.engine
#model-engine-file=model_b1_gpu0_fp32.engine
labelfile-path=labels.txt
batch-size=5
network-mode=0
num-detected-classes=1
gie-unique-id=1
is-classifier=0
maintain-aspect-ratio=1
parse-bbox-func-name=NvDsInferParseCustomYoloV3Tiny
custom-lib-path=nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so
When I enable exporting of tensor metadata, the NvDsInferTensorMeta keeps reference to tensor_out_object (GstNvInferTensorOutputObject). It causes that all resources allocated by batch are released only if the whole batch is processed. But it also uses nvinfer->pool, which is of limited size, and each object from the batch gets one resource from the pool. In the situation when there is more objects in the batch than the pool size is, deadlock happens. The reason is that the next object in the batch waits for the resource from the pool but the pool is not freed until the whole batch is processed. In other words, the code in function gst_nvinfer_process_objects should check the number of objects according to the pool size. I have found this problem just because my cluster settings weren’t perfect and the primary inference sent too many objects to secondary inference but it should have worked anyway.
There the issue is with nvinfer plugin while “output-tensor-meta" is set to 1. I face the same issue while “output_tensor_meta: true” in nvinferserver plugin config. I cannot resolve the error in the plugin as it is not public.
Kindly update the plugin to handle such cases.
Is there a way to get this bug fixed by changing the code from my end?
Thank you for the information. It is a known issue when “output_tensor_meta” enabled.
there are 2 Workarounds:
a. Increase SGIE’s max-batch-size to a larger value. e.g. max_batch_size: 30 . then you can have at most 30*10=300 faces. Note: extra.output_buffer_pool_size must be <=10.
b. Even SGIE nvinferserver is not opensouced, but PGIE nvinfer-plugin is opensouced. If 250 is acceptable. you can update nvinfer code or custom-parse to limit output faces number less than 250.
For
a. The primary model can detect up to 100 objects accurately, so to support 10 channels max_batch_size: 1000(100*10) which does not seems viable.
Also, read about extra.output_buffer_pool_size in the documentation but did not understand its purpose. Presently I have set extra.output_buffer_pool_size:10. Could you elaborate its use?
b. The model I use is a tensorflow graphdef and is not directly convertible to tensorrt plan so have to stick with nvinferserver.
Hope the issue is resolved as part of bug fixes for next release of deepstream. If there is a plan for this please let me know.
This bug is still present using NVIDIA Deepstream 6 nvinfer plugin. I have a onnx based pgie and also the sgie. Currently the onnx2trt converter does not support batch sizes larger than 1 for onnx converted models.
Then my pipeline constantly freezes as soon as the pgie detects too many objects.
Is there any work in progress to fix this? Thanks.
Hi @Fiona.Chen, I can confirm that I am encountering the same problem with DS 6.0. Could you share any timeline regarding a possible fix? This bug prevents us from using Deepstream in production after months of work and investment into this technology. We need to ingest live video streams from more than 50 live cameras placed at busy intersections. Assuming that there might be more than 10 vehicles per intersection, we would need a batch size of 500 which is unfeasible from a memory viewpoint. I understand that the tracker could possibly mitigate the issue, but we can’t risk deploying a software that could potentially freeze every few minutes.
Thank you for your help