How to restart inference programmatically

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

• Hardware Platform Jetson
• DeepStream Version 6.3
• JetPack Version 5.1
• TensorRT Version 5.1

Hi, I have an application that is monitoring multiple RTSP streams. Sometimes these cameras reset, but they don’t send EOS signal so my application doesn’t stop. However, when cameras get back online, Deepstream attempts restart, but fails and doesn’t start inference (FPS is always 0).
The only solution that works so far is to restart entire application, so my question is, is there a way to restart inference programmatically?

Logs:

2024-05-04 08:24:32.344 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 29.99, '00-07-5f-c8-7e-84': 29.99}***
2024-05-04 08:24:37.346 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 16.6, '00-07-5f-c8-7e-84': 16.6}***
2024-05-04 08:24:42.348 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.2}***
2024-05-04 08:24:45.599 | WARNING  | utils.bus_call:bus_call:30 - Warning: gst-stream-error-quark: No data from source since last 10 sec. Trying reconnection (1): /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvdsbins/gstdsnvurisrcbin.cpp(1266): watch_source_status (): /GstPipeline:pipeline0/GstBin:source-bin-0/GstDsNvUriSrcBin:uri-decode-bin
2024-05-04 08:24:45.600 | WARNING  | utils.probe_util:on_warning:514 - ------> uri-decode-bin
2024-05-04 08:24:47.348 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.4}***
2024-05-04 08:24:48.539 | WARNING  | utils.bus_call:bus_call:30 - Warning: gst-stream-error-quark: No data from source since last 13 sec. Trying reconnection (1): /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvdsbins/gstdsnvurisrcbin.cpp(1266): watch_source_status (): /GstPipeline:pipeline0/GstBin:source-bin-1/GstDsNvUriSrcBin:uri-decode-bin
2024-05-04 08:24:48.540 | WARNING  | utils.probe_util:on_warning:514 - ------> uri-decode-bin
2024-05-04 08:24:50.194 | WARNING  | utils.bus_call:bus_call:30 - Warning: gst-resource-error-quark: Could not open resource for reading and writing. (7): gstrtspsrc.c(7893): gst_rtspsrc_retrieve_sdp (): /GstPipeline:pipeline0/GstBin:source-bin-1/GstDsNvUriSrcBin:uri-decode-bin/GstRTSPSrc:src:
Failed to connect. (Generic error)
2024-05-04 08:24:50.196 | WARNING  | utils.probe_util:on_warning:514 - ------> src
2024-05-04 08:24:51.698 | WARNING  | utils.bus_call:bus_call:30 - Warning: gst-resource-error-quark: Could not open resource for reading and writing. (7): gstrtspsrc.c(7893): gst_rtspsrc_retrieve_sdp (): /GstPipeline:pipeline0/GstBin:source-bin-0/GstDsNvUriSrcBin:uri-decode-bin/GstRTSPSrc:src:
Failed to connect. (Generic error)
2024-05-04 08:24:51.700 | WARNING  | utils.probe_util:on_warning:514 - ------> src
2024-05-04 08:24:52.348 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.2}***
2024-05-04 08:24:57.216 | ERROR    | utils.heartbeat_util:send_fps_heartbeat:86 - fps for stream 00-07-5f-e6-cb-5e is 0!
2024-05-04 08:24:57.220 | ERROR    | utils.heartbeat_util:send_fps_heartbeat:86 - fps for stream 00-07-5f-c8-7e-84 is 0!
2024-05-04 08:24:57.349 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.2}***
2024-05-04 08:25:02.349 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.2}***
2024-05-04 08:25:07.350 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.4}***
2024-05-04 08:25:12.350 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:17.351 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:22.351 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:27.352 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:32.352 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:37.352 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:42.353 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:47.353 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:47.987 | WARNING  | utils.bus_call:bus_call:30 - Warning: gst-resource-error-quark: Could not open resource for reading and writing. (7): gstrtspsrc.c(7893): gst_rtspsrc_retrieve_sdp (): /GstPipeline:pipeline0/GstBin:source-bin-0/GstDsNvUriSrcBin:uri-decode-bin/GstRTSPSrc:src:
Failed to connect. (Generic error)
2024-05-04 08:25:47.988 | WARNING  | utils.probe_util:on_warning:514 - ------> src
2024-05-04 08:25:48.978 | WARNING  | utils.bus_call:bus_call:30 - Warning: gst-resource-error-quark: Could not open resource for reading and writing. (7): gstrtspsrc.c(7893): gst_rtspsrc_retrieve_sdp (): /GstPipeline:pipeline0/GstBin:source-bin-1/GstDsNvUriSrcBin:uri-decode-bin/GstRTSPSrc:src:
Failed to connect. (Generic error)
2024-05-04 08:25:48.979 | WARNING  | utils.probe_util:on_warning:514 - ------> src
2024-05-04 08:25:52.353 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:25:57.354 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:02.355 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:05.311 | ERROR    | save_img:_reader:32 - Terminate camera thread is set!
2024-05-04 08:26:07.355 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:12.356 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:17.357 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:22.358 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:27.358 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:32.359 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:37.359 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:42.360 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:45.703 | INFO     | utils.deepstream_util:decodebin_child_added:52 - Decodebin child added: h264parse2
2024-05-04 08:26:45.704 | INFO     | utils.deepstream_util:decodebin_child_added:53 - child proxy: <__gi__.GstDecodeBin object at 0xffff93a96d40 (GstDecodeBin at 0xfffe24060b10)>
2024-05-04 08:26:45.704 | INFO     | utils.deepstream_util:decodebin_child_added:54 - object: <__gi__.GstH264Parse object at 0xffff97b636c0 (GstH264Parse at 0xfffe10085260)>
2024-05-04 08:26:45.704 | INFO     | utils.deepstream_util:decodebin_child_added:55 - user data: <Gst.Bin object at 0xffff938df200 (GstBin at 0x3b6360d0)>
2024-05-04 08:26:45.705 | INFO     | utils.deepstream_util:decodebin_child_added:52 - Decodebin child added: capsfilter2
2024-05-04 08:26:45.706 | INFO     | utils.deepstream_util:decodebin_child_added:53 - child proxy: <__gi__.GstDecodeBin object at 0xffff93a96d40 (GstDecodeBin at 0xfffe24060b10)>
2024-05-04 08:26:45.706 | INFO     | utils.deepstream_util:decodebin_child_added:54 - object: <__gi__.GstCapsFilter object at 0xffff97b636c0 (GstCapsFilter at 0xfffe1c1af220)>
2024-05-04 08:26:45.707 | INFO     | utils.deepstream_util:decodebin_child_added:55 - user data: <Gst.Bin object at 0xffff938df200 (GstBin at 0x3b6360d0)>
2024-05-04 08:26:45.711 | INFO     | utils.deepstream_util:decodebin_child_added:52 - Decodebin child added: nvv4l2decoder2
2024-05-04 08:26:45.712 | INFO     | utils.deepstream_util:decodebin_child_added:53 - child proxy: <__gi__.GstDecodeBin object at 0xffff93a96d40 (GstDecodeBin at 0xfffe24060b10)>
2024-05-04 08:26:45.712 | INFO     | utils.deepstream_util:decodebin_child_added:54 - object: <__gi__.nvv4l2decoder object at 0xffff97b636c0 (nvv4l2decoder at 0xfffe100a3a30)>
2024-05-04 08:26:45.712 | INFO     | utils.deepstream_util:decodebin_child_added:55 - user data: <Gst.Bin object at 0xffff938df200 (GstBin at 0x3b6360d0)>
2024-05-04 08:26:47.360 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:52.361 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.2, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:26:57.362 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.2, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:27:02.362 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.4, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:28:58.001 | WARNING  | utils.bus_call:bus_call:30 - Warning: gst-stream-error-quark: No data from source since last 10 sec. Trying reconnection (1): /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvdsbins/gstdsnvurisrcbin.cpp(1266): watch_source_status (): /GstPipeline:pipeline0/GstBin:source-bin-0/GstDsNvUriSrcBin:uri-decode-bin
2024-05-04 08:28:58.002 | WARNING  | utils.probe_util:on_warning:514 - ------> uri-decode-bin
2024-05-04 08:28:58.002 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.25, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:03.001 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.2, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:08.002 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.2, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:13.002 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.2, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:18.003 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.4, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:23.003 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:28.003 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:33.004 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:38.004 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
2024-05-04 08:29:43.005 | INFO     | utils.FPS:perf_print_callback:101 - ***PERF: {'00-07-5f-e6-cb-5e': 0.0, '00-07-5f-c8-7e-84': 0.0}***
3

Hi, In our DeepStream applications, we typically use GstD and Interpipes.

GstD enables us to control various pipelines programmatically using a client available in Python, C, HTTP, Bash, etc. It ensures thread safety and facilitates communication between pipelines, even if they are executed from different processes since they all operate within the GstD process.

Interpipes allows us to divide different functionalities into distinct pipelines and connect multiple listeners to a single producer asynchronously through interpipesink and interpipesrc.

To solve your issue, we use GstD to monitor the GStreamer bus on the source pipeline. When the rtspsrc element triggers a disconnection message, GstD enables us to restart only the inference pipeline via the gstd client once the source is back online.

We have a sample DeepStream media server that uses GstD and Interpipes, in case you want more information.

1 Like
4

Just from the log attached, the rtsp source cannot connect to the new source.

2024-05-04 08:25:47.987 | WARNING  | utils.bus_call:bus_call:30 - Warning: gst-resource-error-quark: Could not open resource for reading and writing. (7): gstrtspsrc.c(7893): gst_rtspsrc_retrieve_sdp (): /GstPipeline:pipeline0/GstBin:source-bin-0/GstDsNvUriSrcBin:uri-decode-bin/GstRTSPSrc:src:
Failed to connect. (Generic error)

Gstreamer source code: gstrtspsrc.c

It is possible that your camera reset changed something, causing the rtsp source of gstreamer cannot be reconnected. You can simplify your pipeline to just rtspsrc->decoder->sink to narrow down this issue and file a topic to Gstreamer Forum.

1 Like
5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.