Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU) T4 • DeepStream Version 5.1 • JetPack Version (valid for Jetson only) • TensorRT Version 7.2 (from 5.1 NGC container) • NVIDIA GPU Driver Version (valid for GPU only) 470.82.01 • Issue Type( questions, new requirements, bugs) possibly a bug? • How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing) • Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)
Hi All,
I’m looking for some help to explain the situation below. It’s Deepstream app build on top of test5 app with 3 rtsp sources. I’m not even sure what exactly happened here. It looks like the cameras got disconnected for a while. But why, when the error is thrown, the app still keeps operating for hours even though there appears to be zero input? How can I tell if the sources have actually reconnected properly? Are there some tools to detect such a drop in the framerate and notify the users?
The log has shown you there is no data received from the RTSP server for several seconds, so the client(deepstream app) tried to reconnect to the server.
Blockquote
The log has shown you there is no data received from the RTSP server for several seconds, so the client(deepstream app) tried to reconnect to the server.
But I don’t think your reply fully answers my questions from above. What you wrote is the part that I was able to figure out myself.
But why the deepstream app keeps running as if nothing happened even though it claims to be receiving no frames at all?
In the logs we see that 0 FPS continues for hours but the app never tries to reconnect again.
The deepstream-app is just a sample code and totally open source. The reconnection is implemented in deepstream-app source code. For different users, they may need different error handling methods when the rtsp server is out of reach. You can modify the reconnection code to try reconnection as many times as you want, or modify the code to exit the app, or modify the code to finish current session and restart a new session. You can design the application as you like.
So the behaviour that I showed in the first post is what you would expect? rtsp-reconnect-interval-sec was not set at all in sources config group in our case. Is it taking some default value if not set explicitly by the user? If so, what to what value? It doesn’t seem to be in the open source part of the code, but this warning is thrown: NVGSTDS_WARN_MSG_V (“No data from source %d since last %u sec. Trying reconnection”,
In DS5.1 I see the below responsible for checking the stream status:
/**
* Function called at regular interval to check if NV_DS_SOURCE_RTSP type
* source in the pipeline is down / disconnected. This function try to
* reconnect the source by resetting that source pipeline.
*/
static gboolean
watch_source_status (gpointer data)
{
NvDsSrcBin *src_bin = (NvDsSrcBin *) data;
struct timeval current_time;
gettimeofday (¤t_time, NULL);
static struct timeval last_reset_time_global = {0, 0};
gdouble time_diff_msec_since_last_reset =
1000.0 * (current_time.tv_sec - last_reset_time_global.tv_sec) +
(current_time.tv_usec - last_reset_time_global.tv_usec) / 1000.0;
if (src_bin->reconfiguring) {
guint time_since_last_reconnect_sec =
current_time.tv_sec - src_bin->last_reconnect_time.tv_sec;
if (time_since_last_reconnect_sec >= SOURCE_RESET_INTERVAL_SEC) {
if (time_diff_msec_since_last_reset > 3000) {
last_reset_time_global = current_time;
// source is still not up, reconfigure it again.
reset_source_pipeline (src_bin);
}
}
} else {
gint time_since_last_buf_sec = 0;
g_mutex_lock (&src_bin->bin_lock);
if (src_bin->last_buffer_time.tv_sec != 0) {
time_since_last_buf_sec =
current_time.tv_sec - src_bin->last_buffer_time.tv_sec;
}
g_mutex_unlock (&src_bin->bin_lock);
// Reset source bin if no buffers are received in the last
// `rtsp_reconnect_interval_sec` seconds.
if (src_bin->rtsp_reconnect_interval_sec > 0 &&
time_since_last_buf_sec >= src_bin->rtsp_reconnect_interval_sec) {
if (time_diff_msec_since_last_reset > 3000) {
last_reset_time_global = current_time;
NVGSTDS_WARN_MSG_V ("No data from source %d since last %u sec. Trying reconnection",
src_bin->bin_id, time_since_last_buf_sec);
reset_source_pipeline (src_bin);
}
}
}
return TRUE;
}
As this is DS5.1, the rtsp-reconnect-attempts is not available. I’m far from an expert, but shouldn’t the watch_source_status report more attempts to reconnect?
Please refer to latest DeepStreamSDK 6.0.1 source code. It is a must to be familiar with gstreamer knowledge and coding skills before you start with deepstream.