Nvstreammux plugin's live-source option not working

Hi.

I tested live-source option for multi camera syncing.

  1. Jetson Xavier, JetPack 4.3, DeepStream 4.0
    When I tested 8 cams in gstreamer pipeline with live-source option on, it worked.
gst-launch-1.0 \
uridecodebin uri=RTSP_SRC0 ! nvvideoconvert ! m.sink_0 \
...
uridecodebin uri=RTSP_SRC7 ! nvvideoconvert ! m.sink_7 \
nvstreammux name=m width=1920 height=1080 batch-size=8 live-source=1 \
nvegltransform ! nveglglessink sync=0
  1. Jetson Xavier NX, JetPack 4.4, DeepStream 5.0
    But, when I test 4cams in gstreamer pipeline with live-source option on, it didn’t work.
gst-launch-1.0 \
uridecodebin uri=RTSP_SRC ! nvvideoconvert ! m.sink_0 \
...
uridecodebin uri=RTSP_SRC3 ! nvvideoconvert ! m.sink_3 \
nvstreammux name=m width=1920 height=1080 batch-size=4 live-source=1 \
nvegltransform ! nveglglessink sync=0

Timestamp in second case (4 cams and 8 cams 's result are same) in the below.
milli seconds are same but seconds are different.
nvstreammux plugin’s attach-sys-ts option turn off, and call configure_source_for_ntp_sync function in each sources.
[ FRAME_NUM]
CAM_NUM: NTP_TIMESTAMP

[ 191 ]
0: 1598357055452333334
1: 1598357055452333333
2: 1598357054452333333
3: 1598357055452333334
4: 1598357055452333333
5: 1598357054452333334
6: 1598357054452333333
7: 1598357054452333333
[ 192 ]
0: 1598357055719000000
1: 1598357055719000000
2: 1598357054719000000
3: 1598357055719000000
4: 1598357055719000000
5: 1598357054719000000
6: 1598357054719000000
7: 1598357054719000000
[ 193 ]
0: 1598357055919000000
1: 1598357055919000000
2: 1598357054919000000
3: 1598357055919000000
4: 1598357055919000000
5: 1598357054919000000
6: 1598357054919000000
7: 1598357054919000000
[ 194 ]
0: 1598357056252333334
1: 1598357056252333333
2: 1598357055252333333
3: 1598357056252333334
4: 1598357056252333333
5: 1598357055252333334
6: 1598357055252333333
7: 1598357055252333333
[ 195 ]
0: 1598357056519000000
1: 1598357056519000000
2: 1598357055519000000
3: 1598357056519000000
4: 1598357056519000000
5: 1598357055519000000
6: 1598357055519000000
7: 1598357055519000000
[ 196 ]
0: 1598357056752333334
1: 1598357056752333333
2: 1598357055752333333
3: 1598357056752333334
4: 1598357056752333333
5: 1598357055752333334
6: 1598357055752333333
7: 1598357055752333333
[ 197 ]
0: 1598357056985666667
1: 1598357056985666666
2: 1598357055985666667
3: 1598357056985666667
4: 1598357056985666666
5: 1598357055985666667
6: 1598357055985666667
7: 1598357055985666667
[ 198 ]
0: 1598357057252333334
1: 1598357057252333333
2: 1598357056252333333
3: 1598357057252333334
4: 1598357057252333333
5: 1598357056252333334
6: 1598357056252333333
7: 1598357056252333333
[ 199 ]
0: 1598357057485666667
1: 1598357057485666666
2: 1598357056485666667
3: 1598357057485666667
4: 1598357057485666666
5: 1598357056485666667
6: 1598357056485666667
7: 1598357056485666667
[ 200 ]
0: 1598357057719000000
1: 1598357057719000000
2: 1598357056719000000
3: 1598357057719000000
4: 1598357057719000000
5: 1598357056719000000
6: 1598357056719000000
7: 1598357056719000000
[ 201 ]
0: 1598357058019000000
1: 1598357058019000000
2: 1598357057019000000
3: 1598357058019000000
4: 1598357058019000000
5: 1598357057019000000
6: 1598357057019000000
7: 1598357057019000000

I used same camera for test.
In first case multi camera syncing is working, but in second case not working.
I can’t upgrade JetPack Version on Jetson Xavier, because I’m using now.
In this case, what is the problem?
Xavier NX or DeepStream 5.0?

Thank you.

Can this problem be reproduced with just two rtsp streams?

I think it only works when I use 2 cams.
Below is the result according to the number of cameras in Xavier NX (JetPack 4.4).

2 cams

[ 398 ]
0: 1599108010043333333
1: 1599108010042333333
[ 399 ]
0: 1599108010076666667
1: 1599108010075666666
[ 400 ]
0: 1599108010110000000
1: 1599108010109000000
[ 401 ]
0: 1599108010143333333
1: 1599108010142333333
[ 402 ]
0: 1599108010176666667
1: 1599108010175666666

3 cams

[ 398 ]
0: 1599108070975666666
1: 1599108069975666666
2: 1599108070975666667
[ 399 ]
0: 1599108071008999999
1: 1599108070009000000
2: 1599108071009000000
[ 400 ]
0: 1599108071042333333
1: 1599108070042333333
2: 1599108071042333333
[ 401 ]
0: 1599108071075666666
1: 1599108070075666666
2: 1599108071075666667
[ 402 ]
0: 1599108071108999999
1: 1599108070109000000
2: 1599108071109000000

4 cams

[ 398 ]
0: 1599108125009000000
1: 1599108126008999999
2: 1599108126009000000
3: 1599108127009000000
[ 399 ]
0: 1599108125042333333
1: 1599108126042333333
2: 1599108126042333333
3: 1599108127042333333
[ 400 ]
0: 1599108125075666666
1: 1599108126075666666
2: 1599108126075666667
3: 1599108127075666667
[ 401 ]
0: 1599108125109000000
1: 1599108126108999999
2: 1599108126109000000
3: 1599108127109000000
[ 402 ]
0: 1599108125142333333
1: 1599108126142333333
2: 1599108126142333333
3: 1599108127142333333

5 cams

[ 398 ]
0: 1599108185142333333
1: 1599108186142333333
2: 1599108186142333333
3: 1599108186142333333
4: 1599108187142333333
[ 399 ]
0: 1599108185175666666
1: 1599108186175666666
2: 1599108186175666667
3: 1599108186175666666
4: 1599108187175666667
[ 400 ]
0: 1599108185208999999
1: 1599108186208999999
2: 1599108186209000000
3: 1599108186209000000
4: 1599108187209000000
[ 401 ]
0: 1599108185242333333
1: 1599108186242333333
2: 1599108186242333333
3: 1599108186242333333
4: 1599108187242333333
[ 402 ]
0: 1599108185275666666
1: 1599108186275666666
2: 1599108186275666667
3: 1599108186275666666
4: 1599108187275666667

6 cams

[ 398 ]
0: 1599108245775666666
1: 1599108244775666666
2: 1599108244775666666
3: 1599108245775666667
4: 1599108245775666667
5: 1599108245775666666
[ 399 ]
0: 1599108245809000000
1: 1599108244808999999
2: 1599108245809000000
3: 1599108244808999999
4: 1599108245809000000
5: 1599108245808999999
[ 400 ]
0: 1599108245842333333
1: 1599108244842333333
2: 1599108244842333333
3: 1599108245842333333
4: 1599108245842333333
5: 1599108245842333333
[ 401 ]
0: 1599108245875666666
1: 1599108244875666666
2: 1599108244875666666
3: 1599108245875666667
4: 1599108245875666667
5: 1599108245875666666
[ 402 ]
0: 1599108245909000000
1: 1599108244908999999
2: 1599108244908999999
3: 1599108245909000000
4: 1599108245909000000
5: 1599108245908999999

7 cams

[ 398 ]
0: 1599108368009000000
1: 1599108367009000000
2: 1599108368008999999
3: 1599108368008999999
4: 1599108369075666666
5: 1599108369009000000
6: 1599108369009000000
[ 399 ]
0: 1599108368042333333
1: 1599108369042333333
2: 1599108367042333333
3: 1599108368042333333
4: 1599108368042333333
5: 1599108369109000000
6: 1599108369042333333
[ 400 ]
0: 1599108368075666666
1: 1599108369075666666
2: 1599108367075666666
3: 1599108368075666666
4: 1599108368075666666
5: 1599108369142333333
6: 1599108369075666667
[ 401 ]
0: 1599108368109000000
1: 1599108367109000000
2: 1599108368108999999
3: 1599108368108999999
4: 1599108369175666666
5: 1599108369109000000
6: 1599108369109000000
[ 402 ]
0: 1599108368142333333
1: 1599108369142333333
2: 1599108367142333333
3: 1599108368142333333
4: 1599108368142333333
5: 1599108369209000000
6: 1599108369142333333

8 cams

[ 398 ]
0: 1599108431575666666
1: 1599108431675666667
2: 1599108430575666667
3: 1599108431575666666
4: 1599108431575666666
5: 1599108432575666667
6: 1599108432575666667
7: 1599108432575666667
[ 399 ]
0: 1599108431608999999
1: 1599108431709000000
2: 1599108430609000000
3: 1599108431609000000
4: 1599108431609000000
5: 1599108432609000000
6: 1599108432609000000
7: 1599108432609000000
[ 400 ]
0: 1599108431642333333
1: 1599108431742333333
2: 1599108432642333333
3: 1599108430642333333
4: 1599108431642333333
5: 1599108431642333333
6: 1599108432642333333
7: 1599108432642333333
[ 401 ]
0: 1599108431675666666
1: 1599108431775666667
2: 1599108430675666667
3: 1599108431675666666
4: 1599108431675666666
5: 1599108432675666667
6: 1599108432675666667
7: 1599108432675666667
[ 402 ]
0: 1599108431708999999
1: 1599108431809000000
2: 1599108430709000000
3: 1599108431709000000
4: 1599108431709000000
5: 1599108432709000000
6: 1599108432709000000
7: 1599108432709000000

NTP can not be supported with your “gst-launch-1.0” pipelines, how do you implement the NTP functions? What have you modified and how do you get NTP timestamp from the pipeline?
DeepStream 4.0 does not support NTP while DeepStream 5.0 has implemented the function of attaching system timestamp to the batched buffer with streammux.

As I said, it happens when I use Jetson Xavier NX with DeepStream 5.0.
The pipeline above is just a sample. I use C code for implementation.
The code below is part of the code.

GstCaps* caps_filter_src = gst_caps_from_string("video/x-raw(memory:NVMM), format=RGBA");
for (int i = 0; i < CAM_NUM; ++i) {
	sources[i] = gst_element_factory_make("uridecodebin", ("camera-source-" + to_string(i)).c_str());
	g_object_set(G_OBJECT(sources[i]),
		"uri",
		rtsp_sources[i],
		nullptr);

	configure_source_for_ntp_sync(sources[i]); // this is function to set ntp option.

	nvvidconv_srcs[i] = gst_element_factory_make("nvvideoconvert", ("nvvidconv_src" + to_string(i)).c_str());

	filter_srcs[i] = gst_element_factory_make("capsfilter", ("filter_src" + to_string(i)).c_str());
	g_object_set(G_OBJECT(filter_srcs[i]), "caps", caps_filter_src, nullptr);
	g_signal_connect(sources[i], "pad-added", G_CALLBACK(newPad), nvvidconv_srcs[i]);

	gst_bin_add_many(GST_BIN(pipeline), sources[i],
		nvvidconv_srcs[i],
		filter_srcs[i],
		nullptr);
	gst_element_link_many(nvvidconv_srcs[i],
		filter_srcs[i],
		nullptr);
}
gst_caps_unref(caps_filter_src);

video_full_processing_bin = gst_bin_new("video-process-bin");

streammux = gst_element_factory_make("nvstreammux", "stream-muxer");
g_object_set(G_OBJECT(streammux),
	"width", 1920,
	"height", 1080,
	"batch-size", CAM_NUM,
	"live-source", 1,     // this is option to set live-source
	"attach-sys-ts", 0,    // this is option to use rtsp camera ntp
	nullptr);

We do have configure_source_for_ntp_sync() in DeepStream 5.0. But how could you get NTP timestamp with DeepStream 4.0? You have mentioned case 1 is tested with DS 4.0.

And if you want to send batched buffer of streammux at regular interval, you can set “batch-push-timeout” property of nvstreammux. https://docs.nvidia.com/metropolis/deepstream/dev-guide/#page/DeepStream%20Plugins%20Development%20Guide/deepstream_plugin_details.html#wwpID0E04Y0HA

In case 1, multi-camera syncing worked. (I mean there are no ntp timestamp, I just set option “live-option” true.)
When I looked at each frame one by one in my eyes, it worked.
But In case 2, when I looked at each frame one by one in my eyes, it didn’t work.
So I print out ntp timestamp. (in only DeepStream 5.0)
And there are different timestamps.

In conclusion, in DeepStream 4.0 “live-source” option worked very well as I intended.
But in DeepStream 5.0, it doesn’t work.
So I want to know what the problem is.

So there is no NTP timestamp print with case 1, right?
Do you think “live-source” property with nvstreammux will make the multiple camera videos sync?

Yes. There is no NTP timestamp print with case 1.
In case 1, “live-source” property with “nvstreammux” makes the multiple camera videos sync.
In case 2, it doesn’t.

Hi.
Is there anything I can do?

We will fix the problem in next release.

Ok.
Thanks for reply!

Hi.
When will the next version be released?
I want to know the approximate schedule.
Thanks.

The exact date is not decided yet.

Hi.
A new JetPack version has been released, but the DeepStream version remains the same.
This problem hasn’t been solved yet, right?

DeepStream is released separately.

1 Like

Hi.
I heard DeepStream 5.1 version has been released. (Announcing new software update for DeepStream 5.1)
Was this problem solved here?

No, it’s not at this version, but in the coming release.

1 Like