Nvstreammux plugin's live-source option not working

gusghrlrl123 · October 20, 2020, 3:33am

Hi.

I tested live-source option for multi camera syncing.

Jetson Xavier, JetPack 4.3, DeepStream 4.0
When I tested 8 cams in gstreamer pipeline with live-source option on, it worked.

gst-launch-1.0 \
uridecodebin uri=RTSP_SRC0 ! nvvideoconvert ! m.sink_0 \
...
uridecodebin uri=RTSP_SRC7 ! nvvideoconvert ! m.sink_7 \
nvstreammux name=m width=1920 height=1080 batch-size=8 live-source=1 \
nvegltransform ! nveglglessink sync=0

Jetson Xavier NX, JetPack 4.4, DeepStream 5.0
But, when I test 4cams in gstreamer pipeline with live-source option on, it didn’t work.

gst-launch-1.0 \
uridecodebin uri=RTSP_SRC ! nvvideoconvert ! m.sink_0 \
...
uridecodebin uri=RTSP_SRC3 ! nvvideoconvert ! m.sink_3 \
nvstreammux name=m width=1920 height=1080 batch-size=4 live-source=1 \
nvegltransform ! nveglglessink sync=0

Timestamp in second case (4 cams and 8 cams 's result are same) in the below.
milli seconds are same but seconds are different.
nvstreammux plugin’s attach-sys-ts option turn off, and call configure_source_for_ntp_sync function in each sources.
[ FRAME_NUM]
CAM_NUM: NTP_TIMESTAMP
…

[ 191 ]
0: 1598357055452333334
1: 1598357055452333333
2: 1598357054452333333
3: 1598357055452333334
4: 1598357055452333333
5: 1598357054452333334
6: 1598357054452333333
7: 1598357054452333333
[ 192 ]
0: 1598357055719000000
1: 1598357055719000000
2: 1598357054719000000
3: 1598357055719000000
4: 1598357055719000000
5: 1598357054719000000
6: 1598357054719000000
7: 1598357054719000000
[ 193 ]
0: 1598357055919000000
1: 1598357055919000000
2: 1598357054919000000
3: 1598357055919000000
4: 1598357055919000000
5: 1598357054919000000
6: 1598357054919000000
7: 1598357054919000000
[ 194 ]
0: 1598357056252333334
1: 1598357056252333333
2: 1598357055252333333
3: 1598357056252333334
4: 1598357056252333333
5: 1598357055252333334
6: 1598357055252333333
7: 1598357055252333333
[ 195 ]
0: 1598357056519000000
1: 1598357056519000000
2: 1598357055519000000
3: 1598357056519000000
4: 1598357056519000000
5: 1598357055519000000
6: 1598357055519000000
7: 1598357055519000000
[ 196 ]
0: 1598357056752333334
1: 1598357056752333333
2: 1598357055752333333
3: 1598357056752333334
4: 1598357056752333333
5: 1598357055752333334
6: 1598357055752333333
7: 1598357055752333333
[ 197 ]
0: 1598357056985666667
1: 1598357056985666666
2: 1598357055985666667
3: 1598357056985666667
4: 1598357056985666666
5: 1598357055985666667
6: 1598357055985666667
7: 1598357055985666667
[ 198 ]
0: 1598357057252333334
1: 1598357057252333333
2: 1598357056252333333
3: 1598357057252333334
4: 1598357057252333333
5: 1598357056252333334
6: 1598357056252333333
7: 1598357056252333333
[ 199 ]
0: 1598357057485666667
1: 1598357057485666666
2: 1598357056485666667
3: 1598357057485666667
4: 1598357057485666666
5: 1598357056485666667
6: 1598357056485666667
7: 1598357056485666667
[ 200 ]
0: 1598357057719000000
1: 1598357057719000000
2: 1598357056719000000
3: 1598357057719000000
4: 1598357057719000000
5: 1598357056719000000
6: 1598357056719000000
7: 1598357056719000000
[ 201 ]
0: 1598357058019000000
1: 1598357058019000000
2: 1598357057019000000
3: 1598357058019000000
4: 1598357058019000000
5: 1598357057019000000
6: 1598357057019000000
7: 1598357057019000000

I used same camera for test.
In first case multi camera syncing is working, but in second case not working.
I can’t upgrade JetPack Version on Jetson Xavier, because I’m using now.
In this case, what is the problem?
Xavier NX or DeepStream 5.0?

Thank you.

Fiona.Chen · August 28, 2020, 7:09am

Can this problem be reproduced with just two rtsp streams?

gusghrlrl123 · September 3, 2020, 7:01am

I think it only works when I use 2 cams.
Below is the result according to the number of cameras in Xavier NX (JetPack 4.4).

2 cams

[ 398 ]
0: 1599108010043333333
1: 1599108010042333333
[ 399 ]
0: 1599108010076666667
1: 1599108010075666666
[ 400 ]
0: 1599108010110000000
1: 1599108010109000000
[ 401 ]
0: 1599108010143333333
1: 1599108010142333333
[ 402 ]
0: 1599108010176666667
1: 1599108010175666666

3 cams

[ 398 ]
0: 1599108070975666666
1: 1599108069975666666
2: 1599108070975666667
[ 399 ]
0: 1599108071008999999
1: 1599108070009000000
2: 1599108071009000000
[ 400 ]
0: 1599108071042333333
1: 1599108070042333333
2: 1599108071042333333
[ 401 ]
0: 1599108071075666666
1: 1599108070075666666
2: 1599108071075666667
[ 402 ]
0: 1599108071108999999
1: 1599108070109000000
2: 1599108071109000000

4 cams

[ 398 ]
0: 1599108125009000000
1: 1599108126008999999
2: 1599108126009000000
3: 1599108127009000000
[ 399 ]
0: 1599108125042333333
1: 1599108126042333333
2: 1599108126042333333
3: 1599108127042333333
[ 400 ]
0: 1599108125075666666
1: 1599108126075666666
2: 1599108126075666667
3: 1599108127075666667
[ 401 ]
0: 1599108125109000000
1: 1599108126108999999
2: 1599108126109000000
3: 1599108127109000000
[ 402 ]
0: 1599108125142333333
1: 1599108126142333333
2: 1599108126142333333
3: 1599108127142333333

5 cams

[ 398 ]
0: 1599108185142333333
1: 1599108186142333333
2: 1599108186142333333
3: 1599108186142333333
4: 1599108187142333333
[ 399 ]
0: 1599108185175666666
1: 1599108186175666666
2: 1599108186175666667
3: 1599108186175666666
4: 1599108187175666667
[ 400 ]
0: 1599108185208999999
1: 1599108186208999999
2: 1599108186209000000
3: 1599108186209000000
4: 1599108187209000000
[ 401 ]
0: 1599108185242333333
1: 1599108186242333333
2: 1599108186242333333
3: 1599108186242333333
4: 1599108187242333333
[ 402 ]
0: 1599108185275666666
1: 1599108186275666666
2: 1599108186275666667
3: 1599108186275666666
4: 1599108187275666667

6 cams

[ 398 ]
0: 1599108245775666666
1: 1599108244775666666
2: 1599108244775666666
3: 1599108245775666667
4: 1599108245775666667
5: 1599108245775666666
[ 399 ]
0: 1599108245809000000
1: 1599108244808999999
2: 1599108245809000000
3: 1599108244808999999
4: 1599108245809000000
5: 1599108245808999999
[ 400 ]
0: 1599108245842333333
1: 1599108244842333333
2: 1599108244842333333
3: 1599108245842333333
4: 1599108245842333333
5: 1599108245842333333
[ 401 ]
0: 1599108245875666666
1: 1599108244875666666
2: 1599108244875666666
3: 1599108245875666667
4: 1599108245875666667
5: 1599108245875666666
[ 402 ]
0: 1599108245909000000
1: 1599108244908999999
2: 1599108244908999999
3: 1599108245909000000
4: 1599108245909000000
5: 1599108245908999999

7 cams

[ 398 ]
0: 1599108368009000000
1: 1599108367009000000
2: 1599108368008999999
3: 1599108368008999999
4: 1599108369075666666
5: 1599108369009000000
6: 1599108369009000000
[ 399 ]
0: 1599108368042333333
1: 1599108369042333333
2: 1599108367042333333
3: 1599108368042333333
4: 1599108368042333333
5: 1599108369109000000
6: 1599108369042333333
[ 400 ]
0: 1599108368075666666
1: 1599108369075666666
2: 1599108367075666666
3: 1599108368075666666
4: 1599108368075666666
5: 1599108369142333333
6: 1599108369075666667
[ 401 ]
0: 1599108368109000000
1: 1599108367109000000
2: 1599108368108999999
3: 1599108368108999999
4: 1599108369175666666
5: 1599108369109000000
6: 1599108369109000000
[ 402 ]
0: 1599108368142333333
1: 1599108369142333333
2: 1599108367142333333
3: 1599108368142333333
4: 1599108368142333333
5: 1599108369209000000
6: 1599108369142333333

8 cams

[ 398 ]
0: 1599108431575666666
1: 1599108431675666667
2: 1599108430575666667
3: 1599108431575666666
4: 1599108431575666666
5: 1599108432575666667
6: 1599108432575666667
7: 1599108432575666667
[ 399 ]
0: 1599108431608999999
1: 1599108431709000000
2: 1599108430609000000
3: 1599108431609000000
4: 1599108431609000000
5: 1599108432609000000
6: 1599108432609000000
7: 1599108432609000000
[ 400 ]
0: 1599108431642333333
1: 1599108431742333333
2: 1599108432642333333
3: 1599108430642333333
4: 1599108431642333333
5: 1599108431642333333
6: 1599108432642333333
7: 1599108432642333333
[ 401 ]
0: 1599108431675666666
1: 1599108431775666667
2: 1599108430675666667
3: 1599108431675666666
4: 1599108431675666666
5: 1599108432675666667
6: 1599108432675666667
7: 1599108432675666667
[ 402 ]
0: 1599108431708999999
1: 1599108431809000000
2: 1599108430709000000
3: 1599108431709000000
4: 1599108431709000000
5: 1599108432709000000
6: 1599108432709000000
7: 1599108432709000000

Fiona.Chen · September 11, 2020, 2:12am

NTP can not be supported with your “gst-launch-1.0” pipelines, how do you implement the NTP functions? What have you modified and how do you get NTP timestamp from the pipeline?
DeepStream 4.0 does not support NTP while DeepStream 5.0 has implemented the function of attaching system timestamp to the batched buffer with streammux.

gusghrlrl123 · September 11, 2020, 4:02am

As I said, it happens when I use Jetson Xavier NX with DeepStream 5.0.
The pipeline above is just a sample. I use C code for implementation.
The code below is part of the code.

GstCaps* caps_filter_src = gst_caps_from_string("video/x-raw(memory:NVMM), format=RGBA");
for (int i = 0; i < CAM_NUM; ++i) {
	sources[i] = gst_element_factory_make("uridecodebin", ("camera-source-" + to_string(i)).c_str());
	g_object_set(G_OBJECT(sources[i]),
		"uri",
		rtsp_sources[i],
		nullptr);

	configure_source_for_ntp_sync(sources[i]); // this is function to set ntp option.

	nvvidconv_srcs[i] = gst_element_factory_make("nvvideoconvert", ("nvvidconv_src" + to_string(i)).c_str());

	filter_srcs[i] = gst_element_factory_make("capsfilter", ("filter_src" + to_string(i)).c_str());
	g_object_set(G_OBJECT(filter_srcs[i]), "caps", caps_filter_src, nullptr);
	g_signal_connect(sources[i], "pad-added", G_CALLBACK(newPad), nvvidconv_srcs[i]);

	gst_bin_add_many(GST_BIN(pipeline), sources[i],
		nvvidconv_srcs[i],
		filter_srcs[i],
		nullptr);
	gst_element_link_many(nvvidconv_srcs[i],
		filter_srcs[i],
		nullptr);
}
gst_caps_unref(caps_filter_src);

video_full_processing_bin = gst_bin_new("video-process-bin");

streammux = gst_element_factory_make("nvstreammux", "stream-muxer");
g_object_set(G_OBJECT(streammux),
	"width", 1920,
	"height", 1080,
	"batch-size", CAM_NUM,
	"live-source", 1,     // this is option to set live-source
	"attach-sys-ts", 0,    // this is option to use rtsp camera ntp
	nullptr);

Fiona.Chen · September 11, 2020, 5:30am

We do have configure_source_for_ntp_sync() in DeepStream 5.0. But how could you get NTP timestamp with DeepStream 4.0? You have mentioned case 1 is tested with DS 4.0.

And if you want to send batched buffer of streammux at regular interval, you can set “batch-push-timeout” property of nvstreammux. https://docs.nvidia.com/metropolis/deepstream/dev-guide/#page/DeepStream%20Plugins%20Development%20Guide/deepstream_plugin_details.html#wwpID0E04Y0HA

gusghrlrl123 · September 11, 2020, 5:00pm

In case 1, multi-camera syncing worked. (I mean there are no ntp timestamp, I just set option “live-option” true.)
When I looked at each frame one by one in my eyes, it worked.
But In case 2, when I looked at each frame one by one in my eyes, it didn’t work.
So I print out ntp timestamp. (in only DeepStream 5.0)
And there are different timestamps.

In conclusion, in DeepStream 4.0 “live-source” option worked very well as I intended.
But in DeepStream 5.0, it doesn’t work.
So I want to know what the problem is.

Fiona.Chen · September 18, 2020, 5:53am

So there is no NTP timestamp print with case 1, right?
Do you think “live-source” property with nvstreammux will make the multiple camera videos sync?

gusghrlrl123 · September 20, 2020, 7:28am

Yes. There is no NTP timestamp print with case 1.
In case 1, “live-source” property with “nvstreammux” makes the multiple camera videos sync.
In case 2, it doesn’t.

gusghrlrl123 · October 12, 2020, 2:39am

Hi.
Is there anything I can do?

Fiona.Chen · October 20, 2020, 2:24am

We will fix the problem in next release.

gusghrlrl123 · October 13, 2020, 6:59am

Ok.
Thanks for reply!

gusghrlrl123 · December 23, 2020, 7:49am

Hi.
When will the next version be released?
I want to know the approximate schedule.
Thanks.

Fiona.Chen · December 23, 2020, 7:49am

The exact date is not decided yet.

gusghrlrl123 · January 25, 2021, 12:31am

Hi.
A new JetPack version has been released, but the DeepStream version remains the same.
This problem hasn’t been solved yet, right?

Fiona.Chen · January 25, 2021, 1:01am

DeepStream is released separately.

gusghrlrl123 · March 8, 2021, 6:12am

Hi.
I heard DeepStream 5.1 version has been released. (Announcing new software update for DeepStream 5.1)
Was this problem solved here?

kayccc · March 16, 2021, 12:50am

No, it’s not at this version, but in the coming release.