deepstream-test4-app stalled after the first few frames

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Trying deepstream-test4-app in ds3.0 but couldn’t get any message on the analytic server side. I was running analytic server from the 360d smart parking package and changed CONNECTION_STRING in deepstream_test4_app.c to match the kafka in analytic server.

~/DeepStream_Release/sources/apps/sample_apps/deepstream-test4$ ./deepstream-test4-app ~/DeepStream_Release/samples/streams/sample_720p.h264
0:00:00.094144803 2837 0x1aa9b50 DEBUG nvmsgconv gstnvmsgconv.c:180:gst_nvmsgconv_set_property: set_property
0:00:00.094193132 2837 0x1aa9b50 DEBUG nvmsgbroker gstnvmsgbroker.c:172:gst_nvmsgbroker_set_property: set_property
0:00:00.094205983 2837 0x1aa9b50 DEBUG nvmsgbroker gstnvmsgbroker.c:172:gst_nvmsgbroker_set_property: set_property
Now playing: /home/ubuntu/DeepStream_Release/samples/streams/sample_720p.h264
0:00:00.670736014 2837 0x1aa9b50 DEBUG nvmsgbroker gstnvmsgbroker.c:268:gst_nvmsgbroker_start: start
0:00:02.401598580 2837 0x1aa9b50 DEBUG nvmsgconv gstnvmsgconv.c:262:gst_nvmsgconv_start: start

Generating new TRT model engine
Using INT8 data type.

***** Storing serialized engine file as /home/ubuntu/DeepStream_Release/sources/apps/sample_apps/deepstream-test4/…/…/…/…/samples/models/Primary_Detector/resnet10.caffemodel_b1_int8.engine batchsize = 1 *****

Running…
0:00:19.504075576 2837 0x1a2e320 DEBUG nvmsgconv gstnvmsgconv.c:251:gst_nvmsgconv_set_caps: set_caps
0:00:19.504152678 2837 0x1a2e320 DEBUG nvmsgbroker gstnvmsgbroker.c:256:gst_nvmsgbroker_set_caps: set_caps
Frame Number = 0 Number of objects = 5 Vehicle Count = 3 Person Count = 2
0:00:19.680792247 2837 0x1a2d280 LOG nvosd gstnvosd.c:220:gst_nv_osd_transform_ip: SETTING CUDA DEVICE = 0 in nvosd func=gst_nv_osd_transform_ip

0:00:19.692914678 2837 0x1a2e320 DEBUG nvmsgconv gstnvmsgconv.c:338:gst_nvmsgconv_transform_ip: transform_ip
Frame Number = 1 Number of objects = 4 Vehicle Count = 2 Person Count = 2
0:00:19.693673412 2837 0x1a2d280 LOG nvosd gstnvosd.c:220:gst_nv_osd_transform_ip: SETTING CUDA DEVICE = 0 in nvosd func=gst_nv_osd_transform_ip

Frame Number = 2 Number of objects = 5 Vehicle Count = 3 Person Count = 2
0:00:19.695390621 2837 0x1a2d280 LOG nvosd gstnvosd.c:220:gst_nv_osd_transform_ip: SETTING CUDA DEVICE = 0 in nvosd func=gst_nv_osd_transform_ip

Frame Number = 3 Number of objects = 6 Vehicle Count = 3 Person Count = 3
0:00:19.697151690 2837 0x1a2d280 LOG nvosd gstnvosd.c:220:gst_nv_osd_transform_ip: SETTING CUDA DEVICE = 0 in nvosd func=gst_nv_osd_transform_ip

Then the program just stalled. At analytic server, started tcpdump on port 9092 and got some traffic, none related to detected objects though. Any idea how to resolve this issue? Thanks!

01:51:39.418772 IP host-10-10-199-53.openstacklocal.9092 > host-10-10-199-58.openstacklocal.33756: Flags [P.], seq 389:505, ack 106, win 14480, options [nop,nop,TS val 5801506 ecr 2654586], length 116
	0x0000:  4500 00a8 3504 4000 3f06 63c8 0a0a c735  E...5.@.?.c....5
	0x0010:  0a0a c73a 2384 83dc e8ff 55af 1c7c 094c  ...:#.....U..|.L
	0x0020:  8018 3890 a31e 0000 0101 080a 0058 8622  ..8..........X."
	0x0030:  0028 817a 0000 0070 0000 0003 0000 0001  .(.z...p........
	0x0040:  0000 03e9 000c 3130 2e31 302e 3139 392e  ......10.10.199.
	0x0050:  3533 0000 2384 ffff 0016 5466 6161 792d  53..#.....Tfaay-
	0x0060:  412d 536d 4370 5855 5763 7432 6569 6451  A-SmCpXUWct2eidQ
	0x0070:  0000 03e9 0000 0001 0000 000d 6d65 7472  ............metr
	0x0080:  6f6d 696e 642d 7261 7700 0000 0001 0000  omind-raw.......
	0x0090:  0000 0000 0000 03e9 0000 0001 0000 03e9  ................
	0x00a0:  0000 0001 0000 03e9                      ........
01:51:39.419226 IP host-10-10-199-58.openstacklocal.33756 > host-10-10-199-53.openstacklocal.9092: Flags <li>, ack 505, win 237, options [nop,nop,TS val 2654586 ecr 5801506], length 0
	0x0000:  4500 0034 5467 4000 4006 43d9 0a0a c73a  E..4Tg@.@.C....:
	0x0010:  0a0a c735 83dc 2384 1c7c 094c e8ff 5623  ...5..#..|.L..V#
	0x0020:  8010 00ed a2aa 0000 0101 080a 0028 817a  .............(.z
	0x0030:  0058 8622                                .X."
2

Hi, Scott
Do you run deepstream-test4 with GST_DEBUG=5 set? if yes, can you lower down the level and try again?

3

Amy,

Thanks for the reply! I ran with GST_DEBUG=“nvmsgbroker:6,nvmsgconv:6,nvosd:6”. What debug config do you recommend?

I am using ubuntu 16.04 VM without GUI, would that cause a problem like this?

4

Without any debug setting:

~/DeepStream_Release/sources/apps/sample_apps/deepstream-test4$ ./deepstream-test4-app ~/DeepStream_Release/samples/streams/sample_720p.h264
Now playing: /home/ubuntu/DeepStream_Release/samples/streams/sample_720p.h264
>>> Generating new TRT model engine
Using INT8 data type.

 ***** Storing serialized engine file as /home/ubuntu/DeepStream_Release/sources/apps/sample_apps/deepstream-test4/../../../../samples/models/Primary_Detector/resnet10.caffemodel_b1_int8.engine batchsize = 1 *****

Running...
Frame Number = 0 Number of objects = 5 Vehicle Count = 3 Person Count = 2
Frame Number = 1 Number of objects = 4 Vehicle Count = 2 Person Count = 2
Frame Number = 2 Number of objects = 5 Vehicle Count = 3 Person Count = 2
Frame Number = 3 Number of objects = 6 Vehicle Count = 3 Person Count = 3
5

Hi,
Which card you are using? test4 app use nveglglessink as output, which requires nvidia x working,
if you do not have nvidia display card, you can choose change test4 app code nveglglessink to fakesink
and try again.

6

I am able to get messages at the analytic server after changing to fakesink. Thanks!

7

i have the same situation with this,my pipeline stalled after the first few frames,and also ended with fakesink.https://devtalk.nvidia.com/default/topic/1048680/deepstream-for-tesla/gst_base_sink_wait_preroll/
can u help?this is someone reply http://gstreamer-devel.966125.n4.nabble.com/gst-base-sink-wait-preroll-tt4690015.html#none

8

I have the same issue… first few frames are analyzed, a kafka msg is produced, and the video pipe stalls, no info messages from gstreamer after that point using GST_DEBUG=5

I am using the nveglglessink in an X desktop and the first few annotated frames are displayed correctly.

Changing nveglglessink to fakesink does not seem to alleviate the problem.

9
10

amycao, you quoted me but im not sure what value you were trying to add.
The line you added is exactly how i ran my app.
It runs for a few frames and the pipeline stalls.

It would be really great if there were some more information on this issue from nvidia!

11

Hi Jeffrey
I mean using GST_DEBUG=5 “your app” to get more log for analysis, if with high level message
output, app run into some error or something similar, please downgrade the level, let’s say GST_DEBUG=4 and try again.

12

Hi amycao,

Thanks for responding. I have tried running the application using GST_DEBUG=4 and GST_DEBUG=5. Using those log levels changes the amount of logs produced, but, once the pipeline stalls… NO messages are produced.

This will be the 12th message in this thread, hopefully someone can post some relevant debugging techniques for the stalled pipeline, other than playing with the debug levels and using a fakesink. Deepstream borders on useless if we cannot get the analytic data out of the pipeline.

Thanks in advance!

13

Hi,
why you think debug log not helpful? I actually mean got all the logs for analysis, can you paste the logs here for analysis?

14

They are not helpful because they don’t show any errors, just the output of the analysis for the first few frames, and then no output, regardless of log level. My logs look exactly like the ones already posted here. Only difference is using a fakesink does not free up the pipe for me.

15

Hi Jeffrey
May i know which GPU are you using?

16

Quadro p4000 locally; t4 in cloud

17

which one you used have the stalled issue?

18

Both