DeepStream can't decode certain H265 streams properly

Accelerated Computing Intelligent Video Analytics DeepStream SDK
22

hi @mfoglio
please set drop-on-latency=0 in the pipeline on Jan 4, then try again. if pixelation still appears. please use the following method to get some logs.

1.  set env variable. export GST_DEBUG=3,rtpjitterbuffer:6,v4l2videodec:6,v4l2videoenc:6
2.  gst-launch-1.0 ...... >1.log 2>1.log
23

Thank you. We are testing now and we’ll let you know if that fixes it.

24

Hi @fanzh ,
We don’t have pixelation anymore when using drop-on-latency=0. However, we got a segmentation fault after a few hours:

image
image1988×190 53.7 KB

Are you aware of bugs or potential fixes?

25

Also, could you provide the commands to build and apply the fix for DeepStream 6.4? DeepStream 6.4 still freezes for one pipeline.

26

No. The previous debugging was just a test: reduce the version of h265parse. This can not be used officially, there may be a mismatch with other plugins. This may be the root cause of the crash.

We’ll analyze the issue with our hardware decoder and fix in that plugin. Please keep the stream connection you provided OK, the analyze may take some time. Or you can provide us with the stream and methods to set up the servers. Thanks

28

I will keep the video stream online.

Just to clarify : the crash above (nvv4l2decoder segmentation fault) happened inside DeepStream 6.2 .
I didn’t test with DeepStream 6.4 because the pipeline freezes with certain streams as mentioned above.

Also, the crash above happened when decoding h265 streams from an Amcrest camera (as well as some other video streams relayed by us), not when connected to the two streams I shared with you above. We will keep testing to see if the issue is actually coming from being connected to the Amcrest camera. If that’s the case, I see if we can create a public video stream for you. However, it is possible you have be able to verify the error using a live h265 camera on your end.

Let me know if you find anything,

Thank you

29

Hi @yuweiw , I have an update for you.

I tried to use the nvv4l2decoder using cuda pinned memory by changing cudadec-memtype and after a couple of days I got the following error:

Jan 22 05:50:47 b42e99fdfd74.videolink.io kernel: traps: nvv4l2decoder_2[2930321] general protection fault ip:7fbd5d697259 sp:7fb569fed650 error:0 in libgstreamer-1.0.so.0.1603.0[7fbd5d682000+b2000]

When using the cuda device memory I got the error I posted above, which is:

Jan 18 11:41:11 b42e99fdcd06.videolink.io kernel: nvv4l2decoder_1[1026634]: segfault at b028 ip 00007f7d78b03259 sp 00007f769b744650 error 4 in libgstreamer-1.0.so.0.1603.0[7f7d78aee000+b2000]

I don’t know if the errors being different is a coincidence or it is useful to you for debugging the issue.

Let me know if you were able to replicate the issue. We are now trying to understand if the issue is caused by the live cam or by the streams relayed by us.

Thanks

30

Did you run that with the libgstvideoparsersbad.so we provided? It’s just for debugging. There may be compatibility issues. Please wait until we fix the hung issue before looking at subsequent issues.
We didn’t have any problems with the crash issue in our environment.

31

Hi @yuweiw ,

I didn’t test DeepStream 6.4 with the modified libgstvideoparsersbad.so because I got an issue with it when trying to run DeepStream, as reported here: DeepStream can't decode certain H265 streams properly - #15 by mfoglio
I will be happy to test with it if you can help me set it up.
Please note that the hung issue was present only in DeepStream 6.4.

DeepStream 6.2 does not show hung issues, but the crashes above (segmentation fault and kernel trap) happened with DeepStream 6.2 . I am still testing with DeepStream 6.2 because of the hung with DeepStream 6.4 .

Thanks

32

OK.About the crash issue. Could you use gdb tool to debug that？We have not got such problems at present.
You can also check the memory when you are running the pipeline.

33

We identified the stream that was causing the issue. We will do some more testing and possibly sharing it with you.

How can I try the fix ( libgstvideoparsersbad.so ) for DeepStream 6.4? If that solves the issue, it would be even better. However, when I tried to use it, it didn’t work. I am not sure how to integrate it with DeepStream. Could you provide the complete list of commands that I need to run?

34

Hi @mfoglio , we use Gstreamer 1.20 for DeepStream 6.4 version.
The version 1.20 of Gstreamer’s h265parse plugin has some problems parsing your h265_test.mp4 stream, so the nvdecoder is hung. If you want to fix this, you need to replace the 1.16 version of the library we attached before.

sudo cp /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstvideoparsersbad.so /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstvideoparsersbad.so.new
sudo cp libgstvideoparsersbad.so /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstvideoparsersbad.so

Could you attach the h265_test.mp4 to us? You can just message to me.

35

Thank you, I will try this soon.

36

We verified that we have segmentation fault after a couple days with both DeepStream 6.2 and DeepStream 6.3 (we are using Python) when decoding live h265 streams.
We are going to test your fix for DeepStream 6.4 now.

37

As for crash issue, will both sources you provide crash when they are encoding? Or just one of them.

Also could you get some information about the crash, like cpu memory, gpu memory, log, etc…? Thanks

38

Hello, for the samples that we have provided, we are getting a general protection fault in DS 6.3. Here is one of the logs that came from it:

Feb 22 07:44:32 [DEVICE_NUMBER].videolink.io kernel: traps: nvv4l2decoder_0[8689] general protection fault ip:7f54ce8eb259 sp:7f4fdfdb5650 error:0 in libgstreamer-1.0.so.0.1603.0[7f54ce8d6000+b2000]

For the segmentation fault, we are getting those from another device running 6.3 that has live cams connected to the device ria RTSP. We are also getting general protection faults there as well. Here is a snippet from the logs:

Feb 22 18:58:29 [DEVICE_NUMBER].videolink.io kernel: nvv4l2decoder_2[3042492]: segfault at c07186 ip 00007f3f44145259 sp 00007f3685ff9650 error 4 in libgstreamer-1.0.so.0.1603.0[7f3f44130000+b2000]
Feb 18 04:30:25 [DEVICE_NUMBER].videolink.io kernel: traps: nvv4l2decoder_0[2250882] general protection fault ip:7f53818ee259 sp:7f4ba5098650 error:0 in libgstreamer-1.0.so.0.1603.0[7f53818d9000+b2000]
Feb 19 04:19:33 [DEVICE_NUMBER].videolink.io kernel: nvv4l2decoder_9[395450]: segfault at e0da ip 00007f43605f3259 sp 00007f3b1bff9650 error 4 in libgstreamer-1.0.so.0.1603.0[7f43605de000+b2000]

I can get the other information you are asking for soon. I am waiting to see if it happens again so I can get the device data when it happens

39

@yuweiw do you have any update on your side?

40
  1. The cause of the hanging issue is that your stream is not compatible with gstreamer’s latest h265parser plugin(1.20), which may require the gstreamer group to analyze together.
  2. We couldn’t repro the crash problem on our machine. We can’t go further from a few lines of log info.

We recommend that this topic only track the hanging issue. About the crash issue, you can open a new topic and describe your steps in detail.
And it is best to provide the original stream and the method of setting up the rtspserver.

41

Hi @yuweiw , I don’t think the hanging issue could be related just the h265parse as the code doesn’t hang when using the CPU decoder (avdec_h265) but just when using nvv4l2decoder.
@bkh1 will reach out to you to share the live stream regarding the crash.

42

@mfoglio @yuweiw The stream that appears to have caused the crash is the sample_720_h265.mp4 stream. According to our logs the nvv4l2decoder that presents the segfault issue points to a camera running this stream