[Gstreamer] stream drops a lot of frames after some hours later only with JVC camera

[10Key]

Hello.

We are having an issue to display JVC camera’s H.264 stream by gstreamer.
Stream seems fine at first, but about 4 hours later it starts dropping frames from queue.

We have used the gstreamer pipeline below to receive the stream:

gst-launch-1.0 rtspsrc location="<JVC camera url>" latency=500 
! rtph264depay 
! queue max-size-time=2000000000 max-size-buffers=0 max-size-bytes=0 leaky=2 
! h264parse ! omxh264dec 
! queue max-size-time=0 max-size-buffers=4 max-size-bytes=0 leaky=2 
! nvvidconv top=0 left=0 right=1920 bottom=1080 
! video/x-raw, width=1920, height=1080, format=YUY2 
! xvimagesink sync=false max-lateness=5000000 qos=true async=false

About 4 hours later, in the queue element between rtph264depay and h264parse, frames starts dropping because buffers crosses a threshold.

We have confirmed that this issue happens only when gstreamer is receiving H.264 stream from JVC cameras.
We have also tried these below, but not improved.

• Use nvv4l2decoder instead of omxh264dec.
• Increase the amount of queue buffer’s threshold.

On the other hand, streams looks fine on the camera web UI even if this problem is happening.

From these results, our suspicion is that gstreamer plugin omxh264dec and nvv4l2decoder do not work correctly only in this case.
Are these any known issues or compatibility problems?
And we also want to solve this problem. So Could you please assist?

Setup:

• Jetson TX2 [Revision 32.7.3]
• Gstreamer 1.16.2

Thanks in advance.

[DaneLLL]

Hi,
We have deprecated omx plugin. Please use nvv4l2decoder and enabler the property:

  enable-max-performance: Set to enable max performance
                        flags: readable, writable
                        Boolean. Default: false

And default gstreamer version is 1.14.5. You may try the default version and see it is more stable. The default system image has done and passed SQA tests before it is published.

[10Key]

Hello.

Thanks for reply, we tried the option you mentioned by enabling enable-max-performance property.
We’re not sure what this property does to be honest, but the problem didn’t get improved.

We got these logs only after pipeline starts dropping frames.

0:34:15.321717472  9933   0x7f600736d0 WARN               h264parse gsth264parse.c:1351:gst_h264_parse_handle_frame:<h264parse> broken/invalid nal Type: 1 Slice, 
Size: 4473 will be dropped
0:34:15.321993504  9933   0x7f600736d0 WARN               h264parse gsth264parse.c:1351:gst_h264_parse_handle_frame:<h264parse> broken/invalid nal Type: 1 Slice, 
Size: 5429 will be dropped
0:34:15.322148384  9933   0x7f600736d0 WARN               h264parse gsth264parse.c:1351:gst_h264_parse_handle_frame:<h264parse> broken/invalid nal Type: 1 Slice, 
Size: 5041 will be dropped

This is the warning of h264parse , so we don’t think this parse element is also working well.

We also tested this pipeline on Orin AGX (Revision 35.3.1, gstreamer 1.16.3), and this issue did not happen.

We haven’t tested on the default gstreamer version yet.

Are there any other suggestions to solve this?

[DaneLLL]

Hi,
The issue looks to be in rtspsrc ! h264parse of 1.14.5. Looks like it does not handle the packets well and cannot send valid h264 stream to next nvv4l2decoder. One possible solution is to manually build 1.16.3 and upgrade to the version for a try. Please refer to guidance in developer guide

Most gstreamer plugins are open source in
Jetson Linux R32.7.4 | NVIDIA Developer

Driver Package (BSP) Sources

You should also need to rebuild the plugins.

Another possible solution it to check the fix in rtspsrc between 1.14.5 and 1.16.3, and port the fix to 1.14.5 and rebuild rtspsrc only.

[10Key]

Hello.

Sorry for being late. We took a bit time to find TX2 reference board to test default gstreamer version 1.14.5.
This issue still happened even at gstreamer 1.14.5.

We will try it on TX2 (Revision 32.7.4) with gstreamer 1.16.3.
As we already said, the issue didn’t happened on Orin AGX (Revision 35.3.1, gstreamer 1.16.3).

So if this still exists on new version, we guess nvv4l2decoder at R32.7.4 might have some issues or bugs.

In that case, can’t we fix this issue at R32 environment for now?

[DaneLLL]

Hi,
It doesn’t seem to be an issue in nvv4l2decoder. If valid and continuous bitstream can be fed to decoder , we would expect it to work well. We would suggest investigate why h264parse drops the bitstream intermittently.

[10Key]

Hi.

We tested some cases after last reply.

We tried these two cases with fakesink.

1. use fakesink which elements causes

• link fakesink after h264parse ( queue | h264parse | fakesink ) → no stack frames on queue
• link fakesink after nvv4l2decoer ( queue | h264parse | nvv4l2decoder | fakesink) → frames stacked on queue

From this results, we agree with your comment.

The issue looks to be in rtspsrc ! h264parse of 1.14.5. Looks like it does not handle the packets well and cannot send valid h264 stream to next nvv4l2decoder .

And we also tried this down below.

One possible solution is to manually build 1.16.3 and upgrade to the version for a try.

We upgraded gstreamer 1.16.3 on TX2 (R. 32.7.4), but it did not solve the problem.
As I already said, orin AGX (Revision 35.3.1, gstreamer 1.16.3) didn’t cause any issues.

The results are different even though nvv4l2decoder version is same between Orin AGX and TX2.
These two have only revision difference.

We tried all options that you suggested, but unfortunately things haven’t being well.
We are investigateing h264parsebut we’ve been getting nothing right now.

Do you have any other options or suggestions?

[DaneLLL]

Hi,
If you think it is something wrong in nvv4l2decoder, please dump the stream:

gst-launch-1.0 rtspsrc ! h264parse ! video/x-h264,stream-format=byte-stream ! filesink location=dump.264

And try both software decoder and hardware decoder:

gst-launch-1.0 filesrc location=dump.h264 ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw ! xvimagesink sync=0

gst-launch-1.0 filesrc location=dump.h264 ! h264parse ! avdec_h264 ! videoconvert ! video/x-raw ! xvimagesink sync=0

If software decoder can decode the stream without issue and hardware decoder malfunctions, please attach the dumped stream so that we can try and debug further.

[10Key]

Hello,
Sorry for delay to respond.

First, there was no issue for dumped stream to decode by whichever software decoder or hardware decoder.
So we are thinking nvv4l2decoder is not main reason of this issue.

Next, we also thought that timestamp might cause this issue, So we dumped the RTP packets and replayed them by tcpreplay command.
However replaying them, there was no issue on that.

The only difference between receiving normal stream and replaying stream is the src element, rtspsrc and udpsrc.

rtspsrc is created by combining udpsrc and rtpbin, so rtpbin might be the main reason for this issue on whatever the gstreamer version is. (at least between gstreamer1.14.5 and 1.16.3.)

We couldn’t attach dumped RTP packets because of file size.
This issue happens after a while from receiving stream, so it is a bit hard to make attachable size file.
Is there any way to share the packet file?

If rtpbincauses this issue, we have two questions,

1. Is there anything to solve this? (It happens both of gstreamer versions 1.14.5 and 1.16.3)
2. On Orin AGX(R 35.3.1, gst1.16.3), there is no issue. What is the difference?

[DaneLLL]

Hi,
The rtspsrc and h264parse plugins are open source, so you may add prints to the plugins to catch more information. To clarify why invalid nal Type is reported in the error condition.

[10Key]

Hello.

Thanks for reply, we will continue on it.
Lastly, we have one question related to nvidia decoder element.

As we said before, this issue doesn’t happen on Orin AGX(R35.3.1, gst1.16.3).
But it does happen on TX2(R32.7.3, gst1.14.5 and 1.16.3).

Because of using same gstreamer version, we’re thinking only difference is nvv4l2decoder made by nvidia which does HW decode.

Does that mean Orin AGX can decode stream data which include invalid nal type sent from rtpbin ! h264parse, and TX2 cannot do that?
Are there any difference between them?

[DaneLLL]

Hi,
For decoding H264, TX2, Xavier, and Orin have identical hardware block.

Both Jetpack 4 and 5 can be run on Xavier. If you have Xavier or Xavier NX, it may worth a try for comparing Jetpack 4 and 5.