GStream: nvv4l2h264enc: B-Frames display ordering is not right

This bug requires to enable the new b-frames fix see: https://devtalk.nvidia.com/default/topic/1056338/jetson-nano/gstreamer-h264-encoder-bug-gstv4l2h264enc-h-num-b-frames-is-set-to-boolean-instead-int/post/5359591/#5359591

It appears that the nvv4l2h264enc encoder is setting the display timestamps the same as the decoder timestamps.

A strict player will show the lack of proper order of each frame i.e. IPbbPbbPbb… instead of IbbPbbPbb… (Other players will skip one of the b frames and it might looks like if the video is a lower fps )

This issue is not happening on the NMAP 02_video_encoder sample SW.

Hi,

Please share more detail about this. What is the player you run to check the issue? Does the player notify ‘the lack of proper order of each frame’ automatically or we have to enable it?

Hi,
Please share what tool you use to check disorder of b frames and steps of using the tool. Thanks.

Sorry for the delay. (On travel)

I use Intel Video Pro Analyzer (Intel VPA)

I found more details. If you use MPEGTSmuxer you will have an output on the pipe. MPEGTSmuxer “try to fix” the PTSs and generate a stream. If you use other muxers like QTmux then the pipe will stop and you will see errors. some muxers report lack of PTSs.

SO the bug is on the PTS generated by the NV encoder, I tried other H264 encoders on GStreamer and this problem doesn’t exist.

BTW: If you generate an H264 file using GStreamer and use a mp4muxer such as GPAG or BENTO4, and then transmuxed to MPEGTS, the stream will be right. NOTE: GPAC and BENTO4 are capable to create a file without this problem; because they reset the PTS from zero when they create the file. So don’t consider this a an issue of the “muxer”

Hi,
After applying the patch, we are able to use qtmux to generate a good mp4:

$ gst-launch-1.0 videotestsrc num-buffers=300 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12' ! nvv4l2h264enc num-B-Frames=2 ! h264parse ! qtmux ! filesink location=a.mp4

Could you share how to reproduce the error you see with qtmux?

For more information, is Intel VPA freeware?

This is strange: Yes your sample is able to produce an mp4 file, however this stream contains no B frames (Ipppppppppp)

I’dont know what is going on… if I feed a real video, I’m able to get B frames, but continues to fail

I’m using and patching the latest code rebased on 7/8/19

This is what I get if I use qtmux

ERROR: from element /GstPipeline:pipeline0/GstQTMux:qtmux0: Could not multiplex stream.
Additional debug info:
gstqtmux.c(4559): gst_qt_mux_add_buffer (): /GstPipeline:pipeline0/GstQTMux:qtmux0:
<b>Buffer has no PTS.</b>
EOS on shutdown enabled -- waiting for EOS after Error

Hi,
We extract h264 from the mp4:

danel@ce-tw-daneliu:~/D/home/daneliu/temp/ffmpeg/ffmpeg-3.3.1$ ./ffmpeg -i a.mp4 -vcodec copy -bsf:v h264_mp4toannexb test.h264

And inspect it in JM decoder:

danel@ce-tw-daneliu:~/D/home/daneliu/temp/jm19.0$ ./JM/bin/ldecod.exe -i test.h264
Setting Default Parameters...
Parsing Configfile decoder.cfg

******************************************************
*               Decoder Parameters                   *
******************************************************
Parameter InputFile = test.h264
Parameter OutputFile = test_dec.yuv
Parameter RefFile = test_rec.yuv
Parameter WriteUV = 1
Parameter FileFormat = 0
Parameter RefOffset = 0
Parameter POCScale = 2
Parameter DisplayDecParams = 1
Parameter ConcealMode = 0
Parameter RefPOCGap = 2
Parameter POCGap = 2
Parameter Silent = 0
Parameter IntraProfileDeblocking = 1
Parameter DecFrmNum = 0
Parameter DecodeAllLayers = 0
Parameter DPBPLUS0 = 1
Parameter DPBPLUS1 = 0
******************************************************
----------------------------- JM 19.0 (FRExt) -----------------------------
 Input reference file                   : test_rec.yuv does not exist
                                          SNR values are not available
Warning: zero_byte shall exist
POC must = frame# or field# for SNRs to be correct
--------------------------------------------------------------------------
  Frame          POC  Pic#   QP    SnrY     SnrU     SnrV   Y:U:V Time(ms)
--------------------------------------------------------------------------
00000(IDR)        0     0     2                             4:2:0       3
00003( P )        6     1     2                             4:2:0       0
00001( b )        2     2     2                             4:2:0       0
00002( b )        4     2     2                             4:2:0       0
00006( P )       12     2     2                             4:2:0       0
00004( b )        8     3     2                             4:2:0       0
00005( b )       10     3     2                             4:2:0       0
00009( P )       18     3     2                             4:2:0       0
00007( b )       14     4     2                             4:2:0       0
00008( b )       16     4     2                             4:2:0       0
00012( P )       24     4     2                             4:2:0       0
00010( b )       20     5     2                             4:2:0       0
00011( b )       22     5     2                             4:2:0       0
00015( P )       30     5     2                             4:2:0       0
00013( b )       26     6     2                             4:2:0       0
00014( b )       28     6     2                             4:2:0       0
00018( P )       36     6     2                             4:2:0       0
00016( b )       32     7     2                             4:2:0       0
00017( b )       34     7     2                             4:2:0       0
00021( P )       42     7     2                             4:2:0       0
00019( b )       38     8     2                             4:2:0       0
00020( b )       40     8     2                             4:2:0       0
00024( P )       48     8     2                             4:2:0       0
00022( b )       44     9     2                             4:2:0       0
00023( b )       46     9     2                             4:2:0       0
00027( P )       54     9     2                             4:2:0       0
00025( b )       50    10     2                             4:2:0       0
00026( b )       52    10     2                             4:2:0       0
00029( P )       58    10     2                             4:2:0       0
00028( b )       56    11     2                             4:2:0       0
00000(IDR)        0     0     0                             4:2:0       2
00003( P )        6     1     2                             4:2:0       0
00001( b )        2     2     2                             4:2:0       0
00002( b )        4     2     2                             4:2:0       0
00006( P )       12     2     2                             4:2:0       0
00004( b )        8     3     2                             4:2:0       0
00005( b )       10     3     2                             4:2:0       0
00009( P )       18     3     2                             4:2:0       0
00007( b )       14     4     2                             4:2:0       0
00008( b )       16     4     2                             4:2:0       0
00012( P )       24     4     2                             4:2:0       0
00010( b )       20     5     2                             4:2:0       0
00011( b )       22     5     2                             4:2:0       0
00015( P )       30     5     2                             4:2:0       0
00013( b )       26     6     2                             4:2:0       0
00014( b )       28     6     2                             4:2:0       0
00018( P )       36     6     2                             4:2:0       0
00016( b )       32     7     2                             4:2:0       0
00017( b )       34     7     2                             4:2:0       0
00021( P )       42     7     2                             4:2:0       0
00019( b )       38     8     2                             4:2:0       0
00020( b )       40     8     2                             4:2:0       0
00024( P )       48     8     2                             4:2:0       0
00022( b )       44     9     2                             4:2:0       0
00023( b )       46     9     2                             4:2:0       0
00027( P )       54     9     2                             4:2:0       0
00025( b )       50    10     2                             4:2:0       0
00026( b )       52    10     2                             4:2:0       0
00029( P )       58    10     2                             4:2:0       0
00028( b )       56    11     2                             4:2:0       0
00000(IDR)        0     0     0                             4:2:0       2
00003( P )        6     1     2                             4:2:0       0
00001( b )        2     2     2                             4:2:0       0
00002( b )        4     2     2                             4:2:0       0
00006( P )       12     2     2                             4:2:0       0
00004( b )        8     3     2                             4:2:0       0
00005( b )       10     3     2                             4:2:0       0
00009( P )       18     3     2                             4:2:0       0
00007( b )       14     4     2                             4:2:0       0
00008( b )       16     4     2                             4:2:0       0
00012( P )       24     4     2                             4:2:0       0
00010( b )       20     5     2                             4:2:0       0
00011( b )       22     5     2                             4:2:0       0
00015( P )       30     5     2                             4:2:0       0
00013( b )       26     6     2                             4:2:0       0
00014( b )       28     6     2                             4:2:0       0
00018( P )       36     6     2                             4:2:0       0
00016( b )       32     7     2                             4:2:0       0
00017( b )       34     7     2                             4:2:0       0
00021( P )       42     7     2                             4:2:0       0
00019( b )       38     8     2                             4:2:0       0
00020( b )       40     8     2                             4:2:0       0
00024( P )       48     8     2                             4:2:0       0
00022( b )       44     9     2                             4:2:0       0
00023( b )       46     9     2                             4:2:0       0
00027( P )       54     9     2                             4:2:0       0
00025( b )       50    10     2                             4:2:0       0
00026( b )       52    10     2                             4:2:0       0
00029( P )       58    10     2                             4:2:0       0
00028( b )       56    11     2                             4:2:0       0
00000(IDR)        0     0     0                             4:2:0       2
00003( P )        6     1     2                             4:2:0       0
00001( b )        2     2     2                             4:2:0       0
00002( b )        4     2     2                             4:2:0       0
00006( P )       12     2     2                             4:2:0       0
00004( b )        8     3     2                             4:2:0       0
00005( b )       10     3     2                             4:2:0       0
00009( P )       18     3     2                             4:2:0       0
00007( b )       14     4     2                             4:2:0       0
00008( b )       16     4     2                             4:2:0       0
00012( P )       24     4     2                             4:2:0       0
00010( b )       20     5     2                             4:2:0       0
00011( b )       22     5     2                             4:2:0       0
00015( P )       30     5     2                             4:2:0       0
00013( b )       26     6     2                             4:2:0       0
00014( b )       28     6     2                             4:2:0       0
00018( P )       36     6     2                             4:2:0       0
00016( b )       32     7     2                             4:2:0       0
00017( b )       34     7     2                             4:2:0       0
00021( P )       42     7     2                             4:2:0       0
00019( b )       38     8     2                             4:2:0       0
00020( b )       40     8     2                             4:2:0       0
00024( P )       48     8     2                             4:2:0       0
00022( b )       44     9     2                             4:2:0       0
00023( b )       46     9     2                             4:2:0       0
00027( P )       54     9     2                             4:2:0       0
00025( b )       50    10     2                             4:2:0       0
00026( b )       52    10     2                             4:2:0       0
00029( P )       58    10     2                             4:2:0       0
00028( b )       56    11     2                             4:2:0       0
00000(IDR)        0     0     0                             4:2:0       2
00003( P )        6     1     2                             4:2:0       0
00001( b )        2     2     2                             4:2:0       0
00002( b )        4     2     2                             4:2:0       0
00006( P )       12     2     2                             4:2:0       0
00004( b )        8     3     2                             4:2:0       0
00005( b )       10     3     2                             4:2:0       0
00009( P )       18     3     2                             4:2:0       0
00007( b )       14     4     2                             4:2:0       0
00008( b )       16     4     2                             4:2:0       0
00012( P )       24     4     2                             4:2:0       0
00010( b )       20     5     2                             4:2:0       0
00011( b )       22     5     2                             4:2:0       0
00015( P )       30     5     2                             4:2:0       0
00013( b )       26     6     2                             4:2:0       0
00014( b )       28     6     2                             4:2:0       0
00018( P )       36     6     2                             4:2:0       0
00016( b )       32     7     2                             4:2:0       0
00017( b )       34     7     2                             4:2:0       0
00021( P )       42     7     2                             4:2:0       0
00019( b )       38     8     2                             4:2:0       0
00020( b )       40     8     2                             4:2:0       0
00024( P )       48     8     2                             4:2:0       0
00022( b )       44     9     2                             4:2:0       0
00023( b )       46     9     2                             4:2:0       0
00027( P )       54     9     2                             4:2:0       0
00025( b )       50    10     2                             4:2:0       0
00026( b )       52    10     2                             4:2:0       0
00029( P )       58    10     2                             4:2:0       0
00028( b )       56    11     2                             4:2:0       0
00000(IDR)        0     0     0                             4:2:0       2
00003( P )        6     1     2                             4:2:0       0
00001( b )        2     2     2                             4:2:0       0
00002( b )        4     2     2                             4:2:0       0
00006( P )       12     2     2                             4:2:0       0
00004( b )        8     3     2                             4:2:0       0
00005( b )       10     3     2                             4:2:0       0
00009( P )       18     3     2                             4:2:0       0
00007( b )       14     4     2                             4:2:0       0
00008( b )       16     4     2                             4:2:0       0
00012( P )       24     4     2                             4:2:0       0
00010( b )       20     5     2                             4:2:0       0
00011( b )       22     5     2                             4:2:0       0
00015( P )       30     5     2                             4:2:0       0
00013( b )       26     6     2                             4:2:0       0
00014( b )       28     6     2                             4:2:0       0
00018( P )       36     6     2                             4:2:0       0
00016( b )       32     7     2                             4:2:0       0
00017( b )       34     7     2                             4:2:0       0
00021( P )       42     7     2                             4:2:0       0
00019( b )       38     8     2                             4:2:0       0
00020( b )       40     8     2                             4:2:0       0
00024( P )       48     8     2                             4:2:0       0
00022( b )       44     9     2                             4:2:0       0
00023( b )       46     9     2                             4:2:0       0
00027( P )       54     9     2                             4:2:0       0
00025( b )       50    10     2                             4:2:0       0
00026( b )       52    10     2                             4:2:0       0
00029( P )       58    10     2                             4:2:0       0
00028( b )       56    11     2                             4:2:0       0
00000(IDR)        0     0     0                             4:2:0       2
00003( P )        6     1     2                             4:2:0       0
00001( b )        2     2     2                             4:2:0       0
00002( b )        4     2     2                             4:2:0       0
00006( P )       12     2     2                             4:2:0       0
00004( b )        8     3     2                             4:2:0       0
00005( b )       10     3     2                             4:2:0       0
00009( P )       18     3     2                             4:2:0       0
00007( b )       14     4     2                             4:2:0       0
00008( b )       16     4     2                             4:2:0       0
00012( P )       24     4     2                             4:2:0       0
00010( b )       20     5     2                             4:2:0       0
00011( b )       22     5     2                             4:2:0       0
00015( P )       30     5     2                             4:2:0       0
00013( b )       26     6     2                             4:2:0       0
00014( b )       28     6     2                             4:2:0       0
00018( P )       36     6     2                             4:2:0       0
00016( b )       32     7     2                             4:2:0       0
00017( b )       34     7     2                             4:2:0       0
00021( P )       42     7     2                             4:2:0       0
00019( b )       38     8     2                             4:2:0       0
00020( b )       40     8     2                             4:2:0       0
00024( P )       48     8     2                             4:2:0       0
00022( b )       44     9     2                             4:2:0       0
00023( b )       46     9     2                             4:2:0       0
00027( P )       54     9     2                             4:2:0       0
00025( b )       50    10     2                             4:2:0       0
00026( b )       52    10     2                             4:2:0       0
00029( P )       58    10     2                             4:2:0       0
00028( b )       56    11     2                             4:2:0       0
00000(IDR)        0     0     0                             4:2:0       2
00003( P )        6     1     2                             4:2:0       0
00001( b )        2     2     2                             4:2:0       0
00002( b )        4     2     2                             4:2:0       0
00006( P )       12     2     2                             4:2:0       0
00004( b )        8     3     2                             4:2:0       0
00005( b )       10     3     2                             4:2:0       0
00009( P )       18     3     2                             4:2:0       0
00007( b )       14     4     2                             4:2:0       0
00008( b )       16     4     2                             4:2:0       0
00012( P )       24     4     2                             4:2:0       0
00010( b )       20     5     2                             4:2:0       0
00011( b )       22     5     2                             4:2:0       0
00015( P )       30     5     2                             4:2:0       0
00013( b )       26     6     2                             4:2:0       0
00014( b )       28     6     2                             4:2:0       0
00018( P )       36     6     2                             4:2:0       0
00016( b )       32     7     2                             4:2:0       0
00017( b )       34     7     2                             4:2:0       0
00021( P )       42     7     2                             4:2:0       0
00019( b )       38     8     2                             4:2:0       0
00020( b )       40     8     2                             4:2:0       0
00024( P )       48     8     2                             4:2:0       0
00022( b )       44     9     2                             4:2:0       0
00023( b )       46     9     2                             4:2:0       0
00027( P )       54     9     2                             4:2:0       0
00025( b )       50    10     2                             4:2:0       0
00026( b )       52    10     2                             4:2:0       0
00029( P )       58    10     2                             4:2:0       0
00028( b )       56    11     2                             4:2:0       0
00000(IDR)        0     0     0                             4:2:0       2
00003( P )        6     1     2                             4:2:0       0
00001( b )        2     2     2                             4:2:0       0
00002( b )        4     2     2                             4:2:0       0
00006( P )       12     2     2                             4:2:0       0
00004( b )        8     3     2                             4:2:0       0
00005( b )       10     3     2                             4:2:0       0
00009( P )       18     3     2                             4:2:0       0
00007( b )       14     4     2                             4:2:0       0
00008( b )       16     4     2                             4:2:0       0
00012( P )       24     4     2                             4:2:0       0
00010( b )       20     5     2                             4:2:0       0
00011( b )       22     5     2                             4:2:0       0
00015( P )       30     5     2                             4:2:0       0
00013( b )       26     6     2                             4:2:0       0
00014( b )       28     6     2                             4:2:0       0
00018( P )       36     6     2                             4:2:0       0
00016( b )       32     7     2                             4:2:0       0
00017( b )       34     7     2                             4:2:0       0
00021( P )       42     7     2                             4:2:0       0
00019( b )       38     8     2                             4:2:0       0
00020( b )       40     8     2                             4:2:0       0
00024( P )       48     8     2                             4:2:0       0
00022( b )       44     9     2                             4:2:0       0
00023( b )       46     9     2                             4:2:0       0
00027( P )       54     9     2                             4:2:0       0
00025( b )       50    10     2                             4:2:0       0
00026( b )       52    10     2                             4:2:0       0
00029( P )       58    10     2                             4:2:0       0
00028( b )       56    11     2                             4:2:0       0
00000(IDR)        0     0     0                             4:2:0       2
00003( P )        6     1     2                             4:2:0       0
00001( b )        2     2     2                             4:2:0       0
00002( b )        4     2     2                             4:2:0       0
00006( P )       12     2     2                             4:2:0       0
00004( b )        8     3     2                             4:2:0       0
00005( b )       10     3     2                             4:2:0       0
00009( P )       18     3     2                             4:2:0       0
00007( b )       14     4     2                             4:2:0       0
00008( b )       16     4     2                             4:2:0       0
00012( P )       24     4     2                             4:2:0       0
00010( b )       20     5     2                             4:2:0       0
00011( b )       22     5     2                             4:2:0       0
00015( P )       30     5     2                             4:2:0       0
00013( b )       26     6     2                             4:2:0       0
00014( b )       28     6     2                             4:2:0       0
00018( P )       36     6     2                             4:2:0       0
00016( b )       32     7     2                             4:2:0       0
00017( b )       34     7     2                             4:2:0       0
00021( P )       42     7     2                             4:2:0       0
00019( b )       38     8     2                             4:2:0       0
00020( b )       40     8     2                             4:2:0       0
00024( P )       48     8     2                             4:2:0       0
00022( b )       44     9     2                             4:2:0       0
00023( b )       46     9     2                             4:2:0       0
00027( P )       54     9     2                             4:2:0       0
00025( b )       50    10     2                             4:2:0       0
00026( b )       52    10     2                             4:2:0       0
00029( P )       58    10     2                             4:2:0       0
00028( b )       56    11     2                             4:2:0       0
-------------------- Average SNR all frames ------------------------------
 SNR Y(dB)           :  0.00
 SNR U(dB)           :  0.00
 SNR V(dB)           :  0.00
 Total decoding time : 0.245 sec (1224.490 fps)[300 frm/245 ms]
--------------------------------------------------------------------------
 Exit JM 19 (FRExt) decoder, ver 19.0
 Output status file                     : log.dec
300 frames are decoded.

We can see B frames. Although it saves in IPbbPbb…, POC looks correct.

Hi Dane,

The issue of B Frames time-stamps reported on July remains: I know that if the pipe outputs H264 in “OK” mode and the DTS looks fine, but when connected to other components in GSTreamer, the timestamps of the B frames are wrong, In a test I did, it appears that the times of the Frames are a copy of the P Frames.

As you might know, someone developed an FFMPEG Encoder/Decoder for the Jetson Nano. https://github.com/jocover/jetson-ffmpeg If you use this SW and enable B frames encoding, you will see the same problem on FFMPEG as on GSTreamer, so maybe is a driver issue.

Hi Rudoplh,
Is is possible to run a ffmpeg command to show the error? If it can be reproduced with ffmpeg or other freewares, please share us the command. We don’t get the error via JM decoder but probably the tool does not catch this kind of issue. Would be great if there is other tool can show the error. Thanks.

I just got some new tools that might help to identify the timestamps issue. Let me produce a report for you. It is time to get this fix it.

I’m working with Jiang, The developer who created the FFmpeg-Jetson project. He found that the Multimedia API is not passing DTS, he has to recalculate it.

When I enable the b-frame feature, it was done on the CLI interface side, but it appears that the encoder needs to abstract the DTS information and pass it to the pipe connected.