Possible multimedia api regression with decode interlace source

Hi,

similar previous thread Broken picture after l4t decode+transform interlaced source . Sample code for reproduction GitHub - maxlapshin/l4t2-demo This sample decoder → nvbuffertransform → encoder with simulate live work, added sleep 40ms before read next nalu.

Progressive source works well in all cases, broken only interlace source

jetson tx2, xavier nx

# cat /etc/nv_tegra_release 
# R32 (release), REVISION: 5.1, GCID: 27362550, BOARD: t186ref, EABI: aarch64, DATE: Wed May 19 18:16:00 UTC 2021
ii  nvidia-l4t-3d-core                     32.5.1-20210519111140                            arm64        NVIDIA GL EGL Package
ii  nvidia-l4t-apt-source                  32.5.1-20210219084708                            arm64        NVIDIA L4T apt source list debian package
ii  nvidia-l4t-bootloader                  32.5.1-20210614115015                            arm64        NVIDIA Bootloader Package
ii  nvidia-l4t-camera                      32.5.1-20210519111140                            arm64        NVIDIA Camera Package
ii  nvidia-l4t-configs                     32.5.1-20210219084708                            arm64        NVIDIA configs debian package
ii  nvidia-l4t-core                        32.5.1-20210519111140                            arm64        NVIDIA Core Package
ii  nvidia-l4t-cuda                        32.5.1-20210519111140                            arm64        NVIDIA CUDA Package
ii  nvidia-l4t-firmware                    32.5.1-20210519111140                            arm64        NVIDIA Firmware Package
ii  nvidia-l4t-graphics-demos              32.5.1-20210519111140                            arm64        NVIDIA graphics demo applications
ii  nvidia-l4t-gstreamer                   32.5.1-20210519111140                            arm64        NVIDIA GST Application files
ii  nvidia-l4t-init                        32.5.1-20210519111140                            arm64        NVIDIA Init debian package
ii  nvidia-l4t-initrd                      32.5.1-20210614115015                            arm64        NVIDIA initrd debian package
ii  nvidia-l4t-jetson-io                   32.5.1-20210219084708                            arm64        NVIDIA Jetson.IO debian package
ii  nvidia-l4t-jetson-multimedia-api       32.5.1-20210519111140                            arm64        NVIDIA Jetson Multimedia API is a collection of lower-level APIs that support flexible application development.
ii  nvidia-l4t-kernel                      4.9.201-tegra-32.5.1-20210505093723              arm64        NVIDIA Kernel Package
ii  nvidia-l4t-kernel-dtbs                 4.9.201-tegra-32.5.1-20210505093723              arm64        NVIDIA Kernel DTB Package
ii  nvidia-l4t-kernel-headers              4.9.201-tegra-32.5.1-20210505093723              arm64        NVIDIA Linux Tegra Kernel Headers Package
ii  nvidia-l4t-libvulkan                   32.5.1-20210519111140                            arm64        NVIDIA Vulkan Loader Package
ii  nvidia-l4t-multimedia                  32.5.1-20210519111140                            arm64        NVIDIA Multimedia Package
ii  nvidia-l4t-multimedia-utils            32.5.1-20210519111140                            arm64        NVIDIA Multimedia Package
ii  nvidia-l4t-oem-config                  32.5.1-20210219084708                            arm64        NVIDIA OEM-Config Package
ii  nvidia-l4t-tools                       32.5.1-20210614115015                            arm64        NVIDIA Public Test Tools Package
ii  nvidia-l4t-wayland                     32.5.1-20210519111140                            arm64        NVIDIA Wayland Package
ii  nvidia-l4t-weston                      32.5.1-20210519111140                            arm64        NVIDIA Weston Package
ii  nvidia-l4t-x11                         32.5.1-20210519111140                            arm64        NVIDIA X11 Package
ii  nvidia-l4t-xusb-firmware               32.5.1-20210614115015                            arm64        NVIDIA USB Firmware Package

Hi,
We have suggested adjust some properties in
Broken picture after l4t decode+transform interlaced source - #23 by DaneLLL
Does it work for you? With the setting we don’t observe any issue. would like to know your status.

And it is gstreamer in previous topic. This topic is about jetson_multiemdia_api. Do you switch from gstreamer to jetson_multimedia_api now? Would be great if you can provide more information about your use-case, such as a TV box, or?

gstreamer works well, thank you

Yep, use-case using jetson mmapi for transcode live streams; also postprocessing raw data after v4l2 decoder with a nvbuffercomposite or cuda code

Hi khizbulin,

Please try below samples, this sample works well.

/usr/src/jetson_multimedia_api/samples/16_multivideo_transcode

Hi,

your 16_multivideo_transcode don’t use NvBufferTransform or NvBufferCompose.

Hi,
Since NvBufferTransform() and NvBufferComposite() does not touch timestamps, the function calls should not make impact to mis-ordering.

We have checked and confirmed video decoder sends out frames in correct order. By decoding sample.h264 through 00_video_decode, the playback is good, and timestamps are in order with the test patch:

diff --git a/multimedia_api/ll_samples/samples/00_video_decode/video_decode_main.cpp b/multimedia_api/ll_samples/samples/00_video_decode/video_decode_main.cpp
index 8bb14a9..c7baa23 100644
--- a/multimedia_api/ll_samples/samples/00_video_decode/video_decode_main.cpp
+++ b/multimedia_api/ll_samples/samples/00_video_decode/video_decode_main.cpp
@@ -1085,6 +1085,12 @@ dec_capture_loop_fcn(void *arg)
               cout << "[" << v4l2_buf.index << "]" "dec capture plane dqB timestamp [" <<
                   v4l2_buf.timestamp.tv_sec << "s" << v4l2_buf.timestamp.tv_usec << "us]" << endl;
             }
+           static uint64_t prev_ts = 0;
+           uint64_t ts;
+           ts = v4l2_buf.timestamp.tv_sec * 1000000 + v4l2_buf.timestamp.tv_usec;
+           if (prev_ts >= ts) {
+           cout << "ts = " << ts << ", prev_ts = " << prev_ts << endl;
+           } prev_ts = ts;

             if (!ctx->disable_rendering && ctx->stats)
             {

So it is more like the buffers are not sent to encoder in correct order. Could you please check this? Please print out timestamp of each buffer and check if the order is wrong in certain condition.

Hi,

i check timestamps 00_video_decode with patch similar yours

--- jetson_multimedia_api/samples/00_video_decode/video_decode_main.cpp 2021-07-26 22:37:32.000000000 +0300
+++ jetson_multimedia_api1/samples/00_video_decode/video_decode_main.cpp        2021-08-23 13:02:28.161042492 +0300
@@ -1086,6 +1086,16 @@
                   v4l2_buf.timestamp.tv_sec << "s" << v4l2_buf.timestamp.tv_usec << "us]" << endl;
             }
 
+            static int64_t prev_ts = 0;
+            int64_t ts;
+            ts = v4l2_buf.timestamp.tv_sec * 1000000 + v4l2_buf.timestamp.tv_usec;
+            if (prev_ts >= ts) {
+                cout << "broken ts = " << ts << ", prev_ts = " << prev_ts << ", time_delta = " << ts - prev_ts << endl;
+            } else {
+                cout << "ok ts = " << ts << ", prev_ts = " << prev_ts << ", time_delta = " << ts - prev_ts << endl;
+            }
+            prev_ts = ts;
+
             if (!ctx->disable_rendering && ctx->stats)
             {
                 /* EglRenderer requires the fd of the 0th plane to render the buffer. */

./video_decode H264 --disable-rendering --input-nalu --copy-timestamp 0 25 ../22_trans_cuda/sample.h264 produce broken timestamps

ok ts = 40000, prev_ts = 0, time_delta = 40000                
ok ts = 680000, prev_ts = 40000, time_delta = 640000          
broken ts = 600000, prev_ts = 680000, time_delta = -80000  
ok ts = 760000, prev_ts = 600000, time_delta = 160000         
broken ts = 520000, prev_ts = 760000, time_delta = -240000  
ok ts = 1000000, prev_ts = 520000, time_delta = 480000        
broken ts = 920000, prev_ts = 1000000, time_delta = -80000    
ok ts = 1080000, prev_ts = 920000, time_delta = 160000       
broken ts = 840000, prev_ts = 1080000, time_delta = -240000 
ok ts = 1240000, prev_ts = 840000, time_delta = 400000        
ok ts = 1320000, prev_ts = 1240000, time_delta = 80000       
ok ts = 1400000, prev_ts = 1320000, time_delta = 80000       
ok ts = 1480000, prev_ts = 1400000, time_delta = 80000        
ok ts = 1560000, prev_ts = 1480000, time_delta = 80000        
broken ts = 1160000, prev_ts = 1560000, time_delta = -400000 
ok ts = 1800000, prev_ts = 1160000, time_delta = 640000       
ok ts = 1880000, prev_ts = 1800000, time_delta = 80000        
broken ts = 1720000, prev_ts = 1880000, time_delta = -160000  
ok ts = 1960000, prev_ts = 1720000, time_delta = 240000      
ok ts = 2040000, prev_ts = 1960000, time_delta = 80000        
broken ts = 1640000, prev_ts = 2040000, time_delta = -400000  

--input-nalu - does not work correctly
--input-chunk - works correctly

But I don’t think the problem is with timestamps. 00_video_decode produce broken timestamps also for progressive source. It’s bug from read_decoder_input_nalu

I update GitHub - maxlapshin/l4t2-demo with cuda function for transfer raw image data from dma_buf decoder capture plane to dma_buf encoder output plane. With cuda transfer you can see the problem better

./transcoder H264 sample.h264 H264 /mnt/nfs/ts/out_cuda.h264 -ew 1920 -eh 1080 --input-nalu --cuda - works good
./transcoder H264 sample.h264 H264 /mnt/nfs/ts/out_cuda.h264 -ew 1920 -eh 1080 --input-nalu --cuda --live - with simulate live input 40ms sleep the picture broken

i think, it’s bad when behavior depends on input latency

Hi,
For your description the issue is more like decoder does not output correct order in certain condition. Could you make a patch to 00_video_decode so that we can reproduce it and check with other teams? Please help check where we can put usleep(40000); in 00_video_decode and then see misorder in playback.

The logic in --copy-timestamp is for IDR-P-P-P-… encoded stream, but sample.h264 is encoded like:

POC must = frame# or field# for SNRs to be correct
--------------------------------------------------------------------------
  Frame          POC  Pic#   QP    SnrY     SnrU     SnrV   Y:U:V Time(ms)
--------------------------------------------------------------------------
00011( I )       22    20    25                             4:2:0      89
00008( B )       16    21    33                             4:2:0      70
00006( b )       12    22    35                             4:2:0      64
00007( b )       14    22    35                             4:2:0      62
00009( b )       18    22    34                             4:2:0      59
00010( b )       20    22    34                             4:2:0      55
00015( P )       30    22    30                             4:2:0      69
00013( B )       26    23    33                             4:2:0      54

So needs different logic for this kind of stream. But even though the logic is wrong, it should not trigger misorder.

Hi,

by the way, I haven’t tested 00_video_decode with 40ms live sleep.

Yep, 00_video_decode with this patch, the problem is reproduce.

--- ../../../jetson_multimedia_api/samples/00_video_decode/video_decode_main.cpp        2021-07-26 22:37:32.000000000 +0300
+++ ./video_decode_main.cpp     2021-08-24 12:59:14.271718743 +0300
@@ -135,7 +135,7 @@
     buffer->planes[0].bytesused = 4;
     stream_ptr += 4;
 
-    if (ctx->copy_timestamp)
+    //if (ctx->copy_timestamp)
     {
       if (ctx->decoder_pixfmt == V4L2_PIX_FMT_H264) {
         if ((IS_H264_NAL_CODED_SLICE(stream_ptr)) ||
@@ -1706,6 +1706,9 @@
         }
         v4l2_buf.m.planes[0].bytesused = buffer->planes[0].bytesused;
 
+       if (ctx.flag_copyts)
+         usleep(40000);
+
         if (ctx.input_nalu && ctx.copy_timestamp && ctx.flag_copyts)
         {
           /* Update the timestamp. */
/video_decode H264 --disable-rendering -o out_nv12.raw --input-nalu sample.h264
ffmpeg -y -f rawvideo -pixel_format nv12 -s 1920x1080 -i out_nv12.raw -c:v libx264 -preset fast -crf 28 -g 300 out.ts

also problem is reproduce with usleep 40ms before dqbuffer

--- ../../../jetson_multimedia_api/samples/00_video_decode/video_decode_main.cpp        2021-07-26 22:37:32.000000000 +0300
+++ ./video_decode_main.cpp     2021-08-24 13:25:09.852827831 +0300
@@ -135,7 +135,7 @@
     buffer->planes[0].bytesused = 4;
     stream_ptr += 4;
 
-    if (ctx->copy_timestamp)
+    //if (ctx->copy_timestamp)
     {
       if (ctx->decoder_pixfmt == V4L2_PIX_FMT_H264) {
         if ((IS_H264_NAL_CODED_SLICE(stream_ptr)) ||
@@ -1639,6 +1639,8 @@
 
         v4l2_buf.m.planes = planes;
 
+       if (ctx.flag_copyts)
+         usleep(40000);
         /* dequeue a buffer for output plane. */
         if(allow_DQ)
         {
1 Like

Hi,
We can reproduce the issue with your patch. Will check with teams and update.

2 Likes

Hi DaneLLL,

Any update here?

Hi pro.trener,
The issue is still under investigation. Will update once there is further progress.