omxh264enc does not send keyframes at regular intervals

hi,

I am using the following pipeline, on a TX1 to record video (and a variation for recording audio+video) using a custom application.

v4l2src device=/dev/video0 ! 'video/x-raw, width=(int)800, height=(int)600, format=(string)I420' ! omxh264enc profile=1 insert-sps-pps=true ! 'video/x-h264, stream-format=(string)byte-stream' ! h264parse ! splitmuxsink location=/home/nvidia/test-%02d.avi muxer=avimux max-files=5 send-key-frame-requests=true max-size-time=10000000000

When this pipeline is started for the first time, everything works as expected. But when this pipeline is stopped and started a few minutes later, the size of the video is zero initially and then it increases a few minutes later. if the pipeline is stopped while the size of the file is zero then the data is lost. In addition to that, the size of that file is much bigger than it supposed to, given the time contraints set (in max-size-time).

Setting splitmuxsink to debug, i get the following log when the file size increases from zero

0:12:10.385037335  2011       0x6cf230 DEBUG           splitmuxsink gstsplitmuxsink.c:1229:handle_mq_input:<multiqueue:sink_0> Buf TS +0:02:12.197803428 total in_bytes 255911145
0:12:10.385085511  2011       0x6cf230 DEBUG           splitmuxsink gstsplitmuxsink.c:1065:check_queue_length:<multiqueue:sink_0> Checking queue length len 2470 cur_max 2470 queued gops 0
0:12:10.385103428  2011       0x6cf230 INFO            splitmuxsink gstsplitmuxsink.c:1102:check_queue_length:<smux> Multiqueue overrun - enlarging to 2471 buffers ctx 0x66c670
0:12:10.444341494  2011       0x6cf230 DEBUG           splitmuxsink gstsplitmuxsink.c:1229:handle_mq_input:<multiqueue:sink_0> Buf TS +0:02:12.264531001 total in_bytes 255943565
0:12:10.444392223  2011       0x6cf230 DEBUG           splitmuxsink gstsplitmuxsink.c:1065:check_queue_length:<multiqueue:sink_0> Checking queue length len 2471 cur_max 2471 queued gops 0
0:12:10.444408993  2011       0x6cf230 INFO            splitmuxsink gstsplitmuxsink.c:1102:check_queue_length:<smux> Multiqueue overrun - enlarging to 2472 buffers ctx 0x66c670
0:12:10.506073144  2011       0x6cf230 DEBUG           splitmuxsink gstsplitmuxsink.c:1229:handle_mq_input:<multiqueue:sink_0> Buf TS +0:02:12.331269688 total in_bytes 256045960
0:12:10.506116946  2011       0x6cf230 INFO            splitmuxsink gstsplitmuxsink.c:1249:handle_mq_input:<multiqueue:sink_0> Have keyframe with running time +0:02:12.331269688
0:12:10.506142154  2011       0x6cf230 DEBUG           splitmuxsink gstsplitmuxsink.c:1029:check_completed_gop:<smux> Collected GOP is complete. Processing (ctx 0x66c670)
0:12:10.506158925  2011       0x6cf230 DEBUG           splitmuxsink gstsplitmuxsink.c:1065:check_queue_length:<multiqueue:sink_0> Checking queue length len 2472 cur_max 2472 queued gops 1
0:12:10.506175123  2011       0x6cf230 INFO            splitmuxsink gstsplitmuxsink.c:1102:check_queue_length:<smux> Multiqueue overrun - enlarging to 2473 buffers ctx 0x66c670
0:12:10.506231112  2011       0x6cf000 INFO            splitmuxsink gstsplitmuxsink.c:621:complete_or_wait_on_out:<multiqueue:src_0> Woken for new max running time +0:02:12.331269688
0:12:10.569088956  2011       0x6cf230 DEBUG           splitmuxsink gstsplitmuxsink.c:1229:handle_mq_input:<multiqueue:sink_0> Buf TS +0:02:12.397999480 total in_bytes 256086580
0:12:10.569138227  2011       0x6cf230 DEBUG           splitmuxsink gstsplitmuxsink.c:1065:check_queue_length:<multiqueue:sink_0> Checking queue length len 2228 cur_max 2473 queued gops 1
0:12:10.631903259  2011       0x6cf230 DEBUG           splitmuxsink gstsplitmuxsink.c:1229:handle_mq_input:<multiqueue:sink_0> Buf TS +0:02:12.464735491 total in_bytes 256121557
0:12:10.631951227  2011       0x6cf230 DEBUG           splitmuxsink gstsplitmuxsink.c:1065:check_queue_length:<multiqueue:sink_0> Checking queue length len 1923 cur_max 2473 queued gops 1
0:12:10.695171883  2011       0x6cf230 DEBUG           splitmuxsink gstsplitmuxsink.c:1229:handle_mq_input:<multiqueue:sink_0> Buf TS +0:02:12.498099855 total in_bytes 256148710

In particular, when the log shows the line : Have keyframe with running time , the file is populated with data.

I read in a another thread ( https://devtalk.nvidia.com/default/topic/963733/?comment=4973962 ) about a similar problem that gets resolved by setting: insert-sps-pps=true. In this way the codec can be forced to send keyframes. But I did not get that.

I have also set : send-key-frame-requests=TRUE , in splitmuxsink. Since, the version that comes with the TX1 does not support this property, I had to upgrade Gstreamer to version 1.9.2

splitmuxsink will not flush the buffer into the file, unless a keyframe comes in.

Any ideas why am I not getting keyframes at regular intervals from omxh264enc, after a restart. Is there another way to trigger it.

Thanks in advance.

Hi piperak,
Can you try ‘omxh24enc ! splitmuxsink’ ? h264parse may modify the streams.

If you have h264pase, please try ‘config-interval=-1’
https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-bad/html/gst-plugins-bad-plugins-h264parse.html

hi,

I just tried that and Gstreamer is complaining about the -1. I am using version 1.8.1 that comes as standard with r28.2.1 . I changed the number to 1 and it stopped complaining, but the keyframes arrive late, like before. The more times I stop the stream the worse it gets. After 3 stops the first keyframe arrives after 4.34 minutes. I also tried it with 2 different sources. A custom v4lsrc and videotestsrc. But that would not be an issue i guess.
Is there a way to set omxh264 on debug and observe that the keyframe event is sent from splitmuxsink upstream? Thanks in advance.

Joseph

Hi piperak,
Please share a way to reproduce the issue with videotestsrc. With gst-launch-1.0 we can always see SPS/PPS is with IDR frames. Your usecase looks not same as gst-launch-1.0 and we need to reproduce the issue first.

Hi piperak,
You can build libgstomx.so for debugging.
The source code is at https://developer.nvidia.com/embedded/dlc/sources-r2821

The issue looks to be in splitmuxsink. Can you try to set other values to the properties ‘max-files’ and ‘max-size-time’? We don’t have much experience in using splitmuxsink. Please try if other values help.
It should run well with filesink.

hi,

I have tried it with filesink and it worked. I also noticed that a keyframe is transmitted regularly.

I also tried it without the omxh264enc but with splitmuxsink i.e. uncompressed video. That worked also.
I have tried other values for max-files and max-size-time. I also tried max-size-bytes. Nothing seems to have any effect.

I will compile libgstomx.so for debug and try and send you the logs for both omxh264enc and splitmuxsink. Unfortunatelly, I cannot provide you with a copy of our application.

Joseph

Hi piperak,
We do not need a copy of your application, but a simple test app to reproduce the issue such as below posts:
https://devtalk.nvidia.com/default/topic/1020558/jetson-tx1/h265-decode-failed/post/5196041/#5196041
https://devtalk.nvidia.com/default/topic/1010111/jetson-tx1/nvmm-memory/post/5158652/#5158652

If we can reproduce the issue, we can understand the usecase better and do further check.

hi,

I managed to replicate my issue using code from one of the threads, you indicated. However, i have trouble uploading the code, as some characters prevent the code from being accepted in this thread(my IP gets temporarily blocked). Is there any other way that I can post the application code I wrote to demonstrate this issue?

Joseph

nvidia-app.c.zip (1.02 KB)

Hi Joseph, it accepts zip file. Can you zip your code and attach again?

I attached the code I used to replicate this issue. As you will notice 3 video files created before the first pipeline stop. The first 2 are, approximately of equal size, which is expected. After the second start, the size of the first becomes zero and stays like this for quite some time. Then suddenly, it increases by a huge amount. If you observe the debug logs, the splitmuxsink element is waiting for a keyframe. Since, the keyframe is delayed, it buffers all the incoming frames in memory. At some point the keyframe arrives and the entire buffer is flushed into a file. This is splitmuxsink’s expected behavior (buffer until a keyframe). If you notice, the file size is quite larger that the expected one, as specified by the max-size-time. The biggest problem is, that if the pipeline is stopped before the keyframe arrives, all data is lost.

As you can see, I have included both properties, as indicated, in my pipeline. The value of config-interval was set to 1 instead of -1, that was suggested, because gstreamer complains.

Any ideas why this keyframe command is taking so long?

Normally, I use a custom V4L2 source, but I have the same issue with every source I have tried. Thanks again for the quick response.

Joseph
nvidia-app.c.zip (1.02 KB)

Hi piperak,

Could you share the reproduce steps? and where are you compiler your code?
I will use the same steps to reproduce your issue. Thanks!

hi,

the process is simple:

  1. Type export GST_DEBUG=splimuxsink*:5
  2. Start the application on a TX1.
  3. if you do not modify the code the video files will be saved under /home/nvidia under the name video-0X.avi where X is an index that increases
  4. Open a second terminal cd and to /home/nvidia/ . If you type ls -al on the terminal a couple of times you can see the size of the 3 files created.
  5. At some point if you keep repeating step 3 you will see video-01.avi size go to zero. This is expected. This is after the second start of the pipeline.
  6. Repeat step 3. Size does not change. Check the log. It is similar to the log posted at the begging of this post.
  7. At some point a line similar to:
k gstsplitmuxsink.c:1249:handle_mq_input:<multiqueue:sink_0> Have keyframe with running time +0:02:12.331269688

will appear at the log. At the same time if you repeat step 3 you will notice that the size of that file has increased.
7. Let the application complete. Compare the size of video-0.avi and video-1.avi The should be roughly equal, but they are not.

I have noticed that the more times you start and stop a stream the more it takes for a keyframe to arrive.

Attached you will find a copy of the compiled application for aarch64, compressed.

Joseph

nvidia-app.zip (180 KB)

Hi piperak,
Please share your gcc command to build the sample.

Hi,

attached you will find a nsight project, that you can use for compiling this project. Import it and compile for debug.

BR

Joseph
nvidia-app.zip (3.27 MB)

Hi piperak,
In your usecase, are you able to unref the pipeline and re-launch it? This approach should work. Also is avimux must-have? How about using qtmux?

Hi piperak,
Please refer to stopRecording() in tegra_multimedia_api\argus\samples\gstVideoEncode\main.cpp.

You have to send EOS and wait for it before changing to NULL state.

Hi,
My requirement is for h264 in an avi container. If qtmux supports it, then i can switch to that, if it works. Does it work? I think I tried it at some point but without any options with the same results.

When I unref the pipeline everything is fine, but my requirement is to be able to to stop and start a pipeline multiple times. That is mandatory.

Joseph

Please try to send EOS to the pipeline. We have seen invalid mp4 in qtmux+filesink without EOS event. Maybe it is same for avimux+splitmuxsink

Hi,

I can confirm that sending an EOS solves the zero file size issue. But, splitmuxsink does not split the files correctly anymore. In fact it does not split at all after restarting.

Moreover, if you have a pipeline with audio, you need to send an EOS to the corresponding pad also.

Have you figured out why the EOS signal does not reach splitmuxsink?

Hi piperk,
I checked the h264 bitstream and can see SpsPps is attached with IDR in second run. Not sure why h264parse and splitmuxsink cannot work correctly, but can you try the pipeline:

launch_stream  << "videotestsrc is-live=true ! "
    << "video/x-raw, width="<< w <<", height="<< h <<", format=(string)I420 ! "
    << "omxh264enc profile=1 insert-sps-pps=true ! "
    << "video/x-h264, stream-format=(string)byte-stream ! "
    << "h264parse config-interval=1 ! "
    << "splitmuxsink location=/home/nvidia/test-%02d.ts muxer=mpegtsmux max-files=5 max-size-time=60000000000 ";

1 Adding is-live=true gives 30 fps source.
2 Try transport stream