[MMAPI R28.2] deinitPlane() of NvVideoEncoder

After updating to R28.2 from R28.1, deinitPlane() of NvVideoEncoder does not work.


I’m developping a live streaming system on Jetson TX2s.

(1) Input: USB UVC device with 4k resolution and 30fps
(2) Resizing: from 4k to appropriate value for the bit rate dynamically

  • output plane: V4L2_MEMORY_MMAP
  • capture plane: V4L2_MEMORY_MMAP
    (3) Encoding: H.264 (V4L2_MPEG_VIDEO_H264_PROFILE_BASELINE / V4L2_MPEG_VIDEO_H264_LEVEL_5_1)
  • output plane: V4L2_MEMORY_DMABUF
  • capture plane: V4L2_MEMORY_MMAP
    (4) RTP/UDP/IP/Ethernet streaming
    (5) Decoding
    (6) Resizing
    (7) Output: HDMI display with 4k resolution and 30fps

Until the last week, I used L4T R28.1.
In this week, I have updated our Jetson TX2s to the L4T R28.2.
Then it does not work at deinitPlane() of NvVideoEncoder.

When the bit rate is changed and it needs to change the resolution,
it execute these steps.

(a) Stop the input loop of NvVideoConverter: output_plane.qBuffer()
(b) Wait to become empty of output_plane of NvVideoConverter: output_plane.getNumQueuedBuffers() > 0
© Send EOS to NvVideoConverter: output_plane.qBuffer()
(d) Wait to become empty of capture plane of NvVideoConverter: capture_plane.waitForDQThread()
(e) Stop the input loop of NvVideoEncoder: output_plane.qBuffer()
(f) Wait to become empty of output_plane of NvVideoEncoder: output_plane.getNumQueuedBuffers() > 0
(g) Send EOS to NvVideoEncoder: output_plane.qBuffer()
(h) Wait to become empty of capture plane of NvVideoEncoder: capture_plane.waitForDQThread()
(i) Clear queue between NvVideoConverter and NvVideoencoder
(j) Free the buffer of NvVideoConverter: capture_plane.deinitPlane()
(k) Free the buffer of NvVideoEncoder: output_plane.deinitPlane() / capture_plane.deinitPlane()
(l) Re-configure NvVideoConverter: setCapturePlaneFormat() / capture_plane.setupPlane()

  • with new output resolution (capture plane)
    (m) Re-configure NvVideoEncoder: setCapturePlaneFormat() / setOutputPlaneFormat() / setProfile() / setLevel() / …
    / output_plane.setupPlane() / capture_plane.setupPlane()
  • with new input resolution (output plane)
    (n) Restart NvVideoEncoder: setStreamStatus(true)
    (o) Restart NvVideoConverter: setStreamStatus(true)

Our system stopped at step (k) with R28.2.
(It waits for 3sec, and TimeOut occurs in the next module.)

Are there any difference between R28.1 and R28.2 about NvVideoEncoder or NvV4l2ElementPlane ?
I cannot find deinitPlane() of NvVideoEncoder in the sample sources.
I cannot find any diffrence in NvV4l2ElementPlane.cpp about deinitPlane().

Or, is there any wrong in our step (a)-(o) ?

How to stop the dqThread of the output plane ?
I cannot stop it, then waitForDQThread() called from deinitPlane() is blocked.

When the dqThread call a callback function,
v4l2->m.planes[0].bytesused and buffer->planes[0].bytesused is always 0.
I cannot return “false” value with EOS for stopping dpThread.
** output plane is set to DMABUFF.

The output_plane.stopDQThread() does not work.
** Encoder is opened in a blocking mode (default).

I’ve found a difference between R28.1 and R28.2 about NvV4l2ElementPlane.cpp.
I tried R28.1 and R28.2 with adding many printf() into some functions of NvV4l2ElementPlane.cpp,
and compared logs of R28.1 and R28.2.

function NvV4l2ElementPlane::dqBuffer()

[R28.2]

int
NvV4l2ElementPlane::dqBuffer(struct v4l2_buffer &v4l2_buf, NvBuffer ** buffer,
        NvBuffer ** shared_buffer, uint32_t num_retries)
{
    do
    {
        ret = v4l2_ioctl(fd, VIDIOC_DQBUF, &v4l2_buf);
        if (ret == 0)
        {

        }
        else if (errno == EAGAIN)
        {
            if (<b>v4l2_buf.flags & V4L2_BUF_FLAG_LAST</b>)
            {
                break;
            }
        }
        else
        {
            break;
        }
    }
    while (ret && !is_in_error);
    return ret;
}

[R28.1]

int
NvV4l2ElementPlane::dqBuffer(struct v4l2_buffer &v4l2_buf, NvBuffer ** buffer,
        NvBuffer ** shared_buffer, uint32_t num_retries)
{
    do
    {
        ret = v4l2_ioctl(fd, VIDIOC_DQBUF, &v4l2_buf);
        if (ret == 0)
        {

        }
        else if (errno == EAGAIN)
        {
            if (!<b>streamon</b>)
            {
                break;
            }
        }
        else
        {
            break;
        }
    }
    while (ret && !is_in_error);
    return ret;
}

When I want to stop the Encoder, enqueue a EOS (bytesused=0) to output plane.
And call function deinitPlane() for each output plane and capture plane.

function NvV4l2ElementPlane::deinitPlane()

void
NvV4l2ElementPlane::deinitPlane()
{
    <b>setStreamStatus(false)</b>;
    waitForDQThread(-1);
    for (uint32_t i = 0; i < num_buffers; i++)
    {
        call deallocateMemori() / unmap()
    }
    reqbufs(memory_type, 0);
}

Then ioctl(VIDIOC_DQBUF) of output_plane.dqBuffer() returns with errno=EAGAIN for EOS,
and ioctl(VIDIOC_DQBUF) of capture_plane.dqBuffer() returns normally with byteused=0 for EOS.

About the capture plane, dqBuffer() is normally returned and dqThread calls the callback function.
The callback function returns with false because of bytesused==0.
Then dqThread stops and exit.

But about the output plane, ioctl() of dqBuffer() is not normally returned.
In R28.1, setStreamStatus(false) is called from deinitPlane(),
then variable streamon is set to false, and exit from do-while() loop of dqBuffer().
In R28.2, it does not exit from do-while() loop of dqBuffer()
(retry count and continues the loop)
because of the flag (v4l2_buf.flags & V4L2_BUF_FLAG_LAST) is not set to “1” by ioctl(VIDIOC_DQBUF).

What is the reason for the change (exit conditon) ?
Is there any problem if I return it to the previous ?
Or to add condition (! streamon) to

if (v4l2_buf.flags & V4L2_BUF_FLAG_LAST)
if ((v4l2_buf.flags & V4L2_BUF_FLAG_LAST)||(! streamon))

3.6.5. Buffer Flags
https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/buffer.html#buffer-flags
V4L2_BUF_FLAG_LAST
0x00100000
Last buffer produced by the hardware.
mem2mem codec drivers set this flag on the capture queue for the last buffer when the ioctl VIDIOC_QUERYBUF or VIDIOC_DQBUF ioctl is called.
Due to hardware limitations, the last buffer may be empty.
In this case the driver will set the bytesused field to 0,
regardless of the format.
Any Any subsequent call to the VIDIOC_DQBUF ioctl will not block anymore,
but return an EPIPE error code.

Hi mynaemi,
We have the modification to implement https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/vidioc-encoder-cmd.html

The original method is to signal the end of stream is by enqueing a 0 sized buffer. If you are good with the original way, please use r28.1 sample.

Hi DaneLLL,

I truly appreciate your support.
I have some questions about this specification changing.

Q.1 Waiting to be empty

I could not find any sample source to stop the NvVideoEncoder in the MMAPI sample directory.

I want to stop the NvVideoEncoder safely without any discarded data (pictures and NAL units).
The encoder behaves as a FIFO/LILO.
Using the original method, when the EOS (size 0) appears at capture_plane.dqBuffer(),
the encoder is empty (there is no unprocessed picture and no pending output NAL unit).
Then I call setStreamStatus(false), waitForDQThread() and reqbuf(0) in deinitBuffer().

If I use new APIs of R28.2, how do I know the encoder is empty ?
Do I have to use waitAllBuffersDequeued() API, anything else ?

Because, …

According to the V4L2 documents (7.10. ioctl VIDIOC_ENCODER_CMD),
when it receives “V4L2_ENC_CMD_STOP”, it stops immediately and lost data in it.

https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/vidioc-encoder-cmd.html
When the V4L2_ENC_CMD_STOP_AT_GOP_END flag is set, encoding will continue until the end of the current Group Of Pictures, otherwise encoding will stop immediately.

According to the V4L2 documents (7.54. ioctl VIDIOC_STREAMON, VIDIOC_STREAMOFF),
when the setStreamStatus(false) is called to stop the encoder,
data in it are discarded.
https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/vidioc-streamon.html
all images captured but not dequeued yet will be lost

Q.2 About NvVideoDecoder and NvVideoConverter

According to the V4L2 documents (7.7. ioctl VIDIOC_DECODER_CMD),
the driver API of decoder has Start/Stop/Pause/Resume command.
But R28.2 Multimedia API does not support this ioctl() type.
https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/vidioc-decoder-cmd.html

There is not similar API for the NvVideoConverter anyways.

In the future, what do you plan about these APIs ?
(1) Only encoder supported
(2) In the future release decoder supported according to the V4L2 specification
(3) Align the procedure of encoder, decoder and converter

The class NvVideoEncoder, NvVideoDecoder and NvVideoConverter have same base classes,
and are very similar to use.
But now, we have to use different sequense to start and stop, when we use R28.2 APIs.

Q.3 What you recommend to stop NvVideoEncoder

It’s conceivable that I can choose one in 3 approachs.

(1) Using Patched R28.1 libtegrav4l2.so (to fix timestamp issue) and the original EOS method with R28.1 APIs
—> If any other isssue/bug is discoverd, this choice is the worst (not to fix it).
(2) Using R28.2 libtegrav4l2.so and modified original EOS method with R28.2 APIs(common classes)
—> To modify dqBuffer() source as #3. When a new MMAPI releases, we have to modify it as same.
(3) Using R28.2 libtegrav4l2.so and new APIs
—> We have to spend more time, because we do not know the detailed operation about the new APIs.

Which do you recommend ?

Best Regards,

Hi mynaemi,
01_video_encode of r28.2 is probably not fully matched with the spec per your comment. We need some time to check it.

For your case, you have implemented your case per r28.1 samples. Are you able to run the same on r28.2? 01_video_encode of r28.1 should run fine on r28.2.

Hi DaneLLL,

I appreciate your support.

I have choiced (2) (modified dqBuffer() source code), and it run normally now.

If the specification of ioctl(VIDIOC_DQBUF) is changed in any future release,
it may be not able to run normally.
Then we shuould think again.

Best Regards

I have modified only 1 line:

function NvV4l2ElementPlane::dqBuffer()
Original R28.2

int
    NvV4l2ElementPlane::dqBuffer(struct v4l2_buffer &v4l2_buf, NvBuffer ** buffer,
            NvBuffer ** shared_buffer, uint32_t num_retries)
    {
        do
        {
            ret = v4l2_ioctl(fd, VIDIOC_DQBUF, &v4l2_buf);
            if (ret == 0)
            {

            }
            else if (errno == EAGAIN)
            {
                if (v4l2_buf.flags & V4L2_BUF_FLAG_LAST)
                {
                    break;
                }
            }
            else
            {
                break;
            }
        }
        while (ret && !is_in_error);
        return ret;
    }

Modified R28.2

int
    NvV4l2ElementPlane::dqBuffer(struct v4l2_buffer &v4l2_buf, NvBuffer ** buffer,
            NvBuffer ** shared_buffer, uint32_t num_retries)
    {
        do
        {
            ret = v4l2_ioctl(fd, VIDIOC_DQBUF, &v4l2_buf);
            if (ret == 0)
            {

            }
            else if (errno == EAGAIN)
            {
                if ((v4l2_buf.flags & V4L2_BUF_FLAG_LAST)||(! streamon))
                {
                    break;
                }
            }
            else
            {
                break;
            }
        }
        while (ret && !is_in_error);
        return ret;
    }

When I send an EOS to NvVideoEncoder output_plane.qBuffer(),
it returns from output_plane.deinitPlane() normally,
and can re-configure NvVideoEncoder (Picture Resolution of output plane buffer).