Question about Argus library FrameConsumer and acquiring frames

I am working with a TX1 and the Argus software for interfacing with a camera. I am looking to issue a repeat() request to the camera, and use a loop to continuously process images as soon as they are taken. In the Argus library, there is a sample program called “yuvJpeg” that does exactly this. This program is very similar to my usage and structure of my code.

The sample program (“yuvJpeg”), the consumer retrieves the image through the call

iFrameConsumer->acquireFrame()

. The function acquireFrame() has an optional argument that allows the user to specify a timeout.

By default, acquireFrame() waits indefinitely until a frame is ready to be ready, and does not return until a frame is acquired.

If a timeout is optionally specified, acquireFrame() will return if a frame cannot be acquired in the timeout time. Unfortunately, this causes a problem the next time you call acquireFrame(), illustrated in the following snippets:

Expected behavior:

// suppose a frame is not yet ready
Argus::UniqueObj<EGLStream::Frame> frame(iFrameConsumer->acquireFrame(10000000, &status));
// call will fail after 10ms and set status to Argus::STATUS_TIMEOUT
// ... some other things happen and some time elapses
// suppose a frame is now ready
Argus::UniqueObj<EGLStream::Frame> frame(iFrameConsumer->acquireFrame(10000000, &status));
// I would expect this call to succeed without problem, get the frame properly, and set status to Argus::STATUS_OK

Actual behavior:

// suppose a frame is not yet ready
Argus::UniqueObj<EGLStream::Frame> frame(iFrameConsumer->acquireFrame(10000000, &status));
// call will fail after 10ms and set status to Argus::STATUS_TIMEOUT
// ... some other things happen and some time elapses
// suppose a frame is now ready
Argus::UniqueObj<EGLStream::Frame> frame(iFrameConsumer->acquireFrame(10000000, &status));
// This fails immediately and sets status to STATUS::UNAVAILABLE and sends an error to outputstream:
// (Argus) Error InvalidState: Max frames acquired (in src/eglstream/FrameConsumerImpl.cpp, function acquireFrame(), line 254)

This isn’t a total mystery because the documentation for acquireFrame says something about “max frames”:

/**
 * Acquires a new frame from the FrameConsumer, returning a Frame object. If the maximum
 * number of frames are already acquired from the stream, an error will be returned
 * immediately .........

What does this mean? What does the “maximum number of frames acquired from the stream” mean? I notice that you can change this value when you create the FrameConsumer (and doing so appears to fix the above problem), but what does changing this value actually do? It’s not well explained in the headers. Any ideas? Thanks!

Bump

Hi, can you share a patch on yuvJpeg to reproduce the issue?

Sure.

Make the following set of changes.

On lines 112-114:

Original

UniqueObj<Frame> frame(iFrameConsumer->acquireFrame());
        if (!frame)
            break;

Change to

UniqueObj<Frame> frame(iFrameConsumer->acquireFrame(10000000)); // don't wait for more than 10ms for a frame to become available
        if (!frame) { CONSUMER_PRINT("NO FRAME!!!\n"); } else {
            CONSUMER_PRINT("FRAME!!!\n");

And on line 169:

Original

}

Change to

}}

The purpose of these changes is to change the loop so that acquireFrame has a finite timeout. If acquireFrame times out (i.e. no frame is available), it runs some other code and tries again later. You’ll notice that this does not work, and the FrameConsumer stops working after the first occasion in which acquireFrame times out.

Curiously, you can sometimes get around this problem by also modifying the following line 92:

Original

m_consumer = UniqueObj<FrameConsumer>(FrameConsumer::create(m_stream));

Change to

m_consumer = UniqueObj<FrameConsumer>(FrameConsumer::create(m_stream,2));

So clearly it has to do with “max frames” of the FrameConsumer, but I have no idea what that means and it isn’t well documented in FrameConsumer.h

Thanks!

Hi EEG,
In 30fps, frames are with 33ms interval. Setting to 10ms may not work. Could you try to set 40ms or 50 ms?

And for

/**
     * Creates a new FrameConsumer to read frames from an Argus OutputStream.
     *
     * @param[in] outputStream The output stream to read from.
     * @param[in] maxFrames The maximum number of frames that can be acquired
     *                      by this consumer at any point in time. Default is 1.
     * @param[out] status An optional pointer to return an error status code.
     *
     * @returns A new FrameConsumer object, or NULL on error.
     */
    static FrameConsumer* create(Argus::OutputStream* outputStream,
                                 uint32_t maxFrames = 1,
                                 Argus::Status* status = NULL);

Can you also try acquireFrame(40ms or 50ms) ?

I think you are missing the point that I’m trying to make. I’ll try to clarify. To directly answer your question - yes I am aware that 10ms is too short for 30fps. The question I am asking is why does acquireFrame() break its FrameConsumer object if acquireFrames() times-out and maxFrames is less than 2.

Your acquireFrame() function allows you to specify a timeout so that it does not block indefinitely. If it takes too long to acquire a frame, it returns and gives a “status timeout”. This is exactly what I’d expect.

However, after that happens, any future calls to acquireFrame() will immediately fail and return “status unavailable”, even if there are frames to be read in the future. This is strange and undesirable behavior. I would like acquireFrame() to function properly, even if prior calls to acquireFrame() have timed-out.

(As an a side-note, if you set maxFrames to anything greater than 1, it seems to fix this. If maxFrames is greater than 1, calls to acquireFrame() work properly, even if prior calls to acquireFrame() have timed out. But I don’t know why this occurs. I don’t know if it’s guaranteed to always work like this, or if it introduces some additional latency or has other side-effects)

Please refer back to the examples given in my first post for a line-by-line description of what’s happening. It definitely has something to do with the “maxFrames” argument to the FrameConsumer. But I don’t know what this field means and it is not well documented

Hi EEG,
You are correct that app cannot acquire more frames than maxFrames. In the case app needs to release frames and do acquireFrame() again.

Attach a sample based on tegra_multimedia_api/samples/09_camera_jpeg_capture
main.cpp (15.7 KB)

Can you explain in more detail what happens at a lower level? Like is maxFrames some sort of buffer or something? How do I “release frames” like you mentioned in your response?

The following two statements are appear to be true:

  1. If maxFrames is <2, acquireFrame() will fail if prior calls to acquireFrame() have timed-out.
  2. If maxFrames is >=2, acquireFrame() will not fail if prior calls to acquireFrame() have timed-out.
    Why does this behavior happen? Is it always guaranteed to work like this? Is this nehavior intentional or is it a bug?

Would it be possible for you to put me in touch with someone who wrote this code? I feel like we could resolve this in 5 minutes with a phone conversation and we’re not really making any progress here.

Hi EEG,
Please refer to the sample attached in #7. The following call releases the frame:

frame1.reset();

I cannot give the source, but the following is the pseudo code:

acquireFrame(timeout)
{
    if (acquiredFrames > maxFrames)
        return STATUS_UNAVAILABLE;

    EglStream_acquireFrame(timeout);
    if (no_new_frame)
        return STATUS_TIMEOUT;
    
    return new_frame;
}

The timeout mechanism of eglstream is not designed for timeout < frame rate interval. For 30fps(interval 33 ms), it may not work for setting timeout = 10ms. If set timeout = 50ms and occasionally a frame is dropped, making 66 ms invterval, it should work fine.

It can be difficult to have a phone conversation because it is maintained by engineers worldwide, but you can get further support by running through business process. Please contact your region salesperson of NVIDIA.

Thanks! This is more along the lines of what I was asking! Two followup questions:

  1. The pseudocode you posted is very helpful. When is the variable acquiredFrames incremented or decremented?

  2. If the timeout mechanism is not designed for timeout < frame rate interval, then is there some way for me to query the EGLStream to check if a frame is ready before I call acquireFrame()?

Hi EEG,
Please share your usecase in detail so that we can give suggestion.

tegra_multimedia_api/samples/09_camera_jpeg_capture is a good example to begin with. It demonstrates simultaneous JPEG encoding and video preview. Do you run a more complicated usecase?