Why can't the camera get all frames

TLedTS · March 28, 2022, 7:40am

When I was running the sample in jetson_multimedia_api/samples/13_multi_camera with 4 cameras, it can only capture around 49 images, although the parameter DEFAULT_FRAME_COUNT is 100.
After some experiments, I found that, the code here just can not acquire a new frame anymore after 49 images.

    while (m_framesRemaining--)
    {
        for (uint32_t i = 0; i < m_streams.size(); i++)
        {
            /* Acquire a frame */
            UniqueObj<Frame> frame(iFrameConsumers[i]->acquireFrame());
            IFrame *iFrame = interface_cast<IFrame>(frame);
            if (!iFrame)
            {
                std::cout<<"no more frame"<<std::endl;
                break;
            }

So is this normal? It’s due to the software limitation or hardware limitation?
What should I do if I want to capture images instantaneously with 4 cameras at the same time?

If somebody has any suggestion, please kindly let me know! Thanks a lot!

PS:
camera type: leopard IMX334-MIPI_R32.3.1_Xavier_NV 3864x2180@60fps
platform:Nvidia Jetson AGX Xavier Developer kit based on R32.3.1 (Jetpack 4.3).

DaneLLL · March 28, 2022, 9:15am

Hi,
So the application gets stuck in acquiring the frames? Or can successfully exit but does not get 100 frames?

For further checking, please make sure each source sends capture requests for 100 times in

    for (uint32_t i = 0; i < g_frame_count; i++)
    {
        for (uint32_t j = 0; j < streamCount; j++)
        {
            ICaptureSession *iCaptureSession =
                    interface_cast<ICaptureSession>(captureHolders[j].get()->getSession());
            Request *request = captureHolders[j].get()->getRequest();
            uint32_t frameId = iCaptureSession->capture(request);
            if (frameId == 0)
                ORIGINATE_ERROR("Failed to submit capture request");
        }
    }

TLedTS · March 28, 2022, 9:28am

Hi,
Thanks for your help.
The application will not get stuck, but just can’t get 100 frames.
I checked the frameId that you mentioned here, it will send requests for 100 times successfully. But, from the log, it seems that after sending all requests it will get no more frame in several frames later. With even one camera only.

@DaneLLL any idea?🙏

DaneLLL · March 30, 2022, 5:16am

Hi,
We try with our camera and don’t observe the issue. Here is the log for reference:

nvidia@TX2:~/jetson_multimedia_api/samples/13_multi_camera$ ./multi_camera
[INFO] (NvEglRenderer.cpp:110) <renderer0> Setting Screen width 640 height 480
Argus Version: 0.98.3 (multi-process)
CONSUMER: Waiting until producer is connected...
CONSUMER: Producer has connected; continuing.
CONSUMER: Waiting until producer is connected...
CONSUMER: Producer has connected; continuing.
CONSUMER: Waiting until producer is connected...
CONSUMER: Producer has connected; continuing.
CONSUMER: Waiting until producer is connected...
CONSUMER: Producer has connected; continuing.
CONSUMER: Waiting until producer is connected...
CONSUMER: Producer has connected; continuing.
CONSUMER: Waiting until producer is connected...
CONSUMER: Producer has connected; continuing.
CONSUMER: Render frame 1
DstComp rect's width must be multiple of 2, aligning the width
DstComp rect's left must be multiple of 2, aligning the left
DstComp rect's width must be multiple of 2, aligning the width
DstComp rect's width must be multiple of 2, aligning the width
DstComp rect's left must be multiple of 2, aligning the left
DstComp rect's width must be multiple of 2, aligning the width
DstComp rect's width must be multiple of 2, aligning the width
CONSUMER: Render frame 2
CONSUMER: Render frame 3
CONSUMER: Render frame 4
CONSUMER: Render frame 5
CONSUMER: Render frame 6
CONSUMER: Render frame 7
CONSUMER: Render frame 8
CONSUMER: Render frame 9
CONSUMER: Render frame 10
CONSUMER: Render frame 11
CONSUMER: Render frame 12
CONSUMER: Render frame 13
CONSUMER: Render frame 14
CONSUMER: Render frame 15
CONSUMER: Render frame 16
CONSUMER: Render frame 17
CONSUMER: Render frame 18
CONSUMER: Render frame 19
CONSUMER: Render frame 20
CONSUMER: Render frame 21
CONSUMER: Render frame 22
CONSUMER: Render frame 23
CONSUMER: Render frame 24
CONSUMER: Render frame 25
CONSUMER: Render frame 26
CONSUMER: Render frame 27
CONSUMER: Render frame 28
CONSUMER: Render frame 29
CONSUMER: Render frame 30
CONSUMER: Render frame 31
CONSUMER: Render frame 32
CONSUMER: Render frame 33
CONSUMER: Render frame 34
CONSUMER: Render frame 35
CONSUMER: Render frame 36
CONSUMER: Render frame 37
CONSUMER: Render frame 38
CONSUMER: Render frame 39
CONSUMER: Render frame 40
CONSUMER: Render frame 41
CONSUMER: Render frame 42
CONSUMER: Render frame 43
CONSUMER: Render frame 44
CONSUMER: Render frame 45
CONSUMER: Render frame 46
CONSUMER: Render frame 47
CONSUMER: Render frame 48
CONSUMER: Render frame 49
CONSUMER: Render frame 50
CONSUMER: Render frame 51
CONSUMER: Render frame 52
CONSUMER: Render frame 53
CONSUMER: Render frame 54
CONSUMER: Render frame 55
CONSUMER: Render frame 56
CONSUMER: Render frame 57
CONSUMER: Render frame 58
CONSUMER: Render frame 59
CONSUMER: Render frame 60
CONSUMER: Render frame 61
CONSUMER: Render frame 62
CONSUMER: Render frame 63
CONSUMER: Render frame 64
CONSUMER: Render frame 65
CONSUMER: Render frame 66
CONSUMER: Render frame 67
CONSUMER: Render frame 68
CONSUMER: Render frame 69
CONSUMER: Render frame 70
CONSUMER: Render frame 71
CONSUMER: Render frame 72
CONSUMER: Render frame 73
CONSUMER: Render frame 74
CONSUMER: Render frame 75
CONSUMER: Render frame 76
CONSUMER: Render frame 77
CONSUMER: Render frame 78
CONSUMER: Render frame 79
CONSUMER: Render frame 80
CONSUMER: Render frame 81
CONSUMER: Render frame 82
CONSUMER: Render frame 83
CONSUMER: Render frame 84
CONSUMER: Render frame 85
CONSUMER: Render frame 86
CONSUMER: Render frame 87
CONSUMER: Render frame 88
CONSUMER: Render frame 89
CONSUMER: Render frame 90
CONSUMER: Render frame 91
CONSUMER: Render frame 92
CONSUMER: Render frame 93
CONSUMER: Render frame 94
CONSUMER: Render frame 95
CONSUMER: Render frame 96
CONSUMER: Render frame 97
CONSUMER: Render frame 98
CONSUMER: Render frame 99
CONSUMER: Render frame 100
CONSUMER: Done.

Do you see expect framerate in gst-launch-1.0 command? You can run this command to show fps:

gst-launch-1.0 -v nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),format=NV12' ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 nvarguscamerasrc sensor-id=1 ! 'video/x-raw(memory:NVMM),format=NV12' ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0

TLedTS · March 30, 2022, 6:47am

@DaneLLL
Thanks!
If I run it directly, it will be the same log as yours, but actually the code just capture half of the frames. Because, according to the original code, if !frame it will just break the loop for this camera, not the whole process. And at the end of this part, it still can print likeRender frame 50 even if the frame is null. Right?
So maybe you can add this line CONSUMER_PRINT("no more frame");like me.

bool ConsumerThread::threadExecute()
{
    IEGLOutputStream *iEglOutputStreams[MAX_CAMERA_NUM];
    IFrameConsumer *iFrameConsumers[MAX_CAMERA_NUM];

    for (uint32_t i = 0; i < m_streams.size(); i++)
    {
        iEglOutputStreams[i] = interface_cast<IEGLOutputStream>(m_streams[i]);
        iFrameConsumers[i] = interface_cast<IFrameConsumer>(m_consumers[i]);
        if (!iFrameConsumers[i])
            ORIGINATE_ERROR("Failed to get IFrameConsumer interface");

        /* Wait until the producer has connected to the stream */
        CONSUMER_PRINT("Waiting until producer is connected...\n");
        if (iEglOutputStreams[i]->waitUntilConnected() != STATUS_OK)
            ORIGINATE_ERROR("Stream failed to connect.");
        CONSUMER_PRINT("Producer has connected; continuing.\n");
    }

    while (m_framesRemaining--)
    {
        for (uint32_t i = 0; i < m_streams.size(); i++)
        {
            /* Acquire a frame */
            UniqueObj<Frame> frame(iFrameConsumers[i]->acquireFrame());
            IFrame *iFrame = interface_cast<IFrame>(frame);
            if (!iFrame)
            {
                CONSUMER_PRINT("no more frame");
                break;
            }
        ......
        }

        CONSUMER_PRINT("Render frame %d\n", g_frame_count - m_framesRemaining);

    CONSUMER_PRINT("Done.\n");
    ...
}

TLedTS · March 30, 2022, 6:51am

Besides, I tried with gstreamer.
And I got this log, seems it is just 30fps.

GST_ARGUS: Creating output stream
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3864 x 2180 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 30.000000; Exposure Range min 7000, max 166725000;

GST_ARGUS: Running with following settings:
   Camera index = 1 
   Camera mode  = 0 
   Output Stream W = 3864 H = 2180 
   seconds to Run    = 0 
   Frame Rate = 59.999999 
GST_ARGUS: PowerService: requested_clock_Hz=27216000
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3864 x 2180 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 30.000000; Exposure Range min 7000, max 166725000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 0 
   Output Stream W = 3864 H = 2180 
   seconds to Run    = 0 
   Frame Rate = 59.999999 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
CONSUMER: Producer has connected; continuing.
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink1/GstFakeSink:fakesink1: sync = false
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = false
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink1: last-message = rendered: 16, dropped: 0, current: 31.10, average: 31.10
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 17, dropped: 0, current: 33.13, average: 33.13
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink1: last-message = rendered: 32, dropped: 0, current: 29.98, average: 30.53
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 32, dropped: 0, current: 29.08, average: 31.10
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink1: last-message = rendered: 48, dropped: 0, current: 30.11, average: 30.39
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 48, dropped: 0, current: 30.85, average: 31.01

Then I set it in the command line like this:gst-launch-1.0 -v nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=(int)3864,height=(int)2180,framerate=60/1,format=NV12' ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 nvarguscamerasrc sensor-id=1 ! 'video/x-raw(memory:NVMM),width=(int)3864,height=(int)2180,framerate=60/1,format=NV12' ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0.
It seems like I got the 60 fps.

GST_ARGUS: Creating output stream
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3864 x 2180 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 30.000000; Exposure Range min 7000, max 166725000;

GST_ARGUS: Running with following settings:
   Camera index = 1 
   Camera mode  = 0 
   Output Stream W = 3864 H = 2180 
   seconds to Run    = 0 
   Frame Rate = 59.999999 
GST_ARGUS: PowerService: requested_clock_Hz=221117408
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Waiting until producer is connected...
CONSUMER: Producer has connected; continuing.
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3864 x 2180 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 30.000000; Exposure Range min 7000, max 166725000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 0 
   Output Stream W = 3864 H = 2180 
   seconds to Run    = 0 
   Frame Rate = 59.999999 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink1/GstFakeSink:fakesink1: sync = false
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = false
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink1: last-message = rendered: 30, dropped: 0, current: 59.95, average: 59.95
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 32, dropped: 0, current: 63.51, average: 63.51
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink1: last-message = rendered: 60, dropped: 0, current: 59.32, average: 59.63
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 62, dropped: 0, current: 59.56, average: 61.54
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink1: last-message = rendered: 91, dropped: 0, current: 60.59, average: 59.95

TLedTS · April 1, 2022, 2:56am

Btw, I also tested with 4 cameras, and the fps is only 40ish. For 3 cameras it is around 53. @DaneLLL

DaneLLL · April 1, 2022, 4:32am

Hi,
If you are able to achieve target fps in gst-launch-1.0 you may consider use nvcompositor plugin. The 13 sample is to demonstrates NvBufferComposite() function but the architecture may not be efficient. It only has single thread but for multiple camera inputs, it is better to have individual thread for each camera.

TLedTS · April 1, 2022, 5:36am

Good to know. Thank you for your kind help!

system · April 27, 2022, 3:53am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.