argus_openglbox only runs at 15 fps

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Using the Devkit camera, I can:

  • capture 1280x720 images at 60 fps
  • render a spinning box in OpenGL at 60 fps
  • render a spinning box capturing textures from the camera at 15 fps

This can easily be tested in the argus_openglBox demo.

If I remove the capture as part of the loop, and just capture the texture once, I get 60 fps in the OpenGL ES bit, and I think I could get a faster frame rate if I had a faster display.

If I just use a gstreamer pipeline with “nvcamerasrc width=1280xheight=720 ! nvoverlaysink” then I get 60 fps in the overlay (and I think I could get 120 fps if I had a faster display.)

But, capturing one frame of camera per frame in OpenGL drives everything down to 15 fps. Presumably, there is some kind of synchronous stall between the camera capture and the GL render that loses not one, but THREE frames of overlap, do divide the frame rate down to 15 fps. (Or, perhaps, the camera always runs at 30 fps in the GL demo, and it’s just dividing by two?)

2

Hi snarky,
Please try the patch:

diff --git a/public/samples/openglBox/main.cpp b/public/samples/openglBox/main.cpp
index 6c28b43..82612e6 100644
--- a/public/samples/openglBox/main.cpp
+++ b/public/samples/openglBox/main.cpp
@@ -201,7 +201,6 @@ bool ConsumerThread::threadExecute()
     while (eglStreamConsumerAcquireKHR(g_display.get(), m_stream))
     {
         frame++;
-        CONSUMER_PRINT("Acquired frame %d. Rendering.\n", frame);
 
         GLfloat rotMat[16];
         MathUtils::createRotationMatrix((float)frame, 0.3f, 0.5f, 0.2f, rotMat);
@@ -213,7 +212,7 @@ bool ConsumerThread::threadExecute()
 
         PROPAGATE_ERROR(m_context.swapBuffers());
     }
-    CONSUMER_PRINT("No more frames. Cleaning up.\n");
+    CONSUMER_PRINT("No more frames. Cleaning up, total frame %d .\n", frame);
 
     PROPAGATE_ERROR(requestShutdown());
 
@@ -268,6 +267,14 @@ static bool execute(const ExecuteOptions& options)
         ORIGINATE_ERROR("Camera %d not available; there are %d cameras",
                         options.cameraIndex, (unsigned)cameraDevices.size());
 
+
+
+    ICameraProperties *iCameraProperties = interface_cast<ICameraProperties>(cameraDevices[0]);
+    if (!iCameraProperties)
+        ORIGINATE_ERROR("Failed to get ICameraProperties interface");
+
+
+
     // Create the capture session using the specified device and get its interfaces.
     UniqueObj<CaptureSession> captureSession(
             iCameraProvider->createCaptureSession(cameraDevices[options.cameraIndex]));
@@ -305,6 +312,26 @@ static bool execute(const ExecuteOptions& options)
         ORIGINATE_ERROR("Failed to create Request");
     iRequest->enableOutputStream(outputStream.get());
 
+
+
+    ISourceSettings *iSourceSettings = interface_cast<ISourceSettings>(iRequest->getSourceSettings());
+    if (!iSourceSettings)
+        ORIGINATE_ERROR("Failed to get ISourceSettings interface");
+    std::vector<SensorMode*> sensorModes;
+    iCameraProperties->getBasicSensorModes(&sensorModes);
+    if (sensorModes.size() == 0)
+        ORIGINATE_ERROR("Failed to get sensor modes");
+    PRODUCER_PRINT("Available Sensor modes :\n");
+    for (uint32_t i = 0; i < sensorModes.size(); i++) {
+        ISensorMode *iSensorMode = interface_cast<ISensorMode>(sensorModes[i]);
+        Size2D<uint32_t> resolution = iSensorMode->getResolution();
+        PRODUCER_PRINT("[%u] W=%u H=%u\n", i, resolution.width(), resolution.height());
+    }
+    iSourceSettings->setSensorMode(sensorModes[2]);
+    iSourceSettings->setFrameDurationRange(Range<uint64_t>(1e9/60));
+
+
+
     // Submit capture requests.
     PRODUCER_PRINT("Starting repeat capture requests.\n");
     if (iCaptureSession->repeat(request.get()) != STATUS_OK)

nvidia@tegra-ubuntu:~/tegra_multimedia_api/argus/build/samples/openglBox$ ./argus_openglbox -s 30
Executing Argus Sample: argus_openglbox
Argus Version: 0.96.2 (multi-process)
PRODUCER: Creating output stream
PRODUCER: Launching consumer thread
CONSUMER: Creating context.
CONSUMER: Connecting to stream.
CONSUMER: Connected to stream.
CONSUMER: Waiting until producer is connected...
PRODUCER: Available Sensor modes :
PRODUCER: [0] W=2592 H=1944
PRODUCER: [1] W=2592 H=1458
PRODUCER: [2] W=1280 H=720
PRODUCER: Starting repeat capture requests.
CONSUMER: Producer is connected; continuing.
CONSUMER: No more frames. Cleaning up, total frame 1798 .
CONSUMER: Done.
PRODUCER: Done -- exiting.

It does not have sensor mode selection in default sample and always selects 2592x1944p30. Adding the logic of selecting 1280x720p120 gives same result as gstreamer pipeline.

3

You are correct; setting the 720p sensor mode in the code did in fact bump the frame rate up to 60 fps!

Thanks for the help.