How to handle irregular long processes in between acquire and release of new events

Dear Nvidia Support,

In the current setup our SensorManager handles 5 GMSL-cameras. The brief process description is:

Sometimes, the camera frame processing pipeline takes too long and consequently the event acquisition of a new frame is too late. This results in dropped frames. We tried to create a deep copy of the dwCameraFrameHandle_t. This would allow me to be independent of the processing time and to release the event quickly. However, we did not found a way to copy it, because it seems to be an incomplete type:

typedef struct dwCameraFrame* dwCameraFrameHandle_t;

Now, we are wondering how this challenge is usually handled. How do you acquire and release events quick enough to be resilient to the cases where processing an event takes longer than usual? Is it possible to acquire new events before the already polled events are handled and released?

Thanks for your help
Andreas

Please provide the following info (check/uncheck the boxes after creating this topic):
Software Version
DRIVE OS Linux 5.2.6
DRIVE OS Linux 5.2.6 and DriveWorks 4.0
DRIVE OS Linux 5.2.0
DRIVE OS Linux 5.2.0 and DriveWorks 3.5
NVIDIA DRIVE™ Software 10.0 (Linux)
NVIDIA DRIVE™ Software 9.0 (Linux)
other DRIVE OS version
other

Target Operating System
Linux
QNX
other

Hardware Platform
NVIDIA DRIVE™ AGX Xavier DevKit (E3550)
NVIDIA DRIVE™ AGX Pegasus DevKit (E3550)
other

SDK Manager Version
1.7.1.8928
other

Host Machine Version
native Ubuntu 18.04
other

Dear @andreas.schwager,
the camera frame processing pipeline takes too long and consequently the event acquisition of a new frame is too late

Does that mean step 2 is taking too long?

Wait for the 5 tasks to be finished

That means you are processing a frame from each camera before process next frame of a camera?

May I know what is the objective of the application? Is it like you need to data from 5 cameras and process them?

Dear SivaRamaKrishnaNV,

thank you for your quick response!

The process started asynchronously in step 2 takes too long in some cases.
On average the process takes 15-20ms, but occasionally it goes up to 60-80 ms. The executed code is always the same. Each camera frame is processed separately.

std::async(std::launch::async,fnc , cameraObject, dwCameraFrameHandle_t))

In step 3 we wait for all asynchronous tasks to be finished. Each frame is processed in parallel, but the wait check is implemented sequentially.

  for (const auto & status: asyncStatus)
  {
    status.wait();
  }

In the launched “fnc” we get the raw image from the frame only for reading the meta data. Then we get the processed image to transform it and publish it to ROS. We need every image from each camera without any frame drop. Hence, we need a method to be resilient against longer processing times in step 2. Is there a typical method to decouple the step of acquiring and releasing the event from the event processing?
We may replace the async launch by other threading methods, but the base issue will be the same. The acquire and release loop is depending on a fast processing of the acquired event.

Thanks
Andreas

Dear @andreas.schwager,
Why there is need to wait and sync for five frames? If you have to sync for every 5 frames, I don’t see anyway to make it resilent as when an event is ready some thread has to be there to aquire it and process. What if we make sensorManager shared across threads and each thread requests for event to be acquired and process that event? Else, you can check increasing poolsize in dwSensorManager_initialize()?

Dear SivaRamaKrishnaNV,

we are afraid there was a misunderstanding. We acquire one event. This event contains 5 dwCameraFrameHandle_t. We need to wait with the release of the event until all CameraFrameHandles are processed. Each CameraFrameHandle is processed in parallel, and we don’t need the processing to be finished at the same time. However, we need all processing to be finished before we miss the next event. We wait for the processing, because an early release of an event usually results in errors during the processing due to not having an early opportunity to create a deep copy of the CameraFrameHandle data. So, we don’t need to synchronize 5 events, we just need to wait for the processing of the 5 frame handles, taken from one event.

The question is if there is a way to be more independent from the processing time.
Can we acquire multiple events after another, before the first one is released?
Is it possible to deep copy an event or a CameraFrameHandle?
How do you usually acquire an event and process it?
Do you start multiple SensorManagers?

We had a look at the DriveWorks sample applications, and they also seem to follow the steps sequentially: Acquire, process, release. In most cases this process is fine, but sometimes the processing takes longer than accepted by the camera’s frame-rate. Hence, we are looking for a suggestion how to optimize the flow. We expect the challenge of processing time peaks and delayed event acquisition and frame drops to be common.

The poolSize is set in dwSensorManager_initializeFromRigWithParams. We tested values from 10 to 1024 and it did not seem to make a difference.

What if we make sensorManager shared across threads and each thread requests for event to be acquired and process that event?
That would imply acquiring multiple events in parallel is possible.
So, you suggest using a thread pool. Each thread acquires an event, starts the 5 processing threads and releases the event when finished. Meanwhile, the next thread is calling acquire next event before the last event is released.

Best Regards
as

Dear @andreas.schwager,
Releasing an event is not advisable with out completing the processing.
As you are using only cameras, how about extending the sample_camera and process each camera frame individually?
In general, it is expected to finish the task with in frame rate range to avoid missing of frame. I will check internally about this use case and let you know if any WAR can be suggested,