# How to handle irregular long processes in between acquire and release of new events **URL:** \ **Category:** DRIVE AGX Xavier General\ **Tags:** driveworks-cameras\ **Created:** [March 28, 2022, 3:09pm UTC](https://forums.developer.nvidia.com/t/how-to-handle-irregular-long-processes-in-between-acquire-and-release-of-new-events/209579 "2022-03-28T15:09:03Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![andreas.schwager](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@andreas.schwager](https://forums.developer.nvidia.com/u/andreas.schwager)\ **Post date:** [March 28, 2022, 3:09pm UTC](https://forums.developer.nvidia.com/t/how-to-handle-irregular-long-processes-in-between-acquire-and-release-of-new-events/209579/1 "2022-03-28T15:09:03Z")
Dear Nvidia Support, In the current setup our SensorManager handles 5 GMSL-cameras. The brief process description is: - Acquire event with [dwSensorManager\_acquireNextEvent](https://docs.nvidia.com/drive/driveworks-3.5/group __sensormanager__ group.html#ga56e926417609649e645dae125400931e) - Launch async task immediately for each camera for processing - Wait for the 5 tasks to be finished - Release event with [dwSensorManager\_releaseAcquiredEvent](https://docs.nvidia.com/drive/driveworks-3.5/group __sensormanager__ group.html#ga7cd6202c1b27d94460d884ee6df44d63) Sometimes, the camera frame processing pipeline takes too long and consequently the event acquisition of a new frame is too late. This results in dropped frames. We tried to create a deep copy of the [dwCameraFrameHandle\_t](https://docs.nvidia.com/drive/driveworks-3.5/group __camera__ group.html#ga72de5f2a89bf7c65ed27cfb7d0805e45). This would allow me to be independent of the processing time and to release the event quickly. However, we did not found a way to copy it, because it seems to be an incomplete type: typedef struct dwCameraFrame\* [dwCameraFrameHandle\_t](https://docs.nvidia.com/drive/driveworks-3.5/group __camera__ group.html#ga72de5f2a89bf7c65ed27cfb7d0805e45); Now, we are wondering how this challenge is usually handled. How do you acquire and release events quick enough to be resilient to the cases where processing an event takes longer than usual? Is it possible to acquire new events before the already polled events are handled and released? Thanks for your help Andreas Please provide the following info (check/uncheck the boxes after creating this topic): **Software Version** DRIVE OS Linux 5.2.6 DRIVE OS Linux 5.2.6 and DriveWorks 4.0 DRIVE OS Linux 5.2.0 DRIVE OS Linux 5.2.0 and DriveWorks 3.5 NVIDIA DRIVE™ Software 10.0 (Linux) NVIDIA DRIVE™ Software 9.0 (Linux) other DRIVE OS version other **Target Operating System** Linux QNX other **Hardware Platform** NVIDIA DRIVE™ AGX Xavier DevKit (E3550) NVIDIA DRIVE™ AGX Pegasus DevKit (E3550) other **SDK Manager Version** 1.7.1.8928 other **Host Machine Version** native Ubuntu 18.04 other ---
**Author:** ![SivaRamaKrishnaNV](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/sivaramakrishnanv/32/14043_2.png) [@SivaRamaKrishnaNV](https://forums.developer.nvidia.com/u/SivaRamaKrishnaNV)\ **Post date:** [March 28, 2022, 7:15pm UTC](https://forums.developer.nvidia.com/t/how-to-handle-irregular-long-processes-in-between-acquire-and-release-of-new-events/209579/2 "2022-03-28T19:15:22Z")
Dear @andreas.schwager, the camera frame processing pipeline takes too long and consequently the event acquisition of a new frame is too late > > Does that mean step 2 is taking too long? Wait for the 5 tasks to be finished > > That means you are processing a frame from each camera before process next frame of a camera? May I know what is the objective of the application? Is it like you need to data from 5 cameras and process them? ---
**Author:** ![andreas.schwager](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@andreas.schwager](https://forums.developer.nvidia.com/u/andreas.schwager)\ **Post date:** [March 29, 2022, 11:20am UTC](https://forums.developer.nvidia.com/t/how-to-handle-irregular-long-processes-in-between-acquire-and-release-of-new-events/209579/4 "2022-03-29T11:20:00Z")
Dear SivaRamaKrishnaNV, thank you for your quick response! The process started asynchronously in step 2 takes too long in some cases. On average the process takes 15-20ms, but occasionally it goes up to 60-80 ms. The executed code is always the same. Each camera frame is processed separately. ```auto std::async(std::launch::async,fnc , cameraObject, dwCameraFrameHandle_t)) ``` In step 3 we wait for all asynchronous tasks to be finished. Each frame is processed in parallel, but the wait check is implemented sequentially. ```auto for (const auto & status: asyncStatus) { status.wait(); } ``` In the launched “fnc” we get the raw image from the frame only for reading the meta data. Then we get the processed image to transform it and publish it to ROS. We need every image from each camera without any frame drop. Hence, we need a method to be resilient against longer processing times in step 2. Is there a typical method to decouple the step of acquiring and releasing the event from the event processing? We may replace the `async` launch by other threading methods, but the base issue will be the same. The acquire and release loop is depending on a fast processing of the acquired event. Thanks Andreas ---
**Author:** ![SivaRamaKrishnaNV](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/sivaramakrishnanv/32/14043_2.png) [@SivaRamaKrishnaNV](https://forums.developer.nvidia.com/u/SivaRamaKrishnaNV)\ **Post date:** [March 30, 2022, 4:57pm UTC](https://forums.developer.nvidia.com/t/how-to-handle-irregular-long-processes-in-between-acquire-and-release-of-new-events/209579/5 "2022-03-30T16:57:54Z")
Dear @andreas.schwager, Why there is need to wait and sync for five frames? If you have to sync for every 5 frames, I don’t see anyway to make it resilent as when an event is ready some thread has to be there to aquire it and process. What if we make sensorManager shared across threads and each thread requests for event to be acquired and process that event? Else, you can check increasing poolsize in dwSensorManager\_initialize()? ---
**Author:** ![andreas.schwager](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@andreas.schwager](https://forums.developer.nvidia.com/u/andreas.schwager)\ **Post date:** [March 31, 2022, 12:47pm UTC](https://forums.developer.nvidia.com/t/how-to-handle-irregular-long-processes-in-between-acquire-and-release-of-new-events/209579/6 "2022-03-31T12:47:43Z")
Dear SivaRamaKrishnaNV, we are afraid there was a misunderstanding. We [acquire](https://docs.nvidia.com/drive/driveworks-3.5/group __sensormanager__ group.html#ga56e926417609649e645dae125400931e) one [event](https://docs.nvidia.com/drive/driveworks-3.5/group __sensormanager__ group.html#structdwSensorEvent). This event contains 5 [dwCameraFrameHandle\_t](https://docs.nvidia.com/drive/driveworks-3.5/group __camera__ group.html#ga72de5f2a89bf7c65ed27cfb7d0805e45). We need to wait with the [release](https://docs.nvidia.com/drive/driveworks-3.5/group __sensormanager__ group.html#ga7cd6202c1b27d94460d884ee6df44d63) of the event until all CameraFrameHandles are processed. Each CameraFrameHandle is processed in parallel, and we don’t need the processing to be finished at the same time. However, we need all processing to be finished before we miss the next event. We wait for the processing, because an early release of an event usually results in errors during the processing due to not having an early opportunity to create a deep copy of the CameraFrameHandle data. So, we don’t need to synchronize 5 events, we just need to wait for the processing of the 5 frame handles, taken from one event. The question is if there is a way to be more independent from the processing time. Can we acquire multiple events after another, before the first one is released? Is it possible to deep copy an event or a CameraFrameHandle? How do you usually acquire an event and process it? Do you start multiple SensorManagers? We had a look at the DriveWorks sample applications, and they also seem to follow the steps sequentially: Acquire, process, release. In most cases this process is fine, but sometimes the processing takes longer than accepted by the camera’s frame-rate. Hence, we are looking for a suggestion how to optimize the flow. We expect the challenge of processing time peaks and delayed event acquisition and frame drops to be common. The poolSize is set in [dwSensorManager\_initializeFromRigWithParams](https://docs.nvidia.com/drive/driveworks-3.5/group __sensormanager__ group.html#ga75d44dd078442dcec4b1a800f9efc53d). We tested values from 10 to 1024 and it did not seem to make a difference. `What if we make sensorManager shared across threads and each thread requests for event to be acquired and process that event?` That would imply acquiring multiple events in parallel is possible. So, you suggest using a thread pool. Each thread acquires an event, starts the 5 processing threads and releases the event when finished. Meanwhile, the next thread is calling acquire next event before the last event is released. Best Regards as ---
**Author:** ![SivaRamaKrishnaNV](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/sivaramakrishnanv/32/14043_2.png) [@SivaRamaKrishnaNV](https://forums.developer.nvidia.com/u/SivaRamaKrishnaNV)\ **Post date:** [April 12, 2022, 10:07am UTC](https://forums.developer.nvidia.com/t/how-to-handle-irregular-long-processes-in-between-acquire-and-release-of-new-events/209579/7 "2022-04-12T10:07:01Z")
Dear @andreas.schwager, Releasing an event is not advisable with out completing the processing. As you are using only cameras, how about extending the sample\_camera and process each camera frame individually? In general, it is expected to finish the task with in frame rate range to avoid missing of frame. I will check internally about this use case and let you know if any WAR can be suggested,