SIPL — faster sensor is throttled to the slower sensor's frame rate

Robotics & Edge Computing Jetson Systems Jetson Thor
,
1

We are using a single SIPLCaptureOp (holoscan-sensor-bridge) to capture two sensors at the same time through one SIPL

sensor A runs at 30 Hz and sensor B at 45 Hz.

What we observe: sensor B is not delivered at 45 Hz. Both sensors come out at 30 Hz, i.e. the faster sensor isthrottled down to the slower sensor’s rate, and sensor B loses roughly 1/3 of its frames (its frame-number fieldstill increments correctly at the source, so the sensor itself is producing 45 Hz — the frames are dropped incapture).

Question: Is this expected behavior when capturing multiple sensors with different frame rates through SIPL? Ismixed-rate multi-sensor capture supported, and if so how should it be configured?

Looking at the operator source (src/hololink/operators/sipl_capture/sipl_capture.cpp), it appears to be caused by:

compute() builds one combined entity for all sensors and emits it once per tick, and it waits for every sensor’sbuffer before emitting — so the tick rate is gated by the slowest sensor.

The per-sensor acquire_buffer_thread_func keeps only the latest buffer and Release()s any older un-consumed frame(the in-code comment notes “compute() may miss frames and the last known frame is always used”).

So a sensor faster than the combined tick rate appears to lose frames by design. Could you confirm whether this isintended, and advise the recommended approach for capturing sensors of different frame rates?

We also tested splitting the sensors across two SIPLCaptureOp instances in one process (one per rate group). This fails: the first instance initializes and streams, but the second instance’s transport driver fails at probe with “Hololink is already started”, and SetPlatformCfg returns status=10. The two transport drivers resolve to the same shared Hololink object (registry keyed by device serial number), so the second start() is rejected.

Question: Is running multiple SIPLCaptureOp / INvSIPLCamera instances against one HSB device supported at all? If not, all sensors of one HSB must go through a single capture instance — which means mixed frame rates cannot be supported.

3

hello daegyu,

actually, we did not test dual camera with different frame-rate on reference platform.

may I have more details of your use-case to capture two sensors at the same time through one SIPL?
you should have SIPL pipeline config per sensor, and its sensor properties carry per-sensor frameRate.

4

Hello,

We are currently using AGX Thor and testing five heterogeneous sensors (camera, radar, LiDAR, etc.) through a single HSB.

Right now, only up to two sensors can be used, but we have confirmed that this limitation is expected to be removed in JetPack 7.2. Our final target is to operate all five sensors through SIPL.

So far, we have verified that each sensor works correctly with the standard LinuxReceiver path. We also implemented SIPL UDDFs and confirmed that each sensor works correctly when tested individually in SIPL.

The problem described above appeared during the integration stage, when we began testing multiple different sensors together. After checking the SIPLCaptureOp source code, it appears that this is a structural limitation of the current implementation, which is why I asked this question.

In other words, our use case is a multi-sensor integration scenario where multiple heterogeneous sensors need to run simultaneously through one HSB and SIPL.

6

hello daegyu,

I’ve check the code, and it looks JP-7.2 not only remove 2-cam limitation but also have data channel created per sensor configuration.
so, JP-7.2 should support multiple heterogeneous sensors. it’s suggest to wait for JP-7.2 public release available to have verification.

7

Just to make sure I understand correctly: does this mean that this heterogeneous multi-sensor use case is not supported in the current JetPack release yet?

Also, in JP-7.2, besides removing the current two-camera limitation, will a separate data channel be created for each sensor configuration?
If that is the case, can we understand that sensors with different frame rates will be able to run simultaneously through a single HSB and SIPL, with each sensor operating at its own native rate?

8

hello daegyu,

your understand is correct. the upcoming code change running per sensor configuration, it should works for your use-case.
however, it just me checking the code for sharing the assumption, we don’t have heterogeneous multi-sensor for validation.
let’s wait for JP-7.2 public release, please start a new discussion thread if you need any supports.