SensorManager frame drops

zolnierczyk · February 3, 2020, 3:38pm

We build sensor pipeline using SensorManager functionality. When we were using single camera and other various sensors like IMU, CAN, GPS we get everything in time without drops or delays. We are using rig file to initialize Sensor Manager and when we add multiple camera sensor strange things occurs. We got for example from 3 cameras only 3 frames (so one frame per camera) every 100 ms and other frames which should arrived in that time are reported as dropped. Workflow presented in documentation and samples are dead simple and I don’t known what I can tweak.

We also try to set dispatcher function to Sensor Manager and it look promising because it seems to spawn thread per sensor. Sadly I again get strange behaviour that after reciving first frames programs just hangs up. I also don’t find any example which was using dispatcher function sadly.

Did someone also notice such behaviour?

VickNV · February 3, 2020, 4:40pm

Hi zolnierczyk,

I’m not aware of this issue. Could you try if the issue is reproducible with less sensor types in your rig file or 2 cameras? Thanks!

zolnierczyk · February 4, 2020, 1:41pm

Hi VickNV,

In test I am using rig file with vehicle definition and one or two or three cameras (not other sensors only cameras). Code is implemented upon sample framework (main class derived from DriveWorksSample) provided with driveworks library. Camera frame processing is only extracting gl frames for visualization purpose. I made some profiling and here are results. Data description:

CameraEvent - Time between consecutive start of camera frames processing
CameraDelta - Time of camera frames processing
AcqToCamera - Time between call of dwSensorManager_acquireNextEvent and start of camera frames processing

For single camera (min, mean, max) in miliseconds:

CameraEvent  33    33.3274   34
CameraDelta   2     2.6995    5
AcqToCamera  19    28.5157   30

For double camera (min, mean, max) in miliseconds:

CameraEvent 64   66.6637   69
CameraDelta  5    5.3008    9
AcqToCamera 35   58.9469   62

For triple camera (min, mean, max) in miliseconds:

CameraEvent 131   133.32   135
CameraDelta   6     7.17    10
AcqToCamera 101   123.70   125

CameraDelta timing between those setups are behave correctly. Processing is done sequentially and we see linear increase in time. But CameraEvent and AcqToCamera timing are very poor. I see that program waits on call dwSensorManager_acquireNextEvent way to much time. For single camera we have correct result about 30 frames per second but for two cameras we go 15 frames per seconds? Mayby I am just missing some base stuff here but I was supposing that I get every 30 ms sensor camera event with one or two or more camera frames according what I define in rig file.

I can normally run multiple camera samples without problems and delays like drivenet_ncams but I discover that non of that samples using multiple cameras are implement with Sensor Manager. Should I move away from Sensor Manager workflow?

Thanks for replay!

VickNV · February 4, 2020, 8:05pm

Please try with https://docs.nvidia.com/driveworks/dwx_stereo_calibration_sample.html . I tried the sample with two gmsl cameras but didn’t tell the issue.

I checked timestamps from dwImage_getTimestamp() and the difference between consecutive images of the same cameras were always about 33.3 msec.

zolnierczyk · February 6, 2020, 2:06pm

Hi VickNV,

Thanks for pointing me that sample it is very informative. Sadly I found it also malfunction on my setup. And I also spot that I got the same image for both camera sensor.

Here is profiling log from sample using demo movie:

Timing results:
Thread main:
-onProcess                CPU: 20757us, std=1772       | GPU: 20636us, std=1765       | samples=2524
-onRender                 CPU:  1029us, std= 402       | GPU:  1043us, std= 405       | samples=2524

[06-02-2020 13:52:06] Sensor statistics for: camera.virtual, video=.././data/samples/recordings/stereo0/video-left.h264,timestamp=.././data/samples/recordings/stereo0/video-time-left.txt,smoothing=true
[06-02-2020 13:52:06] Events: 0
Errors: 0
Drops: 0
minDelta: 33000.0000000
maxDelta: 67000.0000000
meanDelta: 33386.132813
Standard deviation: 1419.314941
[06-02-2020 13:52:06] Sensor statistics for: camera.virtual, video=.././data/samples/recordings/stereo0/video-right.h264,timestamp=.././data/samples/recordings/stereo0/video-time-right.txt,smoothing=true
[06-02-2020 13:52:06] Events: 0
Errors: 0
Drops: 0
minDelta: 33000.0000000
maxDelta: 67000.0000000
meanDelta: 33386.132813
Standard deviation: 1419.314941

Looks fine. But here is log from sample using my current setup:

[06-02-2020 13:53:54] Platform: Detected DDPX - Tegra A
[06-02-2020 13:53:54] TimeSource: monotonic epoch time offset is 1580986584740608
[06-02-2020 13:53:54] PTP Time is available from NVPPS Driver
[06-02-2020 13:53:54] Platform: number of GPU devices detected 1
[06-02-2020 13:53:54] Platform: currently selected GPU device integrated ID 0
[06-02-2020 13:53:55] SDK: Resources mounted from .././data/
[06-02-2020 13:53:55] SDK: Create NvMediaDevice
[06-02-2020 13:53:55] SDK: use EGL display as provided
[06-02-2020 13:53:55] TimeSource: monotonic epoch time offset is 1580986584740608
[06-02-2020 13:53:55] PTP Time is available from NVPPS Driver
[06-02-2020 13:53:55] Initialize DriveWorks SDK v2.2.3136
[06-02-2020 13:53:55] Release build with GNU 7.3.1 from heads/buildbrain-branch-0-gca7b4b26e65 against Drive PDK v5.1.6.1
[06-02-2020 13:53:55] Loading rig file: /home/nvidia/redmine_repo/niro-workspace/data/front_cam_can_imu.json
[06-02-2020 13:53:55] SensorFactory::createSensor() -> camera.gmsl, camera-type=ar0231-rccb-bae-sf3324,camera-group=a,slave=0,camera-mask=1000,camera-count=4
[06-02-2020 13:53:55] CameraGMSL: Couldnt set CameraGMSL setup thread sched params!
[06-02-2020 13:53:55] CameraGMSL: Create NvMediaIPPManager
[06-02-2020 13:53:55] CameraGMSL: required FPS = 30, resolution = 1920x1208
nvmedia isc: GetCameraPowerControlLevel: 936: Camera power control library: NVCCP
Max96712 Rev 2 detected!
MAX96712: Enable periodic AEQ on Link 0
MAX96705: Pre-emphasis set to 0xaa
Sensor AR0231 RCCB Rev7 detected!
[06-02-2020 13:53:56] CameraGMSL: default IPP event callback function set
[06-02-2020 13:53:56] 1
[06-02-2020 13:53:56] SensorManager: started
[06-02-2020 13:53:56] Initialize DriveWorks VisualizationSDK v2.2.3136
[06-02-2020 13:53:56] Initialize DriveWorksGL SDK v2.2.3136
[06-02-2020 13:53:56] StreamConsumerGL: successfully initialized
[06-02-2020 13:53:56] StreamProducerCUDA: successfully initialized
[06-02-2020 13:53:56] StreamConsumerGL: successfully initialized
[06-02-2020 13:53:56] StreamProducerCUDA: successfully initialized
terminate called after throwing an instance of 'std::runtime_error'
  what():  Left and right timestamps must have the same value. Cameras are not synchronized.

Something is really broken on my setup. Please look that we can see that only one sensor is initialized. Which is very strange because in rig file we have two sensors also sample confirm that we get two sensors. I modify sample_calibration_stereo just to not throw exception to see what we get. Here is log:

Timing results:
Thread main:
-onProcess                CPU: 64405us, std=3997       | GPU: 64280us, std=3999       | samples=263
-onRender                 CPU:  1082us, std=1637       | GPU:  1086us, std=1640       | samples=263

[06-02-2020 14:00:56] Sensor statistics for: camera.gmsl, camera-type=ar0231-rccb-bae-sf3324,camera-group=a,slave=0,camera-mask=1000,camera-count=4
[06-02-2020 14:00:56] Events: 529
Errors: 0
Drops: 0
minDelta: 33284.0000000
maxDelta: 33377.0000000
meanDelta: 33333.253906
Standard deviation: 18.385599

Timing result for onProcess function is about 64 ms which is why I got 15 fps. And also why I got statistic only for first sensor and not for second?
Here are printed timestamps from images:

Timestamps: Left: 1580997655996116 Right: 1580997655962757 delta: 33.359 
Timestamps: Left: 1580997656062772 Right: 1580997656029432 delta: 33.34 
Timestamps: Left: 1580997656129449 Right: 1580997656096087 delta: 33.362 
Timestamps: Left: 1580997656196104 Right: 1580997656162794 delta: 33.31 
Timestamps: Left: 1580997656262754 Right: 1580997656229445 delta: 33.309 
Timestamps: Left: 1580997656329431 Right: 1580997656296121 delta: 33.31 
Timestamps: Left: 1580997656396093 Right: 1580997656362775 delta: 33.318 
Timestamps: Left: 1580997656462770 Right: 1580997656429454 delta: 33.316 
Timestamps: Left: 1580997656529421 Right: 1580997656496107 delta: 33.314 
Timestamps: Left: 1580997656596099 Right: 1580997656562787 delta: 33.312

Looks like left camera is about 30ms after right camera and also we got also about 60ms after previous left one.

And what most strange is that I got left and right camera exactly the same image?!
Below I past my rig configuration without boilerplate numbers:

{
    "rig": {
        "sensors": [
{
	"correction_rig_T": [ ... ],
	"correction_sensor_R_FLU": {
	    "roll-pitch-yaw": [ ... ]
	},
	"name": "FrontRight-A0-60",
	"nominalSensor2Rig_FLU": {
	    "roll-pitch-yaw": [ ... ],
	    "t": [ ... ]
	},
	"parameter": "camera-type=ar0231-rccb-bae-sf3324,camera-group=a,slave=0,camera-mask=1000,camera-count=4",
	"properties": {
	    "Model": "ftheta", [ ... ]
	},
	"protocol": "camera.gmsl"
},
{
"correction_rig_T": [ ... ]
],
"correction_sensor_R_FLU": {
    "roll-pitch-yaw": [ ... ]
},
"name": "FrontLeft-A1-60",
"nominalSensor2Rig_FLU": {
    "roll-pitch-yaw": [ ... ],
    "t": [ ... ]
},
"parameter": "camera-type=ar0231-rccb-bae-sf3324,camera-group=a,slave=0,camera-mask=1100,camera-count=4",
"properties": {
    "Model": "ftheta", [ ... ]
},
"protocol": "camera.gmsl"
}
 
        ],
        "json-merge": "rig/vehicle.json"
    },
    "version": 2
}

I hope that I made some stupid mistake with parameters string with rig :)

Thanks for replay!

VickNV · February 6, 2020, 2:49pm

And what most strange is that I got left and right camera exactly the same image?!
Below I past my rig configuration without boilerplate numbers:

{
    "rig": {
        "sensors": [
{
	"correction_rig_T": [ ... ],
	"correction_sensor_R_FLU": {
	    "roll-pitch-yaw": [ ... ]
	},
	"name": "FrontRight-A0-60",
	"nominalSensor2Rig_FLU": {
	    "roll-pitch-yaw": [ ... ],
	    "t": [ ... ]
	},
	"parameter": "camera-type=ar0231-rccb-bae-sf3324,camera-group=a,slave=0,camera-mask=1000,camera-count=4",
	"properties": {
	    "Model": "ftheta", [ ... ]
	},
	"protocol": "camera.gmsl"
},
{
"correction_rig_T": [ ... ]
],
"correction_sensor_R_FLU": {
    "roll-pitch-yaw": [ ... ]
},
"name": "FrontLeft-A1-60",
"nominalSensor2Rig_FLU": {
    "roll-pitch-yaw": [ ... ],
    "t": [ ... ]
},
"parameter": "camera-type=ar0231-rccb-bae-sf3324,camera-group=a,slave=0,camera-mask=1100,camera-count=4",
"properties": {
    "Model": "ftheta", [ ... ]
},
"protocol": "camera.gmsl"
}
 
        ],
        "json-merge": "rig/vehicle.json"
    },
    "version": 2
}

Do you think this is the cause of only one camera sensor initialized? about 64 ms is because both left and right frames are from the same camera sensor?

"correction_rig_T": [ ... ]
],

zolnierczyk · February 7, 2020, 2:55pm

Hi VickNV,

Thanks for support and pointing me to that sample! The problem was indeed with rig file and exactly in string parameter:

camera-type=ar0231-rccb-bae-sf3324,camera-group=a,slave=0,camera-mask=1000,camera-count=4

I forgot to add siblingIndex that define which camera I would like to get from camera group. Without it every entry in rig file just define sensors with the same camera as source and there for I got such strange results.

Thanks for replay!

VickNV · February 7, 2020, 4:24pm

Good to hear that your clarified the issue and thanks for sharing back!

Topic		Replies	Views
Error Running Multiple Cameras Sample DRIVE AGX Xavier General driveos-nvmedia	21	1612	June 27, 2022
Camera Stream Encoding to H.264 on Drive AGX Orin DRIVE AGX Orin General driveos-nvmedia	38	1218	March 18, 2024
Custom camera work well in "nvsipl_camera" sample but not in driveworks sample DRIVE AGX Orin General driveos-nvmedia	10	1440	June 9, 2023
The nvarguscamerasrc plugin is Frame loss Jetson AGX Orin camera	23	841	January 17, 2024
Video freeze for multiple camera streaming Jetson TX2 camera	9	233	August 28, 2024
How to record with the ar0144 gaze camera? DRIVE AGX Xavier General driveworks	6	804	October 12, 2021
Sekonix cameras: Cannot create a sensor dwSAL_createSensor(..) if more then one camera selected DRIVE AGX Xavier General driveos-nvmedia , driveworks-cameras	8	1451	October 12, 2021
GMSL camera with Drive PX2 DRIVE Hardware	33	4238	September 24, 2020
recorder-tui : could not create NvMedia Image stream producer DriveWorks	6	1047	March 12, 2020
Recorder application failing DRIVE AGX Xavier General driveworks-recording	6	757	February 24, 2022

SensorManager frame drops

Related topics