Blotchy witch GMSL camera and driveworks API

Hardware Platform: DRIVE AGX Pegasus™ Developer Kit
Software Version: DRIVE Software 10
Host Machine Version: native Ubuntu 18.04
SDK Manager Version: 1.0.1.5538

Hi,
I’m trying to set up a video pipeline on pegasus, using driveworks sdk with a gmsl camera that came with the development kit.
After reading the frames from the sensor, I want to encode them to H265 and send them as raw UDP.
I’m using the camera_gmsl sample as a reference.
I have 2 questions:

  1. My problem is that the video coming out from this pipeline is not looking as expected. The video is blotchy and wavy, and the lightning is blinking fast.
    I’m running the sample itself to make sure it’s not my code that’s doing it. The sample is acting the same. But when running the camera_gmsl_raw sample, the video looks a lot better.
    I’m attaching 2 videos that demonstrate the abovesamples.tar.gz (2.9 MB) .
    Unfortunately I’m unable to mock the sample in my code, keeping the H265 encoding.
    This is my original code that worked fine but with blotchy video:
    // ... Initializing ...

    dwSensorParams params{};
    std::string parameterString = "fifo-size=3";
    parameterString += ",output-format=yuv";
    parameterString += ",camera-type=ar0231-rccb-bae-sf3324";
    parameterString += ",camera-group=a";
    params.parameters = parameterString.c_str();
    params.protocol = "camera.gmsl";
    CHECK_DW_ERROR(dwSAL_createSensor(&m_camera, params, m_sal));

    // ...

    std::string newParams = "";
    newParams += "format=h265";
    newParams += ",bitrate=1000000";
    newParams += ",framerate=30";
    newParams += ",type=user";
    m_serializerParams.parameters = newParams.c_str();
    CHECK_DW_ERROR(dwSensorSerializer_initialize(&m_serializer, &m_serializerParams, m_camera));

    // ... Processing ...

    if (m_frame)
    {
        CHECK_DW_ERROR(dwSensorCamera_returnFrame(&m_frame));
    }

    CHECK_DW_ERROR(dwSensorCamera_readFrame(&m_frame, 0, timeout, m_camera));
    CHECK_DW_ERROR(dwSensorSerPreformatted textializer_serializeCameraFrameAsync(m_frame, m_serializer));

And when I tried to change sensor’s output-format to raw, I get DW_INVALING_ARGUMENT from the dwSensorSerializer_initialize function.

  1. I couldn’t find in the SDK reference how to change the GOP of the encoder. Is that possible using the driveworks api?

Hi @eladg1,

Please share the command and steps about how to reproduce the issue with camera_gmsl sample. Thanks!

Why do you want to change? What do you want to change (something like NvMediaEncodeConfigH265::gopLength
in NvMedia)?

Hi VickNV,
thank you for the fast response!

I run the sample like that: ./sample_camera_gmsl from /usr/local/driveworks-2.2/bin/

I’m want to run a law latency video pipeline on a bad network that can sometimes yield high packet loss and jitter.
When the GOP is too high, and I’m losing some P frames, the video shows it. That’s why I want to keep the GOP as low as 15 to 30. The NVMedia API that you referenced is exactly what I’m looking but I’m not sure how to integrate it with Driveworks API. Is that event possible?

Thank you!

I want to highlight that IPP framework will be deprecated from the next release.
So I would suggest you put your efforts into sample_camera_gmsl_custom instead.
You can try if the issue is also reproducible on it.

We just checked our implementation. it is set as 16 now.

If you want to figure out, maybe you can use nvmvid_enc to manipulate the settings for your use cases. Thanks!

Please help to check if you can observe the issue with sample_camera_gmsl_custom (SIPL based application). Thanks!

So I’ve tried using the sample_camera_gmsl_custom as suggested, but I got the following error:

[21-09-2019 01:02:53] Platform: Detected DDPX - Tegra A
[21-09-2019 01:02:53] TimeSource: monotonic epoch time offset is 1569027511545177
[21-09-2019 01:02:53] PTP Time is available from NVPPS Driver
[21-09-2019 01:02:53] Platform: number of GPU devices detected 2
[21-09-2019 01:02:53] Platform: currently selected GPU device discrete ID 0
[21-09-2019 01:02:53] SDK: Resources mounted from .././data/
[21-09-2019 01:02:53] SDK: Create NvMediaDevice
[21-09-2019 01:02:53] SDK: use EGL display as provided
[21-09-2019 01:02:53] TimeSource: monotonic epoch time offset is 1569027511545177
[21-09-2019 01:02:53] PTP Time is available from NVPPS Driver
[21-09-2019 01:02:53] Initialize DriveWorks SDK v2.2.3136
[21-09-2019 01:02:53] Release build with GNU 7.3.1 from heads/buildbrain-branch-0-gca7b4b26e65 against Drive PDK v5.1.6.1
[21-09-2019 01:02:53] Loading rig file: .././data/samples/sensors/camera/camera_gmsl_custom/rig.json
onInitialize: creating camera.gmsl.master with params: config-type=devBlock,configuration={MODE:master|INTERFACE:csi-a|CAMERA_NAME:SF3324|NUM:4},output-format=processed
[21-09-2019 01:02:53] SensorFactory::createSensor() -> camera.gmsl.master, config-type=devBlock,configuration={MODE:master|INTERFACE:csi-a|CAMERA_NAME:SF3324|NUM:4},output-format=processed
libnvsipl_query.so: CNvMCameraModuleInfo.cpp : 76: ParseJson: Could not find matching serializer
libnvsipl_query.so: CNvMDeviceInfoList.cpp : 100: GetCustomCameraModule: Failed to find matching module MAX96712TPG
libnvsipl_query.so: CNvMDeviceBlockInfo.cpp : 156: ParseJson: Failed to find matching camera module
libnvsipl_query.so: CNvMPlatformCfgInfo.cpp : 50: ParseJson: CNvMPlatformCfgInfo::ParseJson failed.
libnvsipl_query.so: CNvMQuery.cpp : 45: ParsePlatformConfig: Failed to parse a PlatformCfg
[21-09-2019 01:02:53] CameraGMSLMaster::parseDevBlock Getting device info list.
[21-09-2019 01:02:53] devBlock: 0 Mode = master Interface = csi-a Camera_name = SF3324 Num = 4
[21-09-2019 01:02:53] Camera Match Name: SF3324 Description: Sekonix SF3324 module - 120-deg FOV, DVP AR0231-RCCB, MAX96705 linkIndex: 4294967295 serInfo.Name: MAX96705
[21-09-2019 01:02:53] SIPLMaster::SIPLMaster: Setting up master camera
[21-09-2019 01:02:53] Platform: devBlockInit
[21-09-2019 01:02:53] Platform Config: devBlockInit
[21-09-2019 01:02:53] Description: Configuration set by Device Block arguments: config-type=devBlock,configuration={MODE:master|INTERFACE:csi-a|CAMERA_NAME:SF3324|NUM:4},output-format=processed
[21-09-2019 01:02:53] Number of device blocks: 1
[21-09-2019 01:02:53] Device Block : 0
[21-09-2019 01:02:53] 	csiPort: 0
[21-09-2019 01:02:53] 	i2cDevice: 0
[21-09-2019 01:02:53] 	Deserializer Name: MAX96712
[21-09-2019 01:02:53] 	Deserializer Description: Maxim 96712 Aggregator
[21-09-2019 01:02:53] 	Deserializer i2cAddress: 41
[21-09-2019 01:02:53] 	Simulator Mode: 0
[21-09-2019 01:02:53] 	Slave Mode: 0
[21-09-2019 01:02:53] 	Phy Mode: 0
[21-09-2019 01:02:53] 	Number of camera modules: 4
[21-09-2019 01:02:53] 	CameraModule index: 0
[21-09-2019 01:02:53] 		Name :SF3324
[21-09-2019 01:02:53] 		Description: Sekonix SF3324 module - 120-deg FOV, DVP AR0231-RCCB, MAX96705
[21-09-2019 01:02:53] 		Serializer name: MAX96705
[21-09-2019 01:02:53] 		Serializer description: Maxim 96705 Serializer
[21-09-2019 01:02:53] 		Serializer i2cAdress: 64
[21-09-2019 01:02:53] 		Number of EEPROMs: 0
[21-09-2019 01:02:53] 		Number of Sensors: 1
[21-09-2019 01:02:53] 		Sensor index: 0
[21-09-2019 01:02:53] 			Sensor ID: 0
[21-09-2019 01:02:53] 			Sensor name: AR0231
[21-09-2019 01:02:53] 			Sensor description: OnSemi AR0231 Sensor
[21-09-2019 01:02:53] 			Sensor i2cAddress: 16
[21-09-2019 01:02:53] 			Sensor isTPGEnabled: 0
[21-09-2019 01:02:53] 			Sensor isTriggerMode: 1
[21-09-2019 01:02:53] 			Sensor Number of Virt. Chan.: 0
[21-09-2019 01:02:53] 			Virt. Chan. Index: 0
[21-09-2019 01:02:53] 				 cfa: 35
[21-09-2019 01:02:53] 				 embeddedTopLines: 24
[21-09-2019 01:02:53] 				 embeddedBottomLines: 4
[21-09-2019 01:02:53] 				 inputFormat: raw12
[21-09-2019 01:02:53] 				 height: 1208
[21-09-2019 01:02:53] 				 width: 1920
[21-09-2019 01:02:53] 				 fps: 30.0000000
[21-09-2019 01:02:53] 				 Embedded Data: 0
[21-09-2019 01:02:53] 	CameraModule index: 1
[21-09-2019 01:02:53] 		Name :SF3324
[21-09-2019 01:02:53] 		Description: Sekonix SF3324 module - 120-deg FOV, DVP AR0231-RCCB, MAX96705
[21-09-2019 01:02:53] 		Serializer name: MAX96705
[21-09-2019 01:02:53] 		Serializer description: Maxim 96705 Serializer
[21-09-2019 01:02:53] 		Serializer i2cAdress: 64
[21-09-2019 01:02:53] 		Number of EEPROMs: 0
[21-09-2019 01:02:53] 		Number of Sensors: 1
[21-09-2019 01:02:53] 		Sensor index: 0
[21-09-2019 01:02:53] 			Sensor ID: 1
[21-09-2019 01:02:53] 			Sensor name: AR0231
[21-09-2019 01:02:53] 			Sensor description: OnSemi AR0231 Sensor
[21-09-2019 01:02:53] 			Sensor i2cAddress: 16
[21-09-2019 01:02:53] 			Sensor isTPGEnabled: 0
[21-09-2019 01:02:53] 			Sensor isTriggerMode: 1
[21-09-2019 01:02:53] 			Sensor Number of Virt. Chan.: 0
[21-09-2019 01:02:53] 			Virt. Chan. Index: 0
[21-09-2019 01:02:53] 				 cfa: 35
[21-09-2019 01:02:53] 				 embeddedTopLines: 24
[21-09-2019 01:02:53] 				 embeddedBottomLines: 4
[21-09-2019 01:02:53] 				 inputFormat: raw12
[21-09-2019 01:02:53] 				 height: 1208
[21-09-2019 01:02:53] 				 width: 1920
[21-09-2019 01:02:53] 				 fps: 30.0000000
[21-09-2019 01:02:53] 				 Embedded Data: 0
[21-09-2019 01:02:53] 	CameraModule index: 2
[21-09-2019 01:02:53] 		Name :SF3324
[21-09-2019 01:02:53] 		Description: Sekonix SF3324 module - 120-deg FOV, DVP AR0231-RCCB, MAX96705
[21-09-2019 01:02:53] 		Serializer name: MAX96705
[21-09-2019 01:02:53] 		Serializer description: Maxim 96705 Serializer
[21-09-2019 01:02:53] 		Serializer i2cAdress: 64
[21-09-2019 01:02:53] 		Number of EEPROMs: 0
[21-09-2019 01:02:53] 		Number of Sensors: 1
[21-09-2019 01:02:53] 		Sensor index: 0
[21-09-2019 01:02:53] 			Sensor ID: 2
[21-09-2019 01:02:53] 			Sensor name: AR0231
[21-09-2019 01:02:53] 			Sensor description: OnSemi AR0231 Sensor
[21-09-2019 01:02:53] 			Sensor i2cAddress: 16
[21-09-2019 01:02:53] 			Sensor isTPGEnabled: 0
[21-09-2019 01:02:53] 			Sensor isTriggerMode: 1
[21-09-2019 01:02:53] 			Sensor Number of Virt. Chan.: 0
[21-09-2019 01:02:53] 			Virt. Chan. Index: 0
[21-09-2019 01:02:53] 				 cfa: 35
[21-09-2019 01:02:53] 				 embeddedTopLines: 24
[21-09-2019 01:02:53] 				 embeddedBottomLines: 4
[21-09-2019 01:02:53] 				 inputFormat: raw12
[21-09-2019 01:02:53] 				 height: 1208
[21-09-2019 01:02:53] 				 width: 1920
[21-09-2019 01:02:53] 				 fps: 30.0000000
[21-09-2019 01:02:53] 				 Embedded Data: 0
[21-09-2019 01:02:53] 	CameraModule index: 3
[21-09-2019 01:02:53] 		Name :SF3324
[21-09-2019 01:02:53] 		Description: Sekonix SF3324 module - 120-deg FOV, DVP AR0231-RCCB, MAX96705
[21-09-2019 01:02:53] 		Serializer name: MAX96705
[21-09-2019 01:02:53] 		Serializer description: Maxim 96705 Serializer
[21-09-2019 01:02:53] 		Serializer i2cAdress: 64
[21-09-2019 01:02:53] 		Number of EEPROMs: 0
[21-09-2019 01:02:53] 		Number of Sensors: 1
[21-09-2019 01:02:53] 		Sensor index: 0
[21-09-2019 01:02:53] 			Sensor ID: 3
[21-09-2019 01:02:53] 			Sensor name: AR0231
[21-09-2019 01:02:53] 			Sensor description: OnSemi AR0231 Sensor
[21-09-2019 01:02:53] 			Sensor i2cAddress: 16
[21-09-2019 01:02:53] 			Sensor isTPGEnabled: 0
[21-09-2019 01:02:53] 			Sensor isTriggerMode: 1
[21-09-2019 01:02:53] 			Sensor Number of Virt. Chan.: 0
[21-09-2019 01:02:53] 			Virt. Chan. Index: 0
[21-09-2019 01:02:53] 				 cfa: 35
[21-09-2019 01:02:53] 				 embeddedTopLines: 24
[21-09-2019 01:02:53] 				 embeddedBottomLines: 4
[21-09-2019 01:02:53] 				 inputFormat: raw12
[21-09-2019 01:02:53] 				 height: 1208
[21-09-2019 01:02:53] 				 width: 1920
[21-09-2019 01:02:53] 				 fps: 30.0000000
[21-09-2019 01:02:53] 				 Embedded Data: 0
[21-09-2019 01:02:53] CameraGMSLMaster::create: Enabling pipeline notifications
[21-09-2019 01:02:53] SIPLMaster:: EnableNotifyer for 4 pipes.
[21-09-2019 01:02:53] CameraGMSLMaster::setOutputDescription Setting output consumer descriptors for sensor: Sensor0HW_ISP0NvMediaImage uIndex: 0 OutputType: 1
[21-09-2019 01:02:53] CameraGMSLMaster::setOutputDescription Setting output consumer descriptors for sensor: Sensor1HW_ISP0NvMediaImage uIndex: 1 OutputType: 1
[21-09-2019 01:02:53] CameraGMSLMaster::setOutputDescription Setting output consumer descriptors for sensor: Sensor2HW_ISP0NvMediaImage uIndex: 2 OutputType: 1
[21-09-2019 01:02:53] CameraGMSLMaster::setOutputDescription Setting output consumer descriptors for sensor: Sensor3HW_ISP0NvMediaImage uIndex: 3 OutputType: 1
[21-09-2019 01:02:53] CameraGMSLMaster:: Initliazing master camera
iscRootDevOpen: can't open /dev/isc-mgr.0.a - Device or resource busy
NvMediaISCRootDeviceCreate: Unable to open root device
libnvsipl_devblk.so: CNvMDeviceBlock.cpp: 135: Init: NvMediaISCRootDeviceCreate failed!
libnvsipl.so: CNvMDeviceBlockMgr.hpp: 90: ThreadFunc: ThreadCreate failed. (7f)
libnvsipl.so: CNvMDeviceBlockMgr.cpp: 83: Init: Failed to initialize device block:0
libnvsipl.so: CNvMCamera.cpp: 169: Init: CNvMCamera::Init failed! with error: 7f
libnvsipl_devblk.so: CNvMDeviceBlock.cpp: 226: Stop: CNvMDeviceBlock has not been started
libnvsipl.so: CNvMDeviceBlockMgr.cpp: 132: Stop: Deviceblock fails to stop
libnvsipl.so: CNvMCamera.cpp: 259: Stop: m_upDeviceBlockMgr fails to stop
[21-09-2019 01:02:53] Driveworks exception thrown: DW_INTERNAL_ERROR: CameraGMSLMaster: Failed to initialize master camera! SIPL errror code = 127

terminate called after throwing an instance of 'std::runtime_error'
  what():  [2019-09-21 01:02:53] DW Error DW_INTERNAL_ERROR executing DW function:
 dwSAL_createSensor(&m_cameraMaster, paramsMaster, m_sal)
 at /dvs/git/dirty/gitlab-master_av/dw/sdk/samples/sensors/camera/camera_gmsl_custom/main.cpp:136
Aborted

What am I doing wrong

Please help to run with below single camera command, screenshot and pinpoint the issue for us to easily communicate with the team. Thanks!

$ /usr/local/driveworks/bin/sample_camera_gmsl_custom --rig=/usr/local/driveworks/data/samples/sensors/camera/camera_gmsl_custom/rigSingleRaw.json --write-file=/home/nvidia/out.raw

Hi @eladg1,

This is a known issue with the module. AR0231 doesn’t support LFM & HDR simultaneously in the same frame.
We have some changes to mitigate it in the upcoming release. Thanks!

Hi @eladg1,

I was talking about “AR0231 IQ changes” mentioned in DRIVE OS 5.2.0 Release Notes.
You can try on DRIVE OS 5.2.0 (available on NVIDIA DRIVE Downloads | NVIDIA Developer) and see if any improvement.
Thanks!