Camera Stream Encoding to H.264 on Drive AGX Orin

Please provide the following info (tick the boxes after creating this topic):
Software Version
DRIVE OS 6.0.8.1
DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-300)
DRIVE AGX Orin Developer Kit (940-63710-0010-200)
DRIVE AGX Orin Developer Kit (940-63710-0010-100)
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
1.9.3.10904
other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

Hello NVIDIA community,

I am currently facing while attempting to encode the camera stream to H.264 on the Drive AGX Orin platform.

I have successfully developed an application that reads raw images using the camera application on Drive AGX Orin, and the process of reading raw images is working as intended. However, when I try to take it a step further and encode the camera stream to H.264, I encounter a segmentation fault.

To initialize the camera stream encoding, I am using the following function:

CHECK_DW_ERROR(dwSensorSerializer_initialize(&data.serializer, &serializerParams, data.camera));

Here is the relevant configuration for the camera rig:

{
  "name": "CAM_WFV_RW_CE_SO",
  "nominalSensor2Rig_FLU": {
    "roll-pitch-yaw": [0.0, 0.0, 0.0],
    "t": [1.8621, -0.1939, 1.3165]
  },
  "parameter": "camera-name=V1SIM728S2RU4120HB20,interface=csi-gh,skip-eeprom=0,CPHY-mode=1,link=2,type=user,gop-length=1,format=h264,output-format=yuv,encoder-instance=0,quality=24,bitrate=50000000,fifo-size=4",
  "properties": {
    "Model": "ftheta",
    "bw-poly": "0.000000000000000 5.35356812179089e-4 4.99266072928606e-10 4.27370422037554e-12 -6.68245573791717e-16",
    "cx": "1927.764404",
    "cy": "1096.686646",
    "height": "2168",
    "width": "3848"
  },
  "protocol": "camera.gmsl"
}

The error occurs during the initialization process, and I receive the following error message:

[05-01-2024 00:19:36] **CameraClient::createSerializer: CameraClient is not bootstrapped**
[05-01-2024 00:19:36] CameraClient: camera params: camera-name=V1SIM728S2RU4120HB20,interface=csi-gh,skip-eeprom=0,CPHY-mode=1,link=2,type=user,gop-length=1,format=h264,output-format=yuv,encoder-instance=0,quality=24,bitrate=50000000,fifo-size=4
Segmentation fault

I would greatly appreciate any guidance or suggestions on resolving this issue.

Dear @abdul.rehman3,
Did you check using sample_camera? (DriveWorks SDK Reference: Camera Sample)

I have modified this application and added some other stuff. The application was working with Drive PX (Driveworks 4.0). I recompiled for Driveworks 5.10 and for Orin. The raw camera stream works.

Can you provide a sample rig file for encoding to h264 similar to the one that I have mentioned so I can verify parameters with sample_camera application?

I have tested a sample camera application
Rig file:

{
    "rig": {
        "sensors": [
            {
                "name": "CAM_WFV_RW_CE_SO",
                "nominalSensor2Rig_FLU": {
                    "roll-pitch-yaw": [
                        0.0,
                        0.0,
                        0.0
                    ],
                    "t": [
                        1.8621,
                        -0.1939,
                        1.3165
                    ]
                },
                "parameter": "camera-name=V1SIM728S2RU4120HB20,interface=csi-gh,skip-eeprom=0,CPHY-mode=1,link=2,type=user,gop-length=1,format=h264,output-format=h264,encoder-instance=0,quality=24,bitrate=50000000",
                "properties": {
                    "Model": "ftheta",
                    "bw-poly": "0.000000000000000 5.35356812179089e-4 4.99266072928606e-10 4.27370422037554e-12 -6.68245573791717e-16",
                    "cx": "1927.764404",
                    "cy": "1096.686646",
                    "height": "2168",
                    "width": "3848"
                },
                "protocol": "camera.gmsl"
            }
        ],
        "vehicle": {
            "valid": false
        }
    },
    "version": 2
}

Output:
[05-01-2024 01:15:49] rig::DatabaseOverlayer: Failed to open a file ‘/tmp/car_vin’ - file likely does not exist - skipping overlay
onInitialize: creating camera.gmsl with params: camera-name=V1SIM728S2RU4120HB20,interface=csi-gh,skip-eeprom=0,CPHY-mode=1,link=2,type=user,gop-length=1,format=h264,output-format=h264,encoder-instance=0,quality=24,bitrate=50000000,fifo-size=4
[05-01-2024 01:15:49] SensorFactory::createSensor() → camera.gmsl, camera-name=V1SIM728S2RU4120HB20,interface=csi-gh,skip-eeprom=0,CPHY-mode=1,link=2,type=user,gop-length=1,format=h264,output-format=h264,encoder-instance=0,quality=24,bitrate=50000000,fifo-size=4
[05-01-2024 01:15:49] CameraGMSLSelector: missing ‘output-format=processed/yuv’ in the sensor parameters, frames might be captured with bad sensor settings(AE/AWB) and reprocess may result image quality issues
[05-01-2024 01:15:49] CameraBase: pool size set to 8
[05-01-2024 01:15:49] SensorFactory::createSensor() → camera.gmsl.master,
[05-01-2024 01:15:50] CameraMaster::parseDevBlock Getting device info list.
[05-01-2024 01:15:50] devBlock: 0 Slave = 0 Interface = csi-gh Camera_name = V1SIM728S2RU4120HB20 Link = 2 GroupInit = 0 RecCfg = 1
[05-01-2024 01:15:50] Camera Match Name: V1SIM728S2RU4120HB20 Description: IMX728 RGGB module - 120FOV - MIPI-IMX728, MAX96717 linkIndex: 4294967295 serInfo.Name: MAX96717F
[05-01-2024 01:15:50] Client, Setting up information for camera ID 0
[05-01-2024 01:15:50] Client, successfully found info for camera ID 0 bound to id 0
[05-01-2024 01:15:50] CameraClient: using NITO found at /usr/share/camera/V1SIM728S2RU4120HB20.nito
[05-01-2024 01:15:50] SIPLMaster::SIPLMaster: Setting up master camera
[05-01-2024 01:15:50] Platform:
[05-01-2024 01:15:50] Platform Config:
[05-01-2024 01:15:50] Description:
[05-01-2024 01:15:50] Number of device blocks: 1
[05-01-2024 01:15:50] Device Block : 0
[05-01-2024 01:15:50] csiPort: 11
[05-01-2024 01:15:50] i2cDevice: 7
[05-01-2024 01:15:50] Deserializer Name: MAX96712_Fusa_nv
[05-01-2024 01:15:50] Deserializer Description: Maxim 96712 Aggregator Fusa_nv
[05-01-2024 01:15:50] Deserializer i2cAddress: 41
[05-01-2024 01:15:50] Simulator Mode: 0
[05-01-2024 01:15:50] Passive Mode: 0
[05-01-2024 01:15:50] Phy Mode: 1
[05-01-2024 01:15:50] Number of camera modules: 1
[05-01-2024 01:15:50] CameraModule index: 2
[05-01-2024 01:15:50] Name :V1SIM728S2RU4120HB20
[05-01-2024 01:15:50] Description: IMX728 RGGB module - 120FOV - MIPI-IMX728, MAX96717
[05-01-2024 01:15:50] Serializer name: MAX96717F
[05-01-2024 01:15:50] Serializer description: Maxim 96717F Serializer
[05-01-2024 01:15:50] Serializer i2cAdress: 64
[05-01-2024 01:15:50] EEPROM name: M24C04
[05-01-2024 01:15:50] EEPROM i2cAddress: 84
[05-01-2024 01:15:50] EEPROM description: M24C04 EEPROM
[05-01-2024 01:15:50] Sensor ID: 0
[05-01-2024 01:15:50] Sensor name: IMX728
[05-01-2024 01:15:50] Sensor description: Sony IMX728 Sensor
[05-01-2024 01:15:50] Sensor i2cAddress: 28
[05-01-2024 01:15:50] Sensor isTPGEnabled: 0
[05-01-2024 01:15:50] Sensor isTriggerMode: 1
[05-01-2024 01:15:50] cfa: 33
[05-01-2024 01:15:50] embeddedTopLines: 1
[05-01-2024 01:15:50] embeddedBottomLines: 20
[05-01-2024 01:15:50] inputFormat: 20
[05-01-2024 01:15:50] height: 2160
[05-01-2024 01:15:50] width: 3840
[05-01-2024 01:15:50] fps: 30.000000
[05-01-2024 01:15:50] Embedded Data: 0
[05-01-2024 01:15:50] CameraMaster available outputs: 0
[05-01-2024 01:15:50] CameraClient: setting pipeline config for camera ID 0
[05-01-2024 01:15:50] CameraClient: Failed to set Pipeline configuration for client with id 0, status 1
[05-01-2024 01:15:50] Driveworks exception thrown: DW_NVMEDIA_ERROR: CameraClient: Failed to set pipeline configuration!

terminate called after throwing an instance of ‘std::runtime_error’
what(): [2024-01-05 01:15:50] DW Error DW_NVMEDIA_ERROR executing DW function:
dwSAL_start(m_sal)
at /usr/local/driveworks-5.10/samples/src/sensors/camera/camera/main.cpp:363
Aborted

Dear @abdul.rehman3,
This sample actually identify the encoding type based on --write-file parameter. You can store --write-file=output.h264 to store in h264 format.

You can check /usr/local/driveworks/data/samples/sensors/camera/camera/rig.json as reference and update the camera-name, interface parameters for quick test. Let us know the update.

The rig file works with camera viewer application and also in this application i can see that the camera is detected.

[05-01-2024 01:23:48] SensorFactory::createSensor() → camera.gmsl.master,
[05-01-2024 01:23:48] CameraMaster::parseDevBlock Getting device info list.
[05-01-2024 01:23:48] devBlock: 0 Slave = 0 Interface = csi-gh Camera_name = V1SIM728S2RU4120HB20 Link = 2 GroupInit = 0 RecCfg = 1
[05-01-2024 01:23:48] Camera Match Name: V1SIM728S2RU4120HB20 Description: IMX728 RGGB module - 120FOV - MIPI-IMX728, MAX96717 linkIndex: 4294967295 serInfo.Name: MAX96717F
[05-01-2024 01:23:48] Client, Setting up information for camera ID 0
[05-01-2024 01:23:48] Client, successfully found info for camera ID 0 bound to id 0
[05-01-2024 01:23:48] CameraClient: using NITO found at /usr/share/camera/V1SIM728S2RU4120HB20.nito
[05-01-2024 01:23:48] SIPLMaster::SIPLMaster: Setting up master camera
[05-01-2024 01:23:48] Platform:
[05-01-2024 01:23:48] Platform Config:
[05-01-2024 01:23:48] Description:
[05-01-2024 01:23:48] Number of device blocks: 1
[05-01-2024 01:23:48] Device Block : 0
[05-01-2024 01:23:48] csiPort: 11
[05-01-2024 01:23:48] i2cDevice: 7
[05-01-2024 01:23:48] Deserializer Name: MAX96712_Fusa_nv
[05-01-2024 01:23:48] Deserializer Description: Maxim 96712 Aggregator Fusa_nv
[05-01-2024 01:23:48] Deserializer i2cAddress: 41
[05-01-2024 01:23:48] Simulator Mode: 0
[05-01-2024 01:23:48] Passive Mode: 0
[05-01-2024 01:23:48] Phy Mode: 1
[05-01-2024 01:23:48] Number of camera modules: 1
[05-01-2024 01:23:48] CameraModule index: 2
[05-01-2024 01:23:48] Name :V1SIM728S2RU4120HB20
[05-01-2024 01:23:48] Description: IMX728 RGGB module - 120FOV - MIPI-IMX728, MAX96717
[05-01-2024 01:23:48] Serializer name: MAX96717F
[05-01-2024 01:23:48] Serializer description: Maxim 96717F Serializer
[05-01-2024 01:23:48] Serializer i2cAdress: 64
[05-01-2024 01:23:48] EEPROM name: M24C04
[05-01-2024 01:23:48] EEPROM i2cAddress: 84
[05-01-2024 01:23:48] EEPROM description: M24C04 EEPROM
[05-01-2024 01:23:48] Sensor ID: 0
[05-01-2024 01:23:48] Sensor name: IMX728
[05-01-2024 01:23:48] Sensor description: Sony IMX728 Sensor
[05-01-2024 01:23:48] Sensor i2cAddress: 28
[05-01-2024 01:23:48] Sensor isTPGEnabled: 0
[05-01-2024 01:23:48] Sensor isTriggerMode: 1
[05-01-2024 01:23:48] cfa: 33
[05-01-2024 01:23:48] embeddedTopLines: 1
[05-01-2024 01:23:48] embeddedBottomLines: 20
[05-01-2024 01:23:48] inputFormat: 20
[05-01-2024 01:23:48] height: 2160
[05-01-2024 01:23:48] width: 3840
[05-01-2024 01:23:48] fps: 30.000000
[05-01-2024 01:23:48] Embedded Data: 0

Does that mean, after using the sample with modified json, you notice sample_camera is working and output.h264 is saved?

No, It is not working.
Error:

[05-01-2024 02:07:22] CameraMaster available outputs: 0
[05-01-2024 02:07:22] CameraClient: setting pipeline config for camera ID 0
[05-01-2024 02:07:22] CameraClient: Failed to set Pipeline configuration for client with id 0, status 1
[05-01-2024 02:07:22] Driveworks exception thrown: DW_NVMEDIA_ERROR: **CameraClient: Failed to set pipeline configuration!**

terminate called after throwing an instance of 'std::runtime_error'
  what():  [2024-01-05 02:07:22] DW Error DW_NVMEDIA_ERROR executing DW function:
 dwSAL_start(m_sal)
 at /usr/local/driveworks-5.10/samples/src/sensors/camera/camera/main.cpp:363
Aborted

Dear @abdul.rehman3,
Does that sample works when you just replace camera-name, interface in /usr/local/driveworks/data/samples/sensors/camera/camera/rig.json and run ./sample_camera --rig=/usr/local/driveworks/data/samples/sensors/camera/camera/rig.json . This confirm if the issue is limited only to encoding.

May I know your camera module details ? Is it listed in https://developer.nvidia.com/drive/ecosystem-orin ? Could you share the complete log of sample_camera?

I am using IMX728 camera.

The application works with the following rig file and create an h264 file.
{
“rig”: {
“sensors”: [
{
“name”: “camera:sample0”,
“nominalSensor2Rig_FLU”: {
“roll-pitch-yaw”: [
0.0,
0.0,
0.0
],
“t”: [
1.8621,
-0.1939,
1.3165
]
},
“parameter”: “camera-name=V1SIM728S2RU4120HB20,interface=csi-gh,skip-eeprom=0,CPHY-mode=1,link=2,gop-length=1,format=h264,output-format=yuv,encoder-instance=0,quality=24,bitrate=50000000”,
“properties”: {
“Model”: “ftheta”,
“bw-poly”: “0.000000000000000 5.35356812179089e-4 4.99266072928606e-10 4.27370422037554e-12 -6.68245573791717e-16”,
“cx”: “1927.764404”,
“cy”: “1096.686646”,
“height”: “2168”,
“width”: “3848”
},
“protocol”: “camera.gmsl”
}
],
“vehicle”: {
“valid”: false
}
},
“version”: 2
}

However, when I am using a parameter type=user then it stops working.

Can you please tell me what is the reason for this error
[06-01-2024 01:04:46] CameraClient::createSerializer: CameraClient is not bootstrapped

After so many attempts I have managed to find a partial solution
I was missing this
CHECK_DW_ERROR(dwSAL_start(m_sal));

I created all the sensor first and then acquire data. While doing so I am getting a following error for the second camera
[06-01-2024 09:58:00] CameraClient: camera params: camera-name=V1SIM728S2RU4120HB20,interface=csi-ab,skip-eeprom=0,CPHY-mode=1,link=1,type=user,gop-length=1,format=h264,output-format=yuv,encoder-instance=0,quality=24
[06-01-2024 09:58:00] CameraClient: serializer bitrate not specified. Using 8000000.
[06-01-2024 09:58:00] CameraMaster: getNvSciImageAttributes failed 6
[06-01-2024 09:58:00] Driveworks exception thrown: DW_INTERNAL_ERROR: CameraClient::getSiplImageAttributes ICP failed to allocate NvSciImageAttrList

terminate called after throwing an instance of ‘std::runtime_error’
what(): [2024-01-06 09:58:00] DW Error DW_INTERNAL_ERROR executing DW function:
dwSensorSerializer_initialize(&data.serializer, &serializerParams, data.camera)

Can you explain the problem.

I hope someone can reply to this and don’t leave it unanswered like many other questions

This indicates NVSIPL_STATUS_INVALID_STATE from NVIDIA DRIVE OS Linux SDK API Reference: SIPL | NVIDIA Docs

From Camera Stream Encoding to H.264 on Drive AGX Orin - #12 by abdul.rehman3, I see you could store h264 with rig file, but failed to do when using type=user. I am assuming this is when using sample_camera sample. The sample is using type=disk by default and uses dwSensorSerializerOnDataFunc_t as null. Do you have a non null functional pointer for dwSensorSerializerOnDataFunc_t?

If you are using another program(not sample_camera), may I know why sample_camera does not work for your case as I see you are a single camera in rig file and sample_camera is expected to work out of the box.

Hi Krishna,

I want to encode multiple cameras to h264. A sample camera is only encoding one camera stream that is FIRST_CAMERA_IDX. How can I encode multiple cameras connected to single csi port. In total I want to use 11 cameras.

I executed sample_camera in 2 different terminals

Terminal 1:
./bin/sample_camera --rig=rig-one.json --write-file=output1.h264 --offscreen=2

Rig file:

{
    "rig": {
        "sensors": [
            {
                "name": "CAM_WFV_RW_CE_SO",
                "nominalSensor2Rig_FLU": {
                    "roll-pitch-yaw": [
                        0.0,
                        0.0,
                        0.0
                    ],
                    "t": [
                        1.8621,
                        -0.1939,
                        1.3165
                    ]
                },
                "parameter": "camera-name=V1SIM728S2RU4120HB20,interface=csi-gh,skip-eeprom=0,CPHY-mode=1,link=2,gop-length=1,format=h264,output-format=yuv,encoder-instance=0,quality=24,type=disk",
                "properties": {
                    "Model": "ftheta",
                    "bw-poly": "0.000000000000000 5.35356812179089e-4 4.99266072928606e-10 4.27370422037554e-12 -6.68245573791717e-16",
                    "cx": "1927.764404",
                    "cy": "1096.686646",
                    "height": "2168",
                    "width": "3848"
                },
                "protocol": "camera.gmsl"
            }
        ],
        "vehicle": {
            "valid": false
        }
    },
    "version": 2
}

Terminal 2:
./bin/sample_camera --rig=rig-two.json --write-file=output2.h264 --offscreen=2
Rig file:

{
    "rig": {
        "sensors": [
            {
                "name": "CAM_WFV_FW_CE_NO",
                "nominalSensor2Rig_FLU": {
                    "roll-pitch-yaw": [
                        0.0,
                        0.0,
                        0.0
                    ],
                    "t": [
                        1.8621,
                        -0.1939,
                        1.3165
                    ]
                },
                "parameter": "camera-name=V1SIM728S2RU4120HB20,interface=csi-gh,skip-eeprom=0,CPHY-mode=1,link=1,type=disk,gop-length=1,format=h264,output-format=yuv,encoder-instance=0,quality=24,bitrate=800000",
                "properties": {
                    "Model": "ftheta",
                    "bw-poly": "0.000000000000000 5.35356812179089e-4 4.99266072928606e-10 4.27370422037554e-12 -6.68245573791717e-16",
                    "cx": "1927.764404",
                    "cy": "1096.686646",
                    "height": "2168",
                    "width": "3848"
                },
                "protocol": "camera.gmsl"
            }
        ],
        "vehicle": {
            "valid": false
        }
    },
    "version": 2
}

On the second terminal, I am getting a following error:
[07-01-2024 07:31:10] CameraMaster available outputs: 1
[07-01-2024 07:31:10] CameraClient: setting pipeline config for camera ID 0
[07-01-2024 07:31:10] CameraMaster: master initiation
[07-01-2024 07:31:10] CameraMaster: master init failed with status = 10
[07-01-2024 07:31:10] Driveworks exception thrown: DW_UNEXPECTED_EVENT: CameraMaster::bootstrap failed with sipl status 10

However when I am use the camera from other csi port it works fine. How can I connect 4 cameras with one csi-port and encode into h264.

Dear @abdul.rehman3,

Got it.

When one camera from a port is active in a process, another process can not access its sibling.

We already have recorder tool to record data from multiple sensors at a time. Did you try our recording tool app? If not, could you please check DriveWorks SDK Reference: Basic Recording Tool
DriveWorks SDK Reference: Configuration Reference
DriveWorks SDK Reference: Rig File Format

Hi Krishna,

How Can I access source code for the recorder application. I will modify my application according to the recorder application.

The currently developed application was working on drive px and was developed on driveworks 4.0. I have found that some new stuff is added in the new driveworks. For e.g. sal (sensor abstraction layer) need to be started after initialization and then serializer is initialized. Are there any other changes?

Dear @abdul.rehman3,
Recorder tool source code is not available in DW SDK release.
The chages across different DW version is captured at porting guide section in DW documentation(DriveWorks SDK Reference: Porting Guide). Please see if it helps your understanding.

I am wondering why you want develop similar application like recorder. Why can’t you use our recorder tool? Do you see any gap/issue in it? Let us know if anything needs to be improved.

Recorder application is used for recording. My application is also performing other functions including streaming h264 on the ethernet ports which will be further used in other functions. Maybe we can solve the error.

The error I am getting is following when I am using multiple cameras
[07-01-2024 09:22:05] CameraMaster: getNvSciImageAttributes failed 6
[07-01-2024 09:22:05] Driveworks exception thrown: DW_INTERNAL_ERROR: CameraClient::getSiplImageAttributes ICP failed to allocate NvSciImageAttrList