H264 encode images from GMSL cameras

Please provide the following info (check/uncheck the boxes after creating this topic):
Software Version
DRIVE OS Linux 5.2.6
DRIVE OS Linux 5.2.6 and DriveWorks 4.0
DRIVE OS Linux 5.2.0
DRIVE OS Linux 5.2.0 and DriveWorks 3.5
NVIDIA DRIVE™ Software 10.0 (Linux)
NVIDIA DRIVE™ Software 9.0 (Linux)
other DRIVE OS version
other

Target Operating System
Linux
QNX
other

Hardware Platform
NVIDIA DRIVE™ AGX Xavier DevKit (E3550)
NVIDIA DRIVE™ AGX Pegasus DevKit (E3550)
other

SDK Manager Version
1.7.1.8928
other

Host Machine Version
native Ubuntu 18.04
other


Hello,
I am currently trying to encode to h264 images coming from a gmsl camera. The cameras are Sekonix GMSL SF3324 and SF3325.
The cameras are running in processed mode, following this page. And they output image of type :
DW_IMAGE_FORMAT_YUV420_UINT8_PLANAR as DW_IMAGE_NVMEDIA type.

I am trying to use the NvMedia IEP API to encode the images. But it is said in the documentation of the function NvMediaIEPCreate that the format of the input images must be of type :

So it doesn’t match. Also when checking the Supported Cemera Output Types I see that DW_CAMERA_OUTPUT_CUDA_YUV420_UINT16_SEMIPLANAR is of type DW_IMAGE_CUDA and not DW_IMAGE_NVMEDIA, Which is very confusing as I believe I need to use nvmedia images with nvmedia IEP.

How can I solve this situation ?
Thanks a lot for your help !

Dear @maxandre.ogeret,
Are you looking to encode to H264 format using DW? Could you check https://docs.nvidia.com/drive/driveworks-3.5/dwx_camera_sample.html ?

I want to encode to h264 using driveworks or nvmedia. I want to get raw H264 bytes to send them to ROS.

I had a look at the sample you mentioned but it’s hard to make sense of it. What is actually used to make the conversion ?

I see that dwSensorSerializer and StreamerGL are used. The only mention of h264 is in the params of the serializer. Can I use the serializer in my case ?

What’s the easiest way for me to convert GMSL camera frames to H264 bytes ?

Thanks

Dear @maxandre.ogeret,
Can I use the serializer in my case ?

Yes. Please see format parameter in DriveWorks SDK Reference: Sensor Serializer . This sample can convert the camera stream to h264 file.

Thanks for your answer.
What’s the difference between using nvmedia_iep and the drivework serializer?

Hi, @maxandre.ogeret

DriveWorks Sensor Serializer API is implemented with DRIVE OS IEP. Thanks.

We did something similar to what you want to achieve here: GitHub - chinitaberrio/gmsl_driver_agx: Sekonix GMSL Camera Drive for NVIDIA Drive AGX Pegasus Developer Kit with DRIVE OS 5.2
In this ROS package, we have the option of publishing the images or storing them in an h264 file using a costume ROS message that keeps track of the frames in the rosbag.

Thanks both for your answers. I will try to implement h264 encoding with the nvmedia_iep library, as it would interface better with the current driver, which already uses the nvmedia_ijpe.

I have one more question :
The supported output types for my cameras are :

  • DW_IMAGE_FORMAT_YUV420_UINT8_PLANAR
  • DW_IMAGE_FORMAT_YUV420_UINT16_SEMIPLANAR
  • DW_IMAGE_FORMAT_YUV_UINT8_PLANAR
  • DW_IMAGE_FORMAT_YUV_UINT16_PLANAR

But the nvmedia_iep API needs :

  • (attr, YUV, 420, SEMI_PLANAR, UINT, [8/10], BL)
  • (attr, YUV, 444, SEMI_PLANAR, UINT, [8/10], BL)

So please answer this question:
How to get either (attr, YUV, 420, SEMI_PLANAR, UINT, [8/10], BL) or (attr, YUV, 444, SEMI_PLANAR, UINT, [8/10], BL) images with a gmsl camera ?

Thanks !

Please refer to Connect SIPL Camera to Video Encoder. Thanks.

Thanks, where can I find a sample that implements SIPL cameras ?
Thanks !

Below are the document and source path on your host system respectively.

  • SIPL Camera
  • ~/nvidia/nvidia_sdk/DRIVE_OS_5.2.0_*/DRIVEOS/drive-t186ref-linux/samples/nvmedia/nvsipl/test/camera/

Hello,
When initializing the sensor serializer, I get the following error :

[16-01-2022 15:50:30] Driveworks exception thrown: DW_INVALID_ARGUMENT: StreamSink type is not supported

[FATAL] [1642341030.469107119]: [2022-01-16 15:50:30] DW Error DW_INVALID_ARGUMENT executing DW function:
 dwSensorSerializer_initialize(serializer_, &serializerParams_, sensorHandle_)
 at /home/maxandre/ros_catkin_ws/src/sekonix_camera_ut/src/Camera.cpp:152

Any idea what could be the problem ? I cannot find anything online or in the documentation.
Thanks !

Dear @maxandre.ogeret,
Could you file a new topic with used DW code snippet. It looks like a parameter issue.

Dear @maxandre.ogeret,
The issue could be with type key(should be disk or user) in parameters field of dwSerializerParam. Please fix it and file a new topic if it does not help.