Sending 12-bit images from Libargus to ROS

Autonomous Machines Jetson & Embedded Systems Jetson Xavier NX
1

I’m trying to capture 12-bit images using Libargus and publish them to ROS. In an attempt to not lose any data, I need the images to be published as BGR16 encoding.

I’ve been following a method mentioned across the forum which is for publishing 8-bit images to ROS:

// Set the pixel format to PIXEL_FMT_YCbCr_420_888
iEGLStreamSettings->setPixelFormat(Argus::PIXEL_FMT_YCbCr_420_888);
...

// Create a buffer with NvBufferColorFormat_ABGR32
m_dmabuf = iNativeBuffer->createNvBuffer(iEglOutputStream->getResolution(), NvBufferColorFormat_ABGR32, NvBufferLayout_Pitch);
...

// Convert to cv::Mat and use cvtColor
void *pdata = NULL;
NvBufferMemMap(m_dmabuf, 0, NvBufferMem_Read, &pdata);
NvBufferMemSyncForCpu(m_dmabuf, 0, &pdata);
cv::Mat imgbuf = cv::Mat(1106, 1936, CV_8UC4, pdata);
cv::Mat display_img;
cvtColor(imgbuf, display_img, CV_RGBA2BGR);
...

// Use cv_bridge to convert the image to a ROS msg
NvBufferMemUnMap(m_dmabuf, 0, &pdata);
cv_bridge::CvImage out_msg;
out_msg.encoding = sensor_msgs::image_encodings::BGR8;
out_msg.image = display_img;
image_pub.publish(out_msg.toImageMsg());

Not surprisingly, this solution doesn’t work for 12-bit images. Each images arrives distorted (each row of pixels is further offset from the last).

Updating CV_8UC4 → CV_16UC4, and BGR8 → BGR16 does not work. The program crashes at the cvtColor function, without returning an error…

Any help is appreciated.

3

hello Hommus,

may I know what’s your sensor device tree implementation,
did you configure csi_pixel_bit_depth=12 for sending 12-bit pixel formats?

4

Hi @JerryChang,
csi_pixel_bit_depth has been set to 12. Here is a portion of the device tree, let me know if you want more or all of it. It’s worth noting that I have successfully received images using these drivers before, even through Argus. I’ve just been struggling with the conversion from Argus → OpenCV → ROS.

	i2c@3180000  {
		tca9548@70 {
		i2c@0 {
			imx265_cam0: rbpcv2_imx265_a@10 {
				compatible = "nvidia,imx265";
				reg = <0x10>;

				devnode = "video0";

				physical_w = "3.680";
				physical_h = "2.760";

				sensor_model = "imx265";

				use_decibel_gain = "true";
				use_sensor_mode_id = "true";

				avdd-reg = "vana";
				iovdd-reg = "vif";
				dvdd-reg = "vdig";

				mode0 {
					mclk_khz = "37125";
					num_lanes = "2";
					tegra_sinterface = "serial_a";
					phy_mode = "DPHY";
					discontinuous_clk = "no";
					dpcm_enable = "false";
					cil_settletime = "0";

					dynamic_pixel_bit_depth = "12";
					csi_pixel_bit_depth = "12";
					mode_type = "bayer";
					pixel_phase = "rggb";

					active_w = "2048"; 
					active_h = "1554"; 
					readout_orientation = "0";
					line_length = "2256";
					inherent_gain = "1";
					mclk_multiplier = "20";
					pix_clk_hz = "198000000";		// 198 MHz
					
					gain_factor = "10";
					min_gain_val = "1"; 			// 0.1dB
					max_gain_val = "480"; 			// 48dB
					step_gain_val = "1";			// 0.1dB
					default_gain = "1";

					min_hdr_ratio = "1";
					max_hdr_ratio = "1";

					framerate_factor = "1000000";
					min_framerate = "1000000"; 		// 1 FPS
					max_framerate = "60000000"; 	// 60 FPS
					step_framerate = "1";
					default_framerate = "27000000"; // 27 FPS

					exposure_factor = "1000000";
					min_exp_time = "30"; 			// us
					max_exp_time = "660000"; 		// us
					step_exp_time = "1";
					default_exp_time = "33334"; 	// us

					embedded_metadata_height = "0";
				};
				mode1 {
					mclk_khz = "37125";
					num_lanes = "2";
					tegra_sinterface = "serial_a";
					phy_mode = "DPHY";
					discontinuous_clk = "no";
					dpcm_enable = "false";
					cil_settletime = "0";

					dynamic_pixel_bit_depth = "12";
					csi_pixel_bit_depth = "12";
					mode_type = "bayer";
					pixel_phase = "rggb";

					active_w = "1936";
					active_h = "1106";
					readout_orientation = "0";
					line_length = "2200";
					inherent_gain = "1";
					mclk_multiplier = "20";
					pix_clk_hz = "148500000";
					
					gain_factor = "10";
					min_gain_val = "1";
					max_gain_val = "480";
					step_gain_val = "1";
					default_gain = "1";

					min_hdr_ratio = "1";
					max_hdr_ratio = "1";

					framerate_factor = "1000000";
					min_framerate = "1000000";
					max_framerate = "60000000";
					step_framerate = "1";
					default_framerate = "30000000";

					exposure_factor = "1000000";
					min_exp_time = "30";
					max_exp_time = "660000";
					step_exp_time = "1";
					default_exp_time = "33334";

					embedded_metadata_height = "0";
				};
				
				mode2 {
					mclk_khz = "37125";
					num_lanes = "2";
					tegra_sinterface = "serial_a";
					phy_mode = "DPHY";
					discontinuous_clk = "no";
					dpcm_enable = "false";
					cil_settletime = "0";

					dynamic_pixel_bit_depth = "12";
					csi_pixel_bit_depth = "12";
					mode_type = "bayer";
					pixel_phase = "rggb";

					active_w = "1032";
					active_h = "772";
					readout_orientation = "0";
					line_length = "2112";
					inherent_gain = "1";
					mclk_multiplier = "20";
					pix_clk_hz = "198000000";
					
					gain_factor = "10";
					min_gain_val = "1";
					max_gain_val = "480";
					step_gain_val = "1";
					default_gain = "1";

					min_hdr_ratio = "1";
					max_hdr_ratio = "1";

					framerate_factor = "1000000";
					min_framerate = "1000000";
					max_framerate = "60000000";
					step_framerate = "1";
					default_framerate = "58000000";

					exposure_factor = "1000000";
					min_exp_time = "30";
					max_exp_time = "660000";
					step_exp_time = "1";
					default_exp_time = "33334";

					embedded_metadata_height = "0";
				};
				ports {
					#address-cells = <1>;
					#size-cells = <0>;
					port@0 {
						reg = <0>;
						rbpcv2_imx265_out0: endpoint {
							status = "okay";
							port-index = <0>;
							bus-width = <2>;
							remote-endpoint = <&rbpcv2_imx265_csi_in0>;
						};
					};
				};
			};
		};
5

please see-also NVBuffer (FD) to opencv Mat - #6 by DaneLLL to create NvBuffer as ABGR32, thanks

6

Hi @JerryChang,

The code you mentioned only works when converting to 8-bit, but does not work for 12 to 16-bit conversion.

Is ABGR32 allocating 8 bits per pixel? (instead of the 12bpp that I want)

8

Hi,
ABGR32 is 8-bit R 8-bit G 8-bit B 8-bit A for each pixel. Hardware blocks in Xavier do not support12-bit formats, so libargus cannot output 12-bit images. This use-case of sending 12-bit images from libargus to ROS is not supported due to hardware constraint.

9

@DaneLLL So if we want the 12-bit images from our camera, we can’t use Libargus?

Isn’t Libargus outputting 12-bit images if we use NV12/YUV420?

10

Hi,
The output format of ISP engine is 8-bit YUV420. 12-bit format is not supported.

11

Thanks for your help @DaneLLL,
That is unfortunate. Can you explain what is happening during the 12-bit to 8-bit conversion on the ISP?

As a side question, what image processing is the ISP doing? (debayering, auto-exposure, etc)

13

Hi,
The ISP engine does lens shading correction, white balance, demosaic, etc. After the filters, the output format is 8-bit YUV420.

15

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.