Leopard imaging IMX377 capture and display

t.frenkel · April 24, 2018, 2:16pm

Dear all,

For the past couple of days I have been ploughing through the topics to find a way to capture and show images using the Leopard Imaging IMX377 board. So far I think I have applied most driver patches and pipeline examples, without any luck.

Situation:

TX2 with Jetpack 28.2 installed
Leopard Imaging IMX377 (three camera board)

I have followed the Leopard dropbox instructions both for rebuilding the kernel (resulted in a hanging TX2 upon startup) and copy and pasting the Image and zImage files (resulted in a TX2 that can start up, but it will lose the WiFi connection, i.e. not present in the devices anymore).
I have tried starting pipelines with v4l2, gstreamer and argus, but without being able to display them. The output from:

v4l2-ctl --list-formats-ext -d /dev/video0

Does give me the overview of the video ports connected:

ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'RGGB'
	Name        : 8-bit Bayer RGRG/GBGB
		Size: Discrete 3840x2160
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x540
			Interval: Discrete 0.004s (240.000 fps)

	Index       : 1
	Type        : Video Capture
	Pixel Format: 'RG10'
	Name        : 10-bit Bayer RGRG/GBGB
		Size: Discrete 3840x2160
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x540
			Interval: Discrete 0.004s (240.000 fps)

	Index       : 2
	Type        : Video Capture
	Pixel Format: 'BG10'
	Name        : 10-bit Bayer BGBG/GRGR
		Size: Discrete 3840x2160
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x540
			Interval: Discrete 0.004s (240.000 fps)

	Index       : 3
	Type        : Video Capture
	Pixel Format: 'RG12'
	Name        : 12-bit Bayer RGRG/GBGB
		Size: Discrete 3840x2160
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x540
			Interval: Discrete 0.004s (240.000 fps)

Could someone please be of any assistence on how to set up a pipeline (with either v4l2, gstreamer or argus) to capture and show a video stream, together with control of the fps, exposure, etc.?

P.S. I have been able to stream a video stream on another TX2 with Jetpack 28.1, but without the ability to set camera settings such as fps, resulting in a huge camera lag of 2 seconds (presumably due to buffering). This was done using a gstreamer pipeline, however the same pipeline does not work on the 28.2 TX2.

Thanks in advance!

SimonZhu · April 25, 2018, 4:20pm

Hi t.frenkel,

I believe you are using below driver guide. This guide only support R28.1. Our team are working on the driver of R28.2. Will keep you posted.

You can use below command to get video output with gstreamer. You can also use argus software to view the video.

gst-launch-1.0 nvcamerasrc fpsRange=“30.0 30.0” sensor-id=0 ! ‘video/x-raw(memory:NVMM), width=(int)4104, height=(int)3046, format=(string)I420, framerate=(fraction)30/1’ ! nvtee ! nvvidconv flip-method=2 ! ‘video/x-raw, format=(string)I420’ ! xvimagesink -e

t.frenkel · April 26, 2018, 1:39pm

Dear SimonZhu,

Indeed this was the issue. I reverted the TX2 to Jetpack 28.1 and got it working fine now, thank you.

Kind regards

t.frenkel · May 1, 2018, 8:09am

Dear SimonZhhu,

As a follow-up, I am now perfectly able to open the camera using the argus_camera command which will load the GUI and in that GUI I can select all three camera’s. The Argus samples in the samples folder also work, however, if I try to compile my own code only executing the following command:

// 1) Create CameraProvider to establish libargus driver connection.
	UniqueObj<CameraProvider> cameraProvider(CameraProvider::create());
	ICameraProvider* iCameraProvider = interface_cast<ICameraProvider>(cameraProvider);
	if (!iCameraProvider)
	{
		ROS_INFO("Failed to establish libargus connection");
		return;
	}

I get the following error message:

NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
PCLHW_DTParser
LoadOverridesFile: looking for override file [/Calib/camera_override.isp] 1/16LoadOverridesFile: looking for override file [/data/nvcam/settings/camera_overrides.isp] 2/16LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/camera_overrides.isp] 3/16LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/camera_overrides.isp] 4/16---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 594: Error: Invalid isp config attribute: "//Thisistheextradelayintroducedwhileprogrammingthegroupholdsettingapartfromdefault//totaldelayof3framesi.e.2framesingroupholdsetttingstoeffectand1frametocapture.//Settingdefaultvalueaszeroasitisdisabledbydefault.ae.MaxGroupHoldDelayFrameCount=0"
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 3177: Error: Invalid isp config attribute element: "fd.Available=TRUE"
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 3178: Error: Invalid isp config attribute element: "fd.ForceEnable=FALSE"
NvCameraIspConfigFileLoad: Config file "isp4.cfg" Line 333: Error: Invalid isp config attribute: "ds.enable=TRUE"
LoadOverridesFile: looking for override file [/Calib/camera_override.isp] 1/16LoadOverridesFile: looking for override file [/data/nvcam/settings/camera_overrides.isp] 2/16LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/camera_overrides.isp] 3/16LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/camera_overrides.isp] 4/16---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 594: Error: Invalid isp config attribute: "//Thisistheextradelayintroducedwhileprogrammingthegroupholdsettingapartfromdefault//totaldelayof3framesi.e.2framesingroupholdsetttingstoeffectand1frametocapture.//Settingdefaultvalueaszeroasitisdisabledbydefault.ae.MaxGroupHoldDelayFrameCount=0"
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 3177: Error: Invalid isp config attribute element: "fd.Available=TRUE"
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 3178: Error: Invalid isp config attribute element: "fd.ForceEnable=FALSE"
NvCameraIspConfigFileLoad: Config file "isp4.cfg" Line 333: Error: Invalid isp config attribute: "ds.enable=TRUE"
LoadOverridesFile: looking for override file [/Calib/camera_override.isp] 1/16LoadOverridesFile: looking for override file [/data/nvcam/settings/camera_overrides.isp] 2/16LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/camera_overrides.isp] 3/16LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/camera_overrides.isp] 4/16---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 594: Error: Invalid isp config attribute: "//Thisistheextradelayintroducedwhileprogrammingthegroupholdsettingapartfromdefault//totaldelayof3framesi.e.2framesingroupholdsetttingstoeffectand1frametocapture.//Settingdefaultvalueaszeroasitisdisabledbydefault.ae.MaxGroupHoldDelayFrameCount=0"
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 3177: Error: Invalid isp config attribute element: "fd.Available=TRUE"
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 3178: Error: Invalid isp config attribute element: "fd.ForceEnable=FALSE"
NvCameraIspConfigFileLoad: Config file "isp4.cfg" Line 333: Error: Invalid isp config attribute: "ds.enable=TRUE"

Looking in the var/nvidia/nvcam/settings folder I can see the camera_overrides.isp which matches the one provided in the Driver.zip from the Leopard Imaging dropbox install https://www.dropbox.com/sh/j7kx7r7e6svp5b3/AAD3M7dtC_3mC4MhEIX3nBEfa?dl=0&preview=IMX377_R28.1_TX2_NV_20180403.txt

Could you please provide assistance regarding this issue?

SimonZhu · May 1, 2018, 4:18pm

Hi t.frenkel,

For “try to compile my own code”, do you mean you use the two patch files (which we provided) to re-compile the code (below steps)?

[i]5. Compile the patch

If you would like to compile the patch file to get the driver. Please refer to below steps.
Download kernel code and Tool Chain from links below.

Kernel code: Dropbox - File Deleted
GCC ToolChain: Dropbox - File Deleted

Compile the kernel under 64 bit Ubuntu OS on Intel x64 PC. (Virtual machine is fine. We are using Ubuntu 16.04 64 bit OS)

Copy compile tool gcc-4.8.5-aarch64.tgz to /opt, and unzip it
sudo tar zxvf gcc-4.8.5-aarch64.tgz
Copy kernel_src.tbz2 and two patch files to /usr/src
sudo tar xpf kernel_src-tx2.tbz2
sudo chown -R <user_name> kernel
sudo chown -R <user_name> hardware
patch -p0 < tri_streaming_imx377_base28.1_TX2_NV_dtbs_20180403.patch
patch -p0 < tri_streaming_imx377_base28.1_TX2_NV_kernel_20180403.patch
Note: <user_name> is the user name of your Ubuntu OS. For example: sudo chown -R leopard kernel
Copy tx2.sh to /usr/src/kernel.
under /usr/src/kernel, do
source tx2.sh
Create a work folder under /home:
sudo mkdir /home/work
sudo chown -R <user_name> /home/work
In “kernel/kernel-4.4” folder, run:

make O=$TEGRA_KERNEL_OUT tegra18_defconfig
make O=$TEGRA_KERNEL_OUT zImage
make O=$TEGRA_KERNEL_OUT dtbs

You will get Image and zImage under /home/work/TX2/kernel/kernel_out/arch/arm64/boot and tegra186-quill-p3310-1000-c03-00-base.dtb under /home/work/TX2/kernel/kernel_out/arch/arm64/boot/dts.[/i]

t.frenkel · May 2, 2018, 9:53am

Dear SimonZhu,

With that statement I meant writing my own application (outside the tegra_multimedia_api folder) that uses the Argus library for streaming video from the three camera’s as I want to process them using openCV. I have tried to built the patch files on the TX2, but did not succeed (this was on the Jetpack 3.2 TX2 and I have now reverted to 3.1), is this required? I have used the Image and zImage files provided in the Driver.zip to flash the TX2, exactly according to the steps before step 5. If it is required could you provide example code showing how to combine the compiled patches in combination with the Argus library?

Kind regards

SimonZhu · May 2, 2018, 4:39pm

Hi t.frenkel,

Please use the argus library from Jetpack 3.1 (R28.1).

Topic		Replies	Views
How to use LI-JETSON-KIT-IMX377M12-X with TX2 Jetson TX2 camera	113	10573	October 18, 2021
Leopard Imaging 6 camera carrier board not working Jetson TX2	4	1768	October 18, 2021
Problem capturing video with IMX377 and argus Jetson TX1	11	1637	July 18, 2018
Green Screen Live Output from Leopard IMX477 Camera + TX2 Developer Kit Jetson TX2	5	2503	May 2, 2018
Leopard LI-TX-CB1 (Rev 1.3) with IMX185 green output Jetson TX2	9	1066	October 18, 2021
Washed out / purple image from IMX377 on TX1 Jetson TX1	3	950	December 24, 2018
Multiple CSI cameras on Tx2 Jetson TX2	18	3288	October 18, 2021
Leopard Imaging Inc. released new Jetson cameras for NVIDIA Jetson TX1/TX2 Developer Kit Announcements	30	6469	August 3, 2020
IMX274 camera on TX1 and L4T R28.1 (Jetpack 3.1) Jetson TX1	3	1360	October 18, 2021
Argus with IMX377 and USB Flir Boson Jetson TX2	18	2920	January 29, 2019

Leopard imaging IMX377 capture and display

Related topics