Leopard imaging IMX377 capture and display

Dear all,

For the past couple of days I have been ploughing through the topics to find a way to capture and show images using the Leopard Imaging IMX377 board. So far I think I have applied most driver patches and pipeline examples, without any luck.

Situation:

  • TX2 with Jetpack 28.2 installed
  • Leopard Imaging IMX377 (three camera board)

I have followed the Leopard dropbox instructions both for rebuilding the kernel (resulted in a hanging TX2 upon startup) and copy and pasting the Image and zImage files (resulted in a TX2 that can start up, but it will lose the WiFi connection, i.e. not present in the devices anymore).
I have tried starting pipelines with v4l2, gstreamer and argus, but without being able to display them. The output from:

v4l2-ctl --list-formats-ext -d /dev/video0

Does give me the overview of the video ports connected:

ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'RGGB'
	Name        : 8-bit Bayer RGRG/GBGB
		Size: Discrete 3840x2160
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x540
			Interval: Discrete 0.004s (240.000 fps)

	Index       : 1
	Type        : Video Capture
	Pixel Format: 'RG10'
	Name        : 10-bit Bayer RGRG/GBGB
		Size: Discrete 3840x2160
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x540
			Interval: Discrete 0.004s (240.000 fps)

	Index       : 2
	Type        : Video Capture
	Pixel Format: 'BG10'
	Name        : 10-bit Bayer BGBG/GRGR
		Size: Discrete 3840x2160
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x540
			Interval: Discrete 0.004s (240.000 fps)

	Index       : 3
	Type        : Video Capture
	Pixel Format: 'RG12'
	Name        : 12-bit Bayer RGRG/GBGB
		Size: Discrete 3840x2160
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x540
			Interval: Discrete 0.004s (240.000 fps)

Could someone please be of any assistence on how to set up a pipeline (with either v4l2, gstreamer or argus) to capture and show a video stream, together with control of the fps, exposure, etc.?

P.S. I have been able to stream a video stream on another TX2 with Jetpack 28.1, but without the ability to set camera settings such as fps, resulting in a huge camera lag of 2 seconds (presumably due to buffering). This was done using a gstreamer pipeline, however the same pipeline does not work on the 28.2 TX2.

Thanks in advance!

Hi t.frenkel,

I believe you are using below driver guide. This guide only support R28.1. Our team are working on the driver of R28.2. Will keep you posted.

https://www.dropbox.com/s/8bhh0w7axatz8jn/IMX377_R28.1_TX2_NV_20180403.txt?dl=0

You can use below command to get video output with gstreamer. You can also use argus software to view the video.

gst-launch-1.0 nvcamerasrc fpsRange=“30.0 30.0” sensor-id=0 ! ‘video/x-raw(memory:NVMM), width=(int)4104, height=(int)3046, format=(string)I420, framerate=(fraction)30/1’ ! nvtee ! nvvidconv flip-method=2 ! ‘video/x-raw, format=(string)I420’ ! xvimagesink -e

Dear SimonZhu,

Indeed this was the issue. I reverted the TX2 to Jetpack 28.1 and got it working fine now, thank you.

Kind regards

Dear SimonZhhu,

As a follow-up, I am now perfectly able to open the camera using the argus_camera command which will load the GUI and in that GUI I can select all three camera’s. The Argus samples in the samples folder also work, however, if I try to compile my own code only executing the following command:

// 1) Create CameraProvider to establish libargus driver connection.
	UniqueObj<CameraProvider> cameraProvider(CameraProvider::create());
	ICameraProvider* iCameraProvider = interface_cast<ICameraProvider>(cameraProvider);
	if (!iCameraProvider)
	{
		ROS_INFO("Failed to establish libargus connection");
		return;
	}

I get the following error message:

NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
PCLHW_DTParser
LoadOverridesFile: looking for override file [/Calib/camera_override.isp] 1/16LoadOverridesFile: looking for override file [/data/nvcam/settings/camera_overrides.isp] 2/16LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/camera_overrides.isp] 3/16LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/camera_overrides.isp] 4/16---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 594: Error: Invalid isp config attribute: "//Thisistheextradelayintroducedwhileprogrammingthegroupholdsettingapartfromdefault//totaldelayof3framesi.e.2framesingroupholdsetttingstoeffectand1frametocapture.//Settingdefaultvalueaszeroasitisdisabledbydefault.ae.MaxGroupHoldDelayFrameCount=0"
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 3177: Error: Invalid isp config attribute element: "fd.Available=TRUE"
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 3178: Error: Invalid isp config attribute element: "fd.ForceEnable=FALSE"
NvCameraIspConfigFileLoad: Config file "isp4.cfg" Line 333: Error: Invalid isp config attribute: "ds.enable=TRUE"
LoadOverridesFile: looking for override file [/Calib/camera_override.isp] 1/16LoadOverridesFile: looking for override file [/data/nvcam/settings/camera_overrides.isp] 2/16LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/camera_overrides.isp] 3/16LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/camera_overrides.isp] 4/16---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 594: Error: Invalid isp config attribute: "//Thisistheextradelayintroducedwhileprogrammingthegroupholdsettingapartfromdefault//totaldelayof3framesi.e.2framesingroupholdsetttingstoeffectand1frametocapture.//Settingdefaultvalueaszeroasitisdisabledbydefault.ae.MaxGroupHoldDelayFrameCount=0"
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 3177: Error: Invalid isp config attribute element: "fd.Available=TRUE"
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 3178: Error: Invalid isp config attribute element: "fd.ForceEnable=FALSE"
NvCameraIspConfigFileLoad: Config file "isp4.cfg" Line 333: Error: Invalid isp config attribute: "ds.enable=TRUE"
LoadOverridesFile: looking for override file [/Calib/camera_override.isp] 1/16LoadOverridesFile: looking for override file [/data/nvcam/settings/camera_overrides.isp] 2/16LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/camera_overrides.isp] 3/16LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/camera_overrides.isp] 4/16---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 594: Error: Invalid isp config attribute: "//Thisistheextradelayintroducedwhileprogrammingthegroupholdsettingapartfromdefault//totaldelayof3framesi.e.2framesingroupholdsetttingstoeffectand1frametocapture.//Settingdefaultvalueaszeroasitisdisabledbydefault.ae.MaxGroupHoldDelayFrameCount=0"
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 3177: Error: Invalid isp config attribute element: "fd.Available=TRUE"
NvCameraIspConfigFileLoad: Config file "common.cfg" Line 3178: Error: Invalid isp config attribute element: "fd.ForceEnable=FALSE"
NvCameraIspConfigFileLoad: Config file "isp4.cfg" Line 333: Error: Invalid isp config attribute: "ds.enable=TRUE"

Looking in the var/nvidia/nvcam/settings folder I can see the camera_overrides.isp which matches the one provided in the Driver.zip from the Leopard Imaging dropbox install https://www.dropbox.com/sh/j7kx7r7e6svp5b3/AAD3M7dtC_3mC4MhEIX3nBEfa?dl=0&preview=IMX377_R28.1_TX2_NV_20180403.txt

Could you please provide assistance regarding this issue?

Hi t.frenkel,

For “try to compile my own code”, do you mean you use the two patch files (which we provided) to re-compile the code (below steps)?

[i]5. Compile the patch

If you would like to compile the patch file to get the driver. Please refer to below steps.
Download kernel code and Tool Chain from links below.

Kernel code: https://www.dropbox.com/s/awzxe90dhipfon2/kernel_src-tx2.tbz2?dl=0
GCC ToolChain: https://www.dropbox.com/sh/zf0eo7s2h5jts0k/AABzU8Ku5BOOxbHUFxFUfnqHa?dl=0

Compile the kernel under 64 bit Ubuntu OS on Intel x64 PC. (Virtual machine is fine. We are using Ubuntu 16.04 64 bit OS)

  1. Copy compile tool gcc-4.8.5-aarch64.tgz to /opt, and unzip it
    sudo tar zxvf gcc-4.8.5-aarch64.tgz

  2. Copy kernel_src.tbz2 and two patch files to /usr/src
    sudo tar xpf kernel_src-tx2.tbz2
    sudo chown -R <user_name> kernel
    sudo chown -R <user_name> hardware
    patch -p0 < tri_streaming_imx377_base28.1_TX2_NV_dtbs_20180403.patch
    patch -p0 < tri_streaming_imx377_base28.1_TX2_NV_kernel_20180403.patch
    Note: <user_name> is the user name of your Ubuntu OS. For example: sudo chown -R leopard kernel

  3. Copy tx2.sh to /usr/src/kernel.
    under /usr/src/kernel, do
    source tx2.sh

  4. Create a work folder under /home:
    sudo mkdir /home/work
    sudo chown -R <user_name> /home/work

  5. In “kernel/kernel-4.4” folder, run:

    make O=$TEGRA_KERNEL_OUT tegra18_defconfig
    make O=$TEGRA_KERNEL_OUT zImage
    make O=$TEGRA_KERNEL_OUT dtbs

You will get Image and zImage under /home/work/TX2/kernel/kernel_out/arch/arm64/boot and tegra186-quill-p3310-1000-c03-00-base.dtb under /home/work/TX2/kernel/kernel_out/arch/arm64/boot/dts.[/i]

Dear SimonZhu,

With that statement I meant writing my own application (outside the tegra_multimedia_api folder) that uses the Argus library for streaming video from the three camera’s as I want to process them using openCV. I have tried to built the patch files on the TX2, but did not succeed (this was on the Jetpack 3.2 TX2 and I have now reverted to 3.1), is this required? I have used the Image and zImage files provided in the Driver.zip to flash the TX2, exactly according to the steps before step 5. If it is required could you provide example code showing how to combine the compiled patches in combination with the Argus library?

Kind regards

Hi t.frenkel,

Please use the argus library from Jetpack 3.1 (R28.1).