nvcamera-daemon crashing and not detecting cameras

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

Hi, I am writing a driver for an IMX290 camera module. I can initialise a camera which outputs frames over CSI-2 when I scope it.

I’ve also written a kernel driver which initialises the camera no problem. But now I’m completely stuck.

The nvcamera-daemon crashes when I run gst-launch-1.0 nvcamerasrc, so I am unable to create a /dev/video0. The offending line seems to be “(NvOdmDevice) Error ModuleNotPresent: V4L2Device not available (in dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function findDevice(), line 231)”.

This error doesn’t really help - how should I go about making the daemon recognise my cameras, so it creates the /dev/video device files, and I can start to fit my driver into the v4l2 pipeline?

I have also tried putting my driver code inside the existing IMX274 driver and using the same name, but it gives similar errors.

Debug output from nvcamera-daemon:

NvPclHwInitializeModule: allocate overrides pathname @ 0x7facbdff10
NvPclHwInitializeModule: allocate overrides pathname @ 0x7facbdffa0
NvPclHwInitializeModule: allocate overrides pathname @ 0x7facbc34b0
NvPclHwInitializeModule: allocate overrides pathname @ 0x7facbc3540
NvPclHwInitializeModule: allocate overrides pathname @ 0x7facbc35d0
LoadOverridesFile: looking for override file [/Calib/camera_override.isp] 1/16CheckOverridesPermissions: cannot stat file [/Calib/camera_override.isp]
LoadOverridesFile: looking for override file [/data/nvcam/settings/camera_overrides.isp] 2/16CheckOverridesPermissions: cannot stat file [/data/nvcam/settings/camera_overrides.isp]
LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/camera_overrides.isp] 3/16CheckOverridesPermissions: cannot stat file [/opt/nvidia/nvcam/settings/camera_overrides.isp]
LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/camera_overrides.isp] 4/16CheckOverridesPermissions: cannot stat file [/var/nvidia/nvcam/settings/camera_overrides.isp]
LoadOverridesFile: looking for override file [/data/nvcam/camera_overrides.isp] 5/16CheckOverridesPermissions: cannot stat file [/data/nvcam/camera_overrides.isp]
LoadOverridesFile: looking for override file [/data/nvcam/settings/imx290_bottom_A6V26.isp] 6/16CheckOverridesPermissions: cannot stat file [/data/nvcam/settings/imx290_bottom_A6V26.isp]
LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/imx290_bottom_A6V26.isp] 7/16CheckOverridesPermissions: cannot stat file [/opt/nvidia/nvcam/settings/imx290_bottom_A6V26.isp]
LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/imx290_bottom_A6V26.isp] 8/16CheckOverridesPermissions: cannot stat file [/var/nvidia/nvcam/settings/imx290_bottom_A6V26.isp]
---- imager: No override file found. ----
Imager: looking for override file [/mnt/factory/camera/factory.bin] 1/16
Imager: looking for override file [/Calib/factory.bin] 2/16
Imager: looking for override file [/Calib/calibration.bin] 3/16
Imager: looking for override file [(null)] 4/16
Imager: looking for override file [(null)] 5/16
Imager: looking for override file [(null)] 6/16
Imager: looking for override file [(null)] 7/16
Imager: looking for override file [(null)] 8/16
Imager: looking for override file [(null)] 9/16
Imager: looking for override file [(null)] 10/16
Imager: looking for override file [(null)] 11/16
Imager: looking for override file [(null)] 12/16
Imager: looking for override file [(null)] 13/16
Imager: looking for override file [(null)] 14/16
Imager: looking for override file [(null)] 15/16
Imager: looking for override file [(null)] 16/16
Imager: looking for override file [/data/nvcam/settings/factory.bin] 1/16
Imager: looking for override file [/data/nvcam/settings/imx290_bottom_A6V26.bin] 2/16
Imager: looking for override file [/opt/nvidia/nvcam/settings/imx290_bottom_A6V26.bin] 3/16
Imager: looking for override file [/var/nvidia/nvcam/settings/imx290_bottom_A6V26.bin] 4/16
Imager: looking for override file [(null)] 5/16
Imager: looking for override file [(null)] 6/16
Imager: looking for override file [(null)] 7/16
Imager: looking for override file [(null)] 8/16
Imager: looking for override file [(null)] 9/16
Imager: looking for override file [(null)] 10/16
Imager: looking for override file [(null)] 11/16
Imager: looking for override file [(null)] 12/16
Imager: looking for override file [(null)] 13/16
Imager: looking for override file [(null)] 14/16
Imager: looking for override file [(null)] 15/16
Imager: looking for override file [(null)] 16/16
NvPclCreateDriver: Found NvPcl Driver Hal dev_name match (v4l2_sensor)
NvPclCreateDriver: Found a Driver name match (v4l2_sensor)
NvPclConnectDrivers: hImager was NULL, creating new imager
NvPclInitializeDrivers: v4l2_sensor ++++++++++++++++++
OFDPropertyGetString: could not read property [devnode-bus]
initialize: imx290 31-001a
(NvOdmDevice) Error ModuleNotPresent: V4L2Device not available (in dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function findDevice(), line 231)
(NvOdmDevice) Error ModuleNotPresent:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function initialize(), line 54)
(NvOdmDevice) Error ModuleNotPresent:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function initialize(), line 97)
NvPclDriverInitializeData: Unable to initialize driver v4l2_sensor
NvPclInitializeDrivers: error: Failed to init camera sub module v4l2_sensor
NvPclStartPlatformDrivers: Failed to start module drivers
NvPclCloseModuleDrivers: deallocate/free overrides pathname @ 0x7facbc34b0
NvPclCloseModuleDrivers: deallocate/free overrides pathname @ 0x7facbc3540
NvPclCloseModuleDrivers: deallocate/free overrides pathname @ 0x7facbc35d0
NvPclCloseModuleDrivers: deallocate/free overrides pathname @ 0x7facbdfe80
NvPclCloseModuleDrivers: deallocate/free overrides pathname @ 0x7facbdff10
NvPclCloseModuleDrivers: deallocate/free overrides pathname @ 0x7facbdffa0
NvPclStateControllerOpen: Failed ImagerGUID 0. (error 0xA000E)
NvPclStateControllerClose: Module imx290_bottom_A6V26 closed
NvPclOpen: PCL Open Failed. Error: 0xf
NvPclClose: ++++++++++++++++++++++
NvPclClose: ----------------------
NvPclOpen: ----------------------
SCF: Error BadParameter: Sensor could not be opened. (in src/services/capture/CaptureServiceDeviceSensor.cpp, function getSourceFromGuid(), line 596)
SCF: Error BadParameter:  (propagating from src/services/capture/CaptureService.cpp, function addSourceByGuid(), line 781)
SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function addSourceByIndex(), line 276)
SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function getSource(), line 439)
Socket read error. Camera Daemon stopped functioning.....
gst_nvcamera_open() failed ret=0
ERROR: Pipeline doesn't want to pause.
ERROR: from element /GstPipeline:pipeline0/GstNvCameraSrc:nvcamerasrc0: GStreamer error: state change failed and some element failed to post a proper error message with the reason for the failure.
Additional debug info:
gstbasesrc.c(3354): gst_base_src_start (): /GstPipeline:pipeline0/GstNvCameraSrc:nvcamerasrc0:
Failed to start
Setting pipeline to NULL ...
Freeing pipeline ...
[1]+  Segmentation fault      (core dumped) /usr/sbin/nvcamera-daemon  (wd: /home/nvidia)
(wd now: /dev)

Lines from device tree:

// from i2c@c240000
		imx290_b@1a {
			compatible = "nvidia,imx290";
			reg = <0x1a>;
			devnode = "video0";
			physical_w = "3.674";
			physical_h = "2.738";
			avdd-reg = "vana";
			iovdd-reg = "vif";
			dvdd-reg = "vdig";
			delayed_gain = "true";
			clocks = <0xd 0x5a 0xd 0x10d>;
                        clock-names = "extperiph2", "pllp_grtba";
                        mclk = "extperiph2";
			reset-gpios = <0x12 0x8d 0x0>;
			vana-supply = <0x1d>;
			vif-supply = <0x1f>;
			vdig-supply = <0x2c>;
                        status = "okay";

			mode0 {
				mclk_khz = "37125";
				num_lanes = [34 00];
				tegra_sinterface = "serial_a";
				discontinuous_clk = "no";
				dpcm_enable = "false";
				cil_settletime = [30 00];
				active_w = "1920";
				active_h = "1080";
				pixel_t = "bayer_rggb";
				readout_orientation = "90";
				line_length = "4200";
				inherent_gain = [31 00];
				mclk_multiplier = "25";
				pix_clk_hz = "756000000";
				min_gain_val = "1.0";
				max_gain_val = "22.2";
				min_hdr_ratio = [31 00];
				max_hdr_ratio = "64";
				min_framerate = "1.462526";
				max_framerate = "60";
				min_exp_time = "16.165";
				max_exp_time = "165770";
				embedded_metadata_height = [31 00];
			};

			ports {
				#address-cells = <0x1>;
				#size-cells = <0x0>;

				port@0 {
					reg = <0x0>;

					endpoint {
						csi-port = <0x0>;
						bus-width = <0x4>;
						remote-endpoint = <0x25>;
						/*linux,phandle = <0xab>;
						phandle = <0xab>;*/
					};
				};
			};
...


// on i2c@3180000
		imx290_c@1a {
			compatible = "nvidia,imx290";
			reg = <0x1a>;
			devnode = "video1";
			physical_w = "3.674";
			physical_h = "2.738";
			avdd-reg = "vana";
			iovdd-reg = "vif";
			dvdd-reg = "vdig";
			delayed_gain = "true";
			clocks = <0xd 0x59 0xd 0x10d>;
                        clock-names = "extperiph1", "pllp_grtba";
                        mclk = "extperiph1";
			reset-gpios = <0x12 0x8d 0x0>;
			vana-supply = <0x1d>;
			vif-supply = <0x1f>;
			vdig-supply = <0x2c>;
			status = "okay";

			mode0 {
				mclk_khz = "37125";
				num_lanes = [34 00];
				tegra_sinterface = "serial_a";
				discontinuous_clk = "no";
				dpcm_enable = "false";
				cil_settletime = [30 00];
				active_w = "1920";
				active_h = "1080";
				pixel_t = "bayer_rggb";
				readout_orientation = "90";
				line_length = "4200";
				inherent_gain = [31 00];
				mclk_multiplier = "25";
				pix_clk_hz = "756000000";
				min_gain_val = "1.0";
				max_gain_val = "22.2";
				min_hdr_ratio = [31 00];
				max_hdr_ratio = "64";
				min_framerate = "1.462526";
				max_framerate = "60";
				min_exp_time = "16.165";
				max_exp_time = "165770";
				embedded_metadata_height = [31 00];
			};

			ports {
				#address-cells = <0x1>;
				#size-cells = <0x0>;

				port@0 {
					reg = <0x0>;

					endpoint {
						csi-port = <0x2>;
						bus-width = <0x4>;
						remote-endpoint = <0x25>;
						/*linux,phandle = <0xab>;
						phandle = <0xab>;*/
					};
				};
			};
		};

...

// on i2c@3190000
		imx290_d@1a {
			compatible = "nvidia,imx290";
			reg = <0x1a>;
			devnode = "video2";
			physical_w = "3.674";
			physical_h = "2.738";
			avdd-reg = "vana";
			iovdd-reg = "vif";
			dvdd-reg = "vdig";
			delayed_gain = "true";
			clocks = <0xd 0x5b 0xd 0x10d>;
                        clock-names = "extperiph3", "pllp_grtba";
                        mclk = "extperiph3";
			reset-gpios = <0x12 0x8d 0x0>;
			vana-supply = <0x1d>;
			vif-supply = <0x1f>;
			vdig-supply = <0x2c>;
			status = "okay";
			/*linux,phandle = <0x177>;
			phandle = <0x177>;*/

			mode0 {
				mclk_khz = "37125";
				num_lanes = [34 00];
				tegra_sinterface = "serial_a";
				discontinuous_clk = "no";
				dpcm_enable = "false";
				cil_settletime = [30 00];
				active_w = "1920";
				active_h = "1080";
				pixel_t = "bayer_rggb";
				readout_orientation = "90";
				line_length = "4200";
				inherent_gain = [31 00];
				mclk_multiplier = "25";
				pix_clk_hz = "756000000";
				min_gain_val = "1.0";
				max_gain_val = "22.2";
				min_hdr_ratio = [31 00];
				max_hdr_ratio = "64";
				min_framerate = "1.462526";
				max_framerate = "60";
				min_exp_time = "16.165";
				max_exp_time = "165770";
				embedded_metadata_height = [31 00];
			};

			ports {
				#address-cells = <0x1>;
				#size-cells = <0x0>;

				port@0 {
					reg = <0x0>;

					endpoint {
						csi-port = <0x4>;
						bus-width = <0x4>;
						remote-endpoint = <0x25>;
						/*linux,phandle = <0xab>;
						phandle = <0xab>;*/
					};
				};
			};
		};


	tegra-camera-platform {
		compatible = "nvidia, tegra-camera-platform";
		num_csi_lanes = <0x4>;
		max_lane_speed = <0x16e360>;
		min_bits_per_pixel = <0xa>;
		vi_peak_byte_per_pixel = <0x2>;
		vi_bw_margin_pct = <0x19>;
		max_pixel_rate = <0xb71b0>;
		isp_peak_byte_per_pixel = <0x5>;
		isp_bw_margin_pct = <0x19>;
		tpg_max_iso = <0x3bc400>;
		linux,phandle = <0x134>;
		phandle = <0x134>;

		modules {

			module0 {
				badge = "imx290_bottomleft_A6V26";
				position = "bottomleft";
				orientation = [31 00];
				status = "okay";
				/*linux,phandle = <0x12c>;*/
				/*phandle = <0x12c>;*/

				drivernode0 {
					pcl_id = "v4l2_sensor";
					devname = "imx290 31-001a";
					proc-device-tree = "/proc/device-tree/i2c@c240000/imx290_b@1a";
					status = "okay";
				};
			};

			module1 {
				badge = "imx290_centerleft_A6V26";
				position = "centerleft";
				orientation = [31 00];
				status = "okay";
				/*linux,phandle = <0x12c>;*/
				/*phandle = <0x12c>;*/

				drivernode0 {
					pcl_id = "v4l2_sensor";
					devname = "imx290 32-001a";
					proc-device-tree = "/proc/device-tree/i2c@3180000/imx290_c@1a";
					status = "okay";
				};
			};

			module2 {
				badge = "imx290_centerright_A6V26";
				position = "centerright";
				orientation = [31 00];
				status = "okay";
				/*linux,phandle = <0x12c>;*/
				/*phandle = <0x12c>;*/

				drivernode0 {
					pcl_id = "v4l2_sensor";
					devname = "imx290 33-001a";
					proc-device-tree = "/proc/device-tree/i2c@3190000/imx290_d@1a";
					status = "okay";
				};
			};
		};

Note: I am developing and rebuilding my kernel on a TX2 according to Build Kernel and Modules - NVIDIA Jetson TX2 - JetsonHacks

Thank you!
James

2

Did you purchase this sensor module or make by yourself?
Does your sensor board have i2c mux? The i2c bus in your DT is looks like not correct.

3

Hi Shane, it’s using 3 custom sensor modules and a custom carrier board, tested and verified as working on other hardware. No I2C mux is used - it’s directly on the TX2 bus.

The I2C address is correct, and the hardware functions correctly from user space for at least 1 camera. We can load the correct register settings on the module and are seeing CSI-2 data on the device pins. What doesn’t look right with the I2C bus?

My driver loads correctly at startup. The problem seems to be when the nvcamera-daemon attempts to create the /dev/video files. The debugging output is not helpful: just a “(NvOdmDevice) Error ModuleNotPresent: V4L2Device not available” error. I have printk lines in my driver, and the daemon is not attempting to set any modes there.

How can we get more information on the cause of this error?

4

Below is not correct. 32 is the bus number. For the i2c@c2400000 is bus 1, i2c@3180000 is bus 2.

devname = “imx290 32-001a”;

5

Hi Shane, thanks for the tip. I renamed them as follows:

// i2c@c2400000
devname = “imx290 1-001a”;
…
// i2c@3180000
devname = “imx290 2-001a”;
…
// i2c@3190000
devname = “imx290 3-001a”;

But it still gives the same error.

6

Did you review the output log from nvcamera-daemon to check any different with previous?

7

Yes, it’s exactly the same error, nothing different in the logs.

8

Does this “initialize: imx290 31-001a” change?

9

Yes, but it doesn’t make any difference - the error is exactly the same.

10

List the v4l2-ctrl --list-devices here.

11

“Failed to open /dev/video0: No such file or directory.”

12

Seems the /dev/videox is not create!!

13

Hi Shane, yes, that was my original question; why isn’t the nvcamera-daemon creating the video0 device? That’s when it’s created, isn’t it? Everything else in my driver works fine, the v4l2 calls in my driver return without error, but the /dev/video0 files for the devices aren’t being created.

Do you have any documentation or hints about the cryptic error codes coming from nvcamera-daemon?

14

If the device node not create that means your sensor driver not loading/probe as v4l2 device could be something wrong with your DT or sensor driver.

You can download the document from the download center to check the sensor programing guide chapter to know the detail.

https://developer.nvidia.com/embedded/dlc/l4t-documentation-28-1

15

Yes, I’ve followed that guide, and the sensor is being probed and initializing fine, as I said in my original post. Could you please give me some help/documentation with the cryptic nvcamera-daemon error messages?

16

I don’t get it if the sensor is being probed and initializing fine why the video node not create?
The problem is the nvcamera-daemon try to query the sensor information from the video node and failed. Please make sure your can pass the v4l2-ctl and v4l2-compliance as to document say.

17

Yes that is the question. I am doing all that according to the docs, but the errors from nvcamera-daemon are cryptic so there is no way to tell what the problem is. You have the source code for nvcamera-daemon sitting there in NVIDIA - could you please give me some hints as to what that error means and how to find out what part of the DT config is causing the error? I’ve already turned on the debugging env vars. Thanks!

18

There’s no clue from the nvcamera-daemon for your case. Your problem should be the device tree have problem to create the video node. There’s no any helpful information from the nvcamera-daemon. You have debug your device tree to figure why. Something need to note is you have to disable the plugin manager, I guess you should done it if you read the document carefully.

19

Yes, I followed all the instructions before I came to the forums, have debugged everything, have an otherwise functioning kernel driver, and have disabled the plugin manager.

You are right; there isn’t helpful information from nvcamera-daemon, but since this is where the problem is, and you could easily check this by browsing the daemon source for me, I want to know if you are willing to help me out from your end. Or find one of your internal developers who could.

20

You are on the wrong way not I am not willing help you here. You should focus on the kernel VI/CSI driver to figure out why the sensor device node not create successful. Even internal developer would tell you focus on on device node.