Jetson TX2 MIPI camera driver debugging: v4l2-ctl works fine but gst-streamer failed

Autonomous Machines Jetson & Embedded Systems Jetson TX2
#1

Hi All,
I’m currently developing a camera sensor driver but I met the following issue:

Run v4l2-ctl:

nvidia@tegra-ubuntu:~$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=3264,height=2448,pixelformat=RG10 --stream-mmap --stream-count=1 --stream-to=test.raw --set-ctrl bypass_mode=0
<
nvidia@tegra-ubuntu:~$

I can get the raw data without any error. After demosaicing I can get a correct picture. However, when I run gst-launch-1.0, I can get only this error:

nvidia@tegra-ubuntu:~$ gst-launch-1.0 nvcamerasrc ! 'video/x-raw(memory:NVMM), width=3264, height=2448, framerate=30/1, format=NV12' ! nvvidconv flip-method=2 ! nvegltransform ! nveglglessink
Setting pipeline to PAUSED ...
Socket read error. Camera Daemon stopped functioning.....
gst_nvcamera_open() failed ret=0
ERROR: Pipeline doesn't want to pause.
Got context from element 'eglglessink0': gst.egl.EGLDisplay=context, display=(GstEGLDisplay)NULL;
ERROR: from element /GstPipeline:pipeline0/GstNvCameraSrc:nvcamerasrc0: GStreamer error: state change failed and some element failed to post a proper error message with the reason for the failure.
Additional debug info:
gstbasesrc.c(3354): gst_base_src_start (): /GstPipeline:pipeline0/GstNvCameraSrc:nvcamerasrc0:
Failed to start
Setting pipeline to NULL ...
Caught SIGSEGV
#0  0x0000007f941ae130 in pthread_join (threadid=547890934272, 
#1  0x0000007f94264e40 in ?? () from /lib/aarch64-linux-gnu/libglib-2.0.so.0
#2  0x0000000000000011 in ?? ()
Spinning.  Please run 'gdb gst-launch-1.0 1960' to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.

The jetpack version is R28.1, and the camera sensor is OV8856, an 8MP mipi sensor. I’ve developed several camera sensor drivers for TX2, in my previous experience, after the drivers worked with v4l2-ctl, they should work with gst-launch-1.0 as well except this one.
Could anyone help me? Thanks a lot.

#2

Did you launch the APP from ssh? Please try on the tegra ubuntu OS. If still not working please enable more log by https://elinux.org/Jetson_TX2/28.1_Camera_BringUp “To enable logs from user-space for more details”

ERROR: from element /GstPipeline:pipeline0/GstNvCameraSrc:nvcamerasrc0: GStreamer error: state change failed and some element failed to post a proper error message with the reason for the failure.

#3

No,I launch the APP from the Ubuntu OS desktop, not from ssh.
I enabled logs and got following messages, any idea?

root@tegra-ubuntu:/home/nvidia# /usr/sbin/nvcamera-daemon
^[[A^[[B^[[BThread 1 getting next capture
Thread 1 is waiting
Thread 2 getting next capture
Thread 2 is waiting
Thread 3 getting next capture
Thread 3 is waiting
Thread 4 getting next capture
Thread 4 is waiting
Thread 5 getting next capture
Thread 5 is waiting
Thread 6 getting next capture
Thread 6 is waiting
Thread 7 getting next capture
Thread 7 is waiting
Thread 8 getting next capture
Thread 8 is waiting
Thread 9 getting next capture
Thread 9 is waiting
Thread 10 getting next capture
Thread 10 is waiting
Thread 11 getting next capture
Thread 11 is waiting
Thread 12 getting next capture
Thread 12 is waiting
Starting services...
Worker thread IspHw statsComplete start
Worker thread IspHw frameComplete start
Worker thread CaptureScheduler checkFramePending start
Worker thread CaptureScheduler frameStart start
Worker thread V4L2CaptureScheduler checkCaptureComplete start
Worker thread V4L2CaptureScheduler issueCaptures start
Worker thread PS handleRequests start
getInstance: s_instance(0x7f98b92570)
getInstance: s_instance(0x7f98b92570)
subscribe: create SensorType(gyroscope) sensor(0x7f98b946f0)
subscribe: create SensorType(accelerometer) sensor(0x7f98b99b50)
AC plugin not present: dlopen "acplugin.so", acplugin.so: cannot open shared object file: No such file or directory
Services are started
NvPclSetHotplugCallback: ++++++++++++++++++++++
---- Imager: Calibration blob file handling supported in this build ----
NvPclHwGetModuleList: OFParserListModules Succeeded
NvPclHwMatchPartNum: Found calibration Part#:P5V27CCV, P5V27C:"P5V27C"
NvPclHwPrintModuleDefinition -- Name: e3326_front_P5V27C
NvPclHwPrintModuleDefinition -- Position: 0
NvPclHwPrintModuleDefinition -- CalibrationData Found: 1
NvPclHwPrintCameraSubModule -- HwCamSubModule[0].Name: v4l2_sensor
NvPclHwPrintCameraSubModule -- HwCamSubModule[0].DevName: ov13850 2-0036
NvPclHwPrintCameraSubModule -- HwCamSubModule[1].Name: v4l2_lens
NvPclHwPrintCameraSubModule -- HwCamSubModule[1].DevName: lc898212 2-0072
NvPclHwGetModuleList: OFParserListModules Succeeded
NvPclModuleListInitialize: NvPclModule list[0]: e3326_front_P5V27C position0
NvPclHwScanExternalCameras -- adding video0 to discover list
NvPclHwScanExternalCameras -- adding video0 to discover list
initialize: /dev/video0
NvPclHwScanExternalCameras -- adding video0 to discover list
initialize: /dev/video0
getHotplugMonitor: Getting hotplug monitor instance
 initializeHotplug++
 hotPlugfunc ++ 
 addWatch: Watch added wd='1'
CheckProcDTExists: INFO: accessing /proc/device-tree/tegra-virtual-camera-platform/modules/module0/drivernode1; No such file or directory
PCLHW_DTParser
CheckProcDTExists: INFO: accessing /proc/device-tree/tegra-virtual-camera-platform/modules/module1/drivernode0; No such file or directory
PCLHW_DTParser
setHotplugCallback: Registered new callback client
NvPclSetHotplugCallback: ----------------------
NvPclOpen: ++++++++++++++++++++++
NvPclStateControllerOpen: Found GUID 0 match at index[0]
NvPclHwInitializeModule: allocate overrides pathname @ 0x7f98bc38d0
NvPclHwInitializeModule: allocate overrides pathname @ 0x7f98bc3960
NvPclHwInitializeModule: allocate overrides pathname @ 0x7f98bc39f0
NvPclHwInitializeModule: allocate overrides pathname @ 0x7f98bc3a80
NvPclHwInitializeModule: allocate overrides pathname @ 0x7f98bc3b10
NvPclHwInitializeModule: allocate overrides pathname @ 0x7f98bc3ba0
LoadOverridesFile: looking for override file [/Calib/camera_override.isp] 1/16CheckOverridesPermissions: cannot stat file [/Calib/camera_override.isp]
LoadOverridesFile: looking for override file [/data/nvcam/settings/camera_overrides.isp] 2/16CheckOverridesPermissions: cannot stat file [/data/nvcam/settings/camera_overrides.isp]
LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/camera_overrides.isp] 3/16CheckOverridesPermissions: cannot stat file [/opt/nvidia/nvcam/settings/camera_overrides.isp]
LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/camera_overrides.isp] 4/16CheckOverridesPermissions: cannot stat file [/var/nvidia/nvcam/settings/camera_overrides.isp]
LoadOverridesFile: looking for override file [/data/nvcam/camera_overrides.isp] 5/16CheckOverridesPermissions: cannot stat file [/data/nvcam/camera_overrides.isp]
LoadOverridesFile: looking for override file [/data/nvcam/settings/e3326_front_P5V27C.isp] 6/16CheckOverridesPermissions: cannot stat file [/data/nvcam/settings/e3326_front_P5V27C.isp]
LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/e3326_front_P5V27C.isp] 7/16CheckOverridesPermissions: cannot stat file [/opt/nvidia/nvcam/settings/e3326_front_P5V27C.isp]
LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/e3326_front_P5V27C.isp] 8/16CheckOverridesPermissions: cannot stat file [/var/nvidia/nvcam/settings/e3326_front_P5V27C.isp]
---- imager: No override file found. ----
Imager: looking for override file [/mnt/factory/camera/factory.bin] 1/16
Imager: looking for override file [/Calib/factory.bin] 2/16
Imager: looking for override file [/Calib/calibration.bin] 3/16
Imager: looking for override file [(null)] 4/16
Imager: looking for override file [(null)] 5/16
Imager: looking for override file [(null)] 6/16
Imager: looking for override file [(null)] 7/16
Imager: looking for override file [(null)] 8/16
Imager: looking for override file [(null)] 9/16
Imager: looking for override file [(null)] 10/16
Imager: looking for override file [(null)] 11/16
Imager: looking for override file [(null)] 12/16
Imager: looking for override file [(null)] 13/16
Imager: looking for override file [(null)] 14/16
Imager: looking for override file [(null)] 15/16
Imager: looking for override file [(null)] 16/16
Imager: looking for override file [/data/nvcam/settings/factory.bin] 1/16
Imager: looking for override file [/data/nvcam/settings/e3326_front_P5V27C.bin] 2/16
Imager: looking for override file [/opt/nvidia/nvcam/settings/e3326_front_P5V27C.bin] 3/16
Imager: looking for override file [/var/nvidia/nvcam/settings/e3326_front_P5V27C.bin] 4/16
Imager: looking for override file [(null)] 5/16
Imager: looking for override file [(null)] 6/16
Imager: looking for override file [(null)] 7/16
Imager: looking for override file [(null)] 8/16
Imager: looking for override file [(null)] 9/16
Imager: looking for override file [(null)] 10/16
Imager: looking for override file [(null)] 11/16
Imager: looking for override file [(null)] 12/16
Imager: looking for override file [(null)] 13/16
Imager: looking for override file [(null)] 14/16
Imager: looking for override file [(null)] 15/16
Imager: looking for override file [(null)] 16/16
NvPclCreateDriver: Found NvPcl Driver Hal dev_name match (v4l2_sensor)
NvPclCreateDriver: Found a Driver name match (v4l2_sensor)
NvPclCreateDriver: Found NvPcl Driver Hal dev_name match (v4l2_lens)
NvPclCreateDriver: Found a Driver name match (v4l2_lens)
NvPclConnectDrivers: hImager was NULL, creating new imager
NvPclInitializeDrivers: v4l2_sensor ++++++++++++++++++
OFDPropertyGetString: could not read property [devnode-bus]
initialize: ov13850 2-0036
(NvOdmDevice) Error ModuleNotPresent: V4L2Device not available (in dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function findDevice(), line 231)
(NvOdmDevice) Error ModuleNotPresent:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function initialize(), line 54)
(NvOdmDevice) Error ModuleNotPresent:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function initialize(), line 97)
NvPclDriverInitializeData: Unable to initialize driver v4l2_sensor
NvPclInitializeDrivers: error: Failed to init camera sub module v4l2_sensor
NvPclStartPlatformDrivers: Failed to start module drivers
NvPclDriver_V4L2_Focuser_Stub_Close: Invalid NULL input pPclDriver
NvPclCloseModuleDrivers: deallocate/free overrides pathname @ 0x7f98bc3a80
NvPclCloseModuleDrivers: deallocate/free overrides pathname @ 0x7f98bc3b10
NvPclCloseModuleDrivers: deallocate/free overrides pathname @ 0x7f98bc3ba0
NvPclCloseModuleDrivers: deallocate/free overrides pathname @ 0x7f98bc38d0
NvPclCloseModuleDrivers: deallocate/free overrides pathname @ 0x7f98bc3960
NvPclCloseModuleDrivers: deallocate/free overrides pathname @ 0x7f98bc39f0
NvPclStateControllerOpen: Failed ImagerGUID 0. (error 0xA000E)
NvPclStateControllerClose: Module e3326_front_P5V27C closed
NvPclOpen: PCL Open Failed. Error: 0xf
NvPclClose: ++++++++++++++++++++++
NvPclClose: ----------------------
NvPclOpen: ----------------------
SCF: Error BadParameter: Sensor could not be opened. (in src/services/capture/CaptureServiceDeviceSensor.cpp, function getSourceFromGuid(), line 596)
SCF: Error BadParameter:  (propagating from src/services/capture/CaptureService.cpp, function addSourceByGuid(), line 781)
SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function addSourceByIndex(), line 276)
SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function getSource(), line 439)
Segmentation fault (core dumped)
root@tegra-ubuntu:/home/nvidia#
#4

After compared the logs of a working sensor and a failed sensor, I believe the following error is the key. I’m just confused that why it said V4L2Device is not available while I can capture the image via v4l2-ctl?

(NvOdmDevice) Error ModuleNotPresent: V4L2Device not available (in dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function findDevice(), line 231)
(NvOdmDevice) Error ModuleNotPresent:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function initialize(), line 54)
(NvOdmDevice) Error ModuleNotPresent:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function initialize(), line 97)
NvPclDriverInitializeData: Unable to initialize driver v4l2_sensor
NvPclInitializeDrivers: error: Failed to init camera sub module v4l2_sensor
NvPclStartPlatformDrivers: Failed to start module drivers
#5

Check the dev name of the DT is the same with the v4l2 kernel driver report.

#6

Oh, thank you so much, after I fixed the dev name the problem is gone!

#7

I’m having the same issue. Can you explain to me how to check that? Thanks

#8

Check the devname with the kernel sensor driver.

tegra-camera-platform {
 
 		compatible = "nvidia, tegra-camera-platform";
 		/**
 		* The general guideline for naming badge_info contains 3 parts, and is as follows,
 		* The first part is the camera_board_id for the module; if the module is in a FFD
 		* platform, then use the platform name for this part.
 		* The second part contains the position of the module, ex. “rear” or “front”.
 		* The third part contains the last 6 characters of a part number which is found
 		* in the module's specsheet from the vender.
 		*/
 		modules {
 			module0 {
 				badge = "e3326_front_P5V27C";
 				position = "rear";
 				orientation = "1";
 				drivernode0 {
 					/* Declare PCL support driver (classically known as guid)  */
 					pcl_id = "v4l2_sensor";
 					/* Driver's v4l2 device name */
 					devname = "ov5693 6-0036";
 					/* Declare the device-tree hierarchy to driver instance */
 					proc-device-tree = "/proc/device-tree/host1x/i2c@546c0000/ov5693_c@36";
 				};
 				drivernode1 {
 					/* Declare PCL support driver (classically known as guid)  */
 					pcl_id = "v4l2_lens";
 					proc-device-tree = "/proc/device-tree/e3326_lens_ov5693@P5V27C/";
 				};
 			};
 		};
 	};
#9

can you tell me how to open the device tree file? I can’t find it!

#10

You might find this of use:
https://devtalk.nvidia.com/default/topic/1037131/jetson-tx1/does-anybody-know-how-to-enable-and-program-an-spi-protocol-on-the-tx1-/post/5269528/#5269528

As some additional information, there is a kernel build target (after configuring) for “make dtbs”. This assembles a number of “.dts” and “.dtsi” files, and then does the equivalent of what you find in that above URL. The Jetson itself essentially has a binary format on it which is a result of that kernel build target, and this can be reverse compiled. There may be some very slight edits in CBOOT to the one actually in a partition, but much of it is an exact copy. It depends on the release. The dts “source” form is human readable. The dtb “binary” format is what the system reads.

Important: “make dtbs” is 100% independent of the actual kernel and kernel build. Changing the device tree in no way changes the kernel code in any direct manner. However, drivers make reference to some symbols, and the device tree is how the driver looks up the value belonging to that symbol. So the two work together, but only at run time…changing data values rather than logic.

#11

thank you very much!!!you help me remove a stone in heart! thanks！

#12

Hello,
Where can I find the devname in kernel sensor driver ?
For example devname is “ov5693 6-0036” in my device tree. Do I need to find the same string in driver source code ? Could you please spesify spesific source file if you could ?

Thanks,

#13

Someone else will need to answer that, but as a general tip, some hardware is “plug n play”, meaning the hardware self-reports what it is and where to find it…then a driver can be installed to the device. In the case of hardware being described in a device tree the tree will often describe where to find the hardware and what the hardware is compatible with…the things which are missing when a device cannot self-report. This often includes an association of device name with device driver, along with some method of finding the device (e.g., a physical address).

Someone else will need to answer the specifics of your question since I don’t work with cameras.