28.2 nvcamera-daemon crashes with a segmentation fault on spi sensor

Hi,

I have a SPI CMOS sensor connected to a tx1 evk with 28.2.1, and nvcamera-daemon crashes when starting a pipeline using nvcamerasrc.

The first error messages I see (using the environment macros documented at Jetson TX2 Camera BringUp - eLinux.org are :

OFFileToString: fgets() error.  No characters have been read
OFDPropertyAdd: could not read path /proc/device-tree/spi@7000d600/imx264@1/spi-lsb-first [spi-lsb-first]
OFDriverNodeToDict: Failed to store property 'spi-lsb-first'
OFParserEnumerateDriverNode: v4l2_sensor: failed to parse device-tree to driver node

I don’t know yet if that’s the root cause for the crash of nvcamera-daemon, but that’s clearly an error in nvcamera-daemon as spi-lsb-first, and its friends spi-pha and spi-cpol are empty files :

nvidia@tegra-ubuntu:~$ ls -l /proc/device-tree/spi@7000d600/imx264@1/spi-*
-r--r--r-- 1 root root 0 Jul  6 13:35 /proc/device-tree/spi@7000d600/imx264@1/spi-cpha
-r--r--r-- 1 root root 0 Jul  6 13:35 /proc/device-tree/spi@7000d600/imx264@1/spi-cpol
-r--r--r-- 1 root root 0 Jul  6 13:35 /proc/device-tree/spi@7000d600/imx264@1/spi-lsb-first
nvidia@tegra-ubuntu:~$

I have removed the spi-cpha, spi-cpol and spi-lsb-first entry of my dts file and hardcoded those settings in my driver, but that does not make nvcamera-daemon to succeed.

Here is nvcamera-daemon debug log :

Thread 1 getting next capture
Thread 1 is waiting
Thread 2 getting next capture
Thread 2 is waiting
Thread 3 getting next capture
Thread 3 is waiting
Thread 4 getting next capture
Thread 4 is waiting
Thread 5 getting next capture
Thread 5 is waiting
Thread 6 getting next capture
Thread 6 is waiting
Thread 7 getting next capture
Thread 8 getting next capture
Thread 7 is waiting
Thread 9 getting next capture
Thread 8 is waitingThread 10 getting next capture

Thread 11 getting next capture
Thread 9 is waiting
Thread 12 getting next capture
Thread 10 is waiting
Thread 11 is waiting
Thread 12 is waiting
Starting services...
Worker thread IspHw statsComplete start
Worker thread IspHw frameComplete start
Worker thread IspHw statsComplete start
Worker thread IspHw frameComplete start
Worker thread CaptureScheduler checkFramePending start
Worker thread CaptureScheduler frameStart start
Worker thread V4L2CaptureScheduler checkCaptureComplete start
Worker thread V4L2CaptureScheduler issueCaptures start
Worker thread PS handleRequests start
getInstance: s_instance(0x7f89180fb0)
getInstance: s_instance(0x7f89180fb0)
subscribe: create SensorType(gyroscope) sensor(0x7f89183130)
subscribe: create SensorType(accelerometer) sensor(0x7f89188590)
AC plugin not present: dlopen "acplugin.so", acplugin.so: cannot open shared object file: No such file or directory
Services are started
NvPclSetHotplugCallback: ++++++++++++++++++++++
---- Imager: Calibration blob file handling supported in this build ----
NvPclHwGetModuleList: OFParserListModules Succeeded
NvPclHwMatchPartNum: Found calibration Part#: A6V24:"A6V24"
NvPclHwGetModuleList: No module data found
NvPclHwPrintModuleDefinition -- Name: imx264_rear_A6V24
NvPclHwPrintModuleDefinition -- Position: 0
NvPclHwPrintModuleDefinition -- CalibrationData Found: 1
NvPclHwPrintCameraSubModule -- HwCamSubModule[0].Name: v4l2_sensor
NvPclHwPrintCameraSubModule -- HwCamSubModule[0].DevName: imx264 30-0010
NvPclHwGetModuleList: OFParserListModules Succeeded
NvPclModuleListInitialize: NvPclModule list[0]: imx264_rear_A6V24 position0
NvPclHwScanExternalCameras -- adding video0 to discover list
NvPclHwScanExternalCameras -- adding video0 to discover list
initialize: /dev/video0
queryControls: 9963795 control Gain
queryControls: 10100736 control Frame Length
queryControls: 10100737 control Coarse Time
queryControls: 10100738 control Coarse Time Short
queryControls: 10100739 control Group Hold
queryControls: 10100740 control HDR enable
queryControls: 10100836 control Bypass Mode
queryControls: 10100837 control Override Enable
queryControls: 10100838 control Height Align
queryControls: 10100839 control Size Align
queryControls: 10100840 control Write ISP format
queryControls: 10100841 control Sensor Signal Properties
queryControls: 10100842 control Sensor Image Properties
queryControls: 10100843 control Sensor Control Properties
queryControls: 10100844 control Sensor DV Timings
queryControls: 10100866 control Sensor Modes
Ioctl failed: Invalid argument
NvPclHwScanExternalCameras -- adding video0 to discover list
initialize: /dev/video0
queryControls: 9963795 control Gain
queryControls: 10100736 control Frame Length
queryControls: 10100737 control Coarse Time
queryControls: 10100738 control Coarse Time Short
queryControls: 10100739 control Group Hold
queryControls: 10100740 control HDR enable
queryControls: 10100836 control Bypass Mode
queryControls: 10100837 control Override Enable
queryControls: 10100838 control Height Align
queryControls: 10100839 control Size Align
queryControls: 10100840 control Write ISP format
queryControls: 10100841 control Sensor Signal Properties
queryControls: 10100842 control Sensor Image Properties
queryControls: 10100843 control Sensor Control Properties
queryControls: 10100844 control Sensor DV Timings
queryControls: 10100866 control Sensor Modes
Ioctl failed: Invalid argument
getHotplugMonitor: Getting hotplug monitor instance
 initializeHotplug++
 hotPlugfunc ++  addWatch: Watch added wd='1'
CheckProcDTExists: INFO: accessing /proc/device-tree/tegra-virtual-camera-platform/modules; No such file or directory
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
setHotplugCallback: Registered new callback client
NvPclSetHotplugCallback: ----------------------

NvPclOpen: ++++++++++++++++++++++
NvPclStateControllerOpen: Found GUID 0 match at index[0]
NvPclHwInitializeModule: allocate overrides pathname @ 0x7f891bb170
NvPclHwInitializeModule: allocate overrides pathname @ 0x7f891bb200
NvPclHwInitializeModule: allocate overrides pathname @ 0x7f891bb290
NvPclHwInitializeModule: allocate overrides pathname @ 0x7f891bb320
NvPclHwInitializeModule: allocate overrides pathname @ 0x7f891bb3b0
NvPclHwInitializeModule: allocate overrides pathname @ 0x7f891bb440
 hotPlugfunc: reading from inotify FD, Thread waiting 
LoadOverridesFile: looking for override file [/Calib/camera_override.isp] 1/16CheckOverridesPermissions: cannot stat file [/Calib/camera_override.isp]
LoadOverridesFile: looking for override file [/data/nvcam/settings/camera_overrides.isp] 2/16CheckOverridesPermissions: cannot stat file [/data/nvcam/settings/camera_overrides.isp]
LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/camera_overrides.isp] 3/16CheckOverridesPermissions: cannot stat file [/opt/nvidia/nvcam/settings/camera_overrides.isp]
LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/camera_overrides.isp] 4/16CheckOverridesPermissions: cannot stat file [/var/nvidia/nvcam/settings/camera_overrides.isp]
LoadOverridesFile: looking for override file [/data/nvcam/camera_overrides.isp] 5/16CheckOverridesPermissions: cannot stat file [/data/nvcam/camera_overrides.isp]
LoadOverridesFile: looking for override file [/data/nvcam/settings/imx264_rear_A6V24.isp] 6/16CheckOverridesPermissions: cannot stat file [/data/nvcam/settings/imx264_rear_A6V24.isp]
LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/imx264_rear_A6V24.isp] 7/16CheckOverridesPermissions: cannot stat file [/opt/nvidia/nvcam/settings/imx264_rear_A6V24.isp]
LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/imx264_rear_A6V24.isp] 8/16CheckOverridesPermissions: cannot stat file [/var/nvidia/nvcam/settings/imx264_rear_A6V24.isp]
---- imager: No override file found. ----
Imager: looking for override file [/mnt/factory/camera/factory.bin] 1/16
Imager: looking for override file [/Calib/factory.bin] 2/16
Imager: looking for override file [/Calib/calibration.bin] 3/16
Imager: looking for override file [(null)] 4/16
Imager: looking for override file [(null)] 5/16
Imager: looking for override file [(null)] 6/16
Imager: looking for override file [(null)] 7/16
Imager: looking for override file [(null)] 8/16
Imager: looking for override file [(null)] 9/16
Imager: looking for override file [(null)] 10/16
Imager: looking for override file [(null)] 11/16
Imager: looking for override file [(null)] 12/16
Imager: looking for override file [(null)] 13/16
Imager: looking for override file [(null)] 14/16
Imager: looking for override file [(null)] 15/16
Imager: looking for override file [(null)] 16/16
Imager: looking for override file [/data/nvcam/settings/factory.bin] 1/16
Imager: looking for override file [/data/nvcam/settings/imx264_rear_A6V24.bin] 2/16
Imager: looking for override file [/opt/nvidia/nvcam/settings/imx264_rear_A6V24.bin] 3/16
Imager: looking for override file [/var/nvidia/nvcam/settings/imx264_rear_A6V24.bin] 4/16
Imager: looking for override file [(null)] 5/16
Imager: looking for override file [(null)] 6/16
Imager: looking for override file [(null)] 7/16
Imager: looking for override file [(null)] 8/16
Imager: looking for override file [(null)] 9/16
Imager: looking for override file [(null)] 10/16
Imager: looking for override file [(null)] 11/16
Imager: looking for override file [(null)] 12/16
Imager: looking for override file [(null)] 13/16
Imager: looking for override file [(null)] 14/16
Imager: looking for override file [(null)] 15/16
Imager: looking for override file [(null)] 16/16
NvPclCreateDriver: Found NvPcl Driver Hal dev_name match (v4l2_sensor)
NvPclCreateDriver: Found a Driver name match (v4l2_sensor)
NvPclConnectDrivers: hImager was NULL, creating new imager
NvPclInitializeDrivers: v4l2_sensor ++++++++++++++++++
OFDPropertyGetString: could not read property [devnode-bus]
initialize: imx264 30-0010
(NvOdmDevice) Error ModuleNotPresent: V4L2Device not available (in dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function findDevice(), line 245)
(NvOdmDevice) Error ModuleNotPresent:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function initialize(), line 55)
(NvOdmDevice) Error ModuleNotPresent:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function initialize(), line 103)
NvPclDriverInitializeData: Unable to initialize driver v4l2_sensor
NvPclInitializeDrivers: error: Failed to init camera sub module v4l2_sensor
NvPclStartPlatformDrivers: Failed to start module drivers
NvPclCloseModuleDrivers: deallocate/free overrides pathname @ 0x7f891bb320
NvPclCloseModuleDrivers: deallocate/free overrides pathname @ 0x7f891bb3b0
NvPclCloseModuleDrivers: deallocate/free overrides pathname @ 0x7f891bb440
NvPclCloseModuleDrivers: deallocate/free overrides pathname @ 0x7f891bb170
NvPclCloseModuleDrivers: deallocate/free overrides pathname @ 0x7f891bb200
NvPclCloseModuleDrivers: deallocate/free overrides pathname @ 0x7f891bb290
NvPclStateControllerOpen: Failed ImagerGUID 0. (error 0xA000E)
NvPclStateControllerClose: Module imx264_rear_A6V24 closed
NvPclOpen: PCL Open Failed. Error: 0xf
NvPclClose: ++++++++++++++++++++++
NvPclClose: ----------------------
NvPclOpen: ----------------------
SCF: Error BadParameter: Sensor could not be opened. (in src/services/capture/CaptureServiceDeviceSensor.cpp, function getSourceFromGuid(), line 598)
SCF: Error BadParameter:  (propagating from src/services/capture/CaptureService.cpp, function addSourceByGuid(), line 781)
SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function addSourceByIndex(), line 276)
SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function getSource(), line 439)
2573 Segmentation fault

find also attached the strace log
nvcamera-daemon.syc2.gz (33.4 KB)

Does the devname in DT the same with your kernel imx264 driver name?

In /proc/device-tree, the path to my sensor is ‘/proc/device-tree/spi@7000d600/imx264@1’.

This is also the value that I have put in “/proc/device-tree/tegra-camera-platform/modules/module0/drivernode0/proc-device-tree”.

Which value should I set for ‘devname’ ?

Where is that documented ?

I have now understood that for a SPI sensor, the equivalent of I2C’s “30-0010” is “spi1.1”.

I have thus set my devname to “imx264 spi1.1”

That still fails :

NvPclCreateDriver: Found NvPcl Driver Hal dev_name match (v4l2_sensor)
NvPclCreateDriver: Found a Driver name match (v4l2_sensor)
NvPclConnectDrivers: hImager was NULL, creating new imager
NvPclInitializeDrivers: v4l2_sensor ++++++++++++++++++
OFDPropertyGetString: could not read property [devnode-bus]
initialize: imx264 spi1.1
(NvOdmDevice) Error ModuleNotPresent: V4L2Device not available (in dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function findDevice(), line 245)
(NvOdmDevice) Error ModuleNotPresent:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/V4L2Device.cpp, function initialize(), line 55)
(NvOdmDevice) Error ModuleNotPresent:  (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function initialize(), line 103)
NvPclDriverInitializeData: Unable to initialize driver v4l2_sensor
NvPclInitializeDrivers: error: Failed to init camera sub module v4l2_sensor
NvPclStartPlatformDrivers: Failed to start module drivers
NvPclCloseModuleDrivers: deallocate/free overrides pathname @ 0x7f991bb320
NvPclCloseModuleDrivers: deallocate/free overrides pathname @ 0x7f991bb3b0
NvPclCloseModuleDrivers: deallocate/free overrides pathname @ 0x7f991bb440
NvPclCloseModuleDrivers: deallocate/free overrides pathname @ 0x7f991bb170
NvPclCloseModuleDrivers: deallocate/free overrides pathname @ 0x7f991bb200
NvPclCloseModuleDrivers: deallocate/free overrides pathname @ 0x7f991bb290
NvPclStateControllerOpen: Failed ImagerGUID 0. (error 0xA000E)
NvPclStateControllerClose: Module imx264_rear_A6V24 closed
NvPclOpen: PCL Open Failed. Error: 0xf
NvPclClose: ++++++++++++++++++++++
NvPclClose: ----------------------
NvPclOpen: ----------------------
SCF: Error BadParameter: Sensor could not be opened. (in src/services/capture/CaptureServiceDeviceSensor.cpp, function getSourceFromGuid(), line 598)
SCF: Error BadParameter:  (propagating from src/services/capture/CaptureService.cpp, function addSourceByGuid(), line 781)
SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function addSourceByIndex(), line 276)
SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function getSource(), line 439)
2179 Segmentation faul

core.gz (12.4 MB)

What the v4l2-ctl -D shows?

nvidia@evk-ubuntu:~$ v4l2-ctl -D
Driver Info (not using libv4l2):
        Driver name   : tegra-video
        Card type     : vi-output, imx264
        Bus info      : platform:54080000.vi:0
        Driver version: 4.4.38
        Capabilities  : 0x84200001
                Video Capture
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps   : 0x04200001
                Video Capture
                Streaming
                Extended Pix Format
nvidia@evk-ubuntu:~$
phdm@K2000:~/libmelpi-benchmarks$

Try to repo the devname as devname = “imx264” only

drivernode0 {
pcl_id = “v4l2_sensor”;
devname = “ov5693 30-0036”;

a core file is now attached to comment #5

Could you explain what is the core file for?

A ‘core’ file is a disk file containing an image of the process’s memory at the time of termination. You can use it with gdb and a not-stripped version of nvcamera-daemon as follows :

gdb nvcamera-daemon core

to debug the segmentation fault crash

I have attached it here on request from Shan Neng Chen - 07/12/2018 1:51 AM on https://developer.nvidia.com/nvidia_bug/2256334

@phdm
I don’t think I can help you to run the gdb nvcamera-daemon. They is no problem with reference camera board.

@ShaneCC,

when a process terminates with a ‘segmentation fault’, is is a bug. Only people with access ro the source code of the program can solve that bug, As the source code of nvcamera-daemon seems to be only available at nvidia, only nvidia can solve that bug at the moment. So there are two way you can help to solve that bug :

  • either use the core file I have provided, a debugger program like gdb and the source code of nvcamera-deamon, figure out what the problem is, fix it and make another nvcamera-daemon binary available
  • either provide me a link to the sources of nvcamera-daemon so that I can try to fix the bug myself

Best regards

The nvcamera-daemon may not handle well some driver or DT implement problem cause the segmentation fault you should fix it by sensor driver or DT struggling on the binary isn’t help on this problem.

When a program crashes it is a bug. A program should never crash because of unexpected inputs.

However, if you point me to the documentation of the required and accepted inputs for nvcamera-daemon, I can try to fix my dt or driver.

Just a suggestion…run under strace. The system call near the moment of failure might give some useful information. If you run in a terminal with a large scrollback you won’t need a log file, and the usual culprit occurs in the last few hundred lines so you won’t need more than about a 300 line scrollback buffer. strace can be used without debug symbols and without source code.

ltrace would be similar for library calls, but eventually library calls end up as system calls so I wouldn’t bother with that unless strace fails to help.

@linuxdev

thank you for your suggestion

I have already run nvcamera-daemon under strace and ltrace, but that gave me absolutely no clue about the possible crash cause.

You could copy and paste the last few hundred lines here if you use the “code” icon in the upper right (looks like “</>” and adds scrollbars and preserves formatting), or name it with “.txt” and attach to an existing post (the paper clip attach icon only appears if you hover the mouse over the quote mark of an existing post).

I’d recommend logging something like this:

strace -o Log.txt -a 120

…I’m not sure how much log you’d get prior to the last few hundred lines, but if you have issues with the log being too large prior to getting what you want you might consider using the similar command to attach to a running process right before you think a crash is about to start.

Hi linuxdev,

thank you for your answer

As I wrote before, I had already used strace and ltrace to try to know a little bit more about nvcamera-daemon. However it’s not that simple :

  1. nvcamera-daemon is a multithreaded process, so I needed to add at least the ‘-f’ option to strace and ltrace

  2. using ltrace I found no call to strcmp- or scanf- like functions

I am far from my devkit at the moment and I am still waiting (hoping) for a documentation from nvidia people.

For this particular “nvcamera-daemon with spi” problem, I finally gave the answer in [url]https://devtalk.nvidia.com/default/topic/1038008/nvcamera-daemon-dt-device-tree-documentation-for-l4t-28-2-/?offset=19#5274046[/url].

Basically, for a spi sensor, dev_info and friends show something like ‘imx264 spi1.1’ in kernel messages, but the subdev is simply called ‘imx264’, while for i2c sensors both strings are equal.

And nvcamera-daemon does not explain what it is searching for, nor where it is searching it, even with both debugging levels set to 5, and then crashes.

Actually the problem is in the kernel not giving as name “imx264 spi1.1” to the subdev, but simply “imx264”, but nvcamera-daemon should give more informative error messages, and not crash.