libargus does not find any sensors

stefan.zegenhagen · May 6, 2020, 5:32pm

It seems that libargus from R28.1 on TX2 is not happy with our cameras. The following log is given:

Thread 1 getting next capture
Thread 1 is waiting
Thread 2 getting next capture
Thread 2 is waiting
Thread 3 getting next capture
Thread 3 is waiting
Thread 4 getting next capture
Thread 4 is waiting
Thread 5 getting next capture
Thread 5 is waiting
Thread 6 getting next capture
Thread 6 is waiting
Thread 7 getting next capture
Thread 7 is waiting
Thread 8 getting next capture
Thread 8 is waiting
Thread 9 getting next capture
Thread 9 is waiting
Thread 10 getting next capture
Thread 10 is waiting
Thread 11 getting next capture
Thread 11 is waitingThread 12 getting next capture

Thread 12 is waiting
Starting services...
Worker thread IspHw statsComplete start
Worker thread IspHw frameComplete start
Worker thread CaptureScheduler checkFramePending start
Worker thread CaptureScheduler frameStart start
Worker thread V4L2CaptureScheduler checkCaptureComplete start
Worker thread V4L2CaptureScheduler issueCaptures start
Worker thread PS handleRequests start
getInstance: s_instance(0x1020560)
getInstance: s_instance(0x1020560)
subscribe: create SensorType(gyroscope) sensor(0x10226e0)
subscribe: create SensorType(accelerometer) sensor(0x1027b40)
AC plugin not present: dlopen "acplugin.so", acplugin.so: cannot open shared object file: No such file or directory
Services are started
NvPclSetHotplugCallback: ++++++++++++++++++++++
---- Imager: Calibration blob file handling supported in this build ----
NvPclHwGetModuleList: OFParserListModules Succeeded
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwPrintModuleDefinition -- Name: imx290_front_DCTAFC
NvPclHwPrintModuleDefinition -- Position: 1
NvPclHwPrintModuleDefinition -- CalibrationData Found: 1
NvPclHwPrintCameraSubModule -- HwCamSubModule[0].Name: v4l2_sensor
NvPclHwPrintCameraSubModule -- HwCamSubModule[0].DevName: imx290 2-0032
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwPrintModuleDefinition -- Name: imx290_bottom_DCTAFC
NvPclHwPrintModuleDefinition -- Position: 0
NvPclHwPrintModuleDefinition -- CalibrationData Found: 1
NvPclHwPrintCameraSubModule -- HwCamSubModule[0].Name: v4l2_sensor
NvPclHwPrintCameraSubModule -- HwCamSubModule[0].DevName: imx290 2-0031
NvPclHwGetModuleList: OFParserListModules Succeeded
NvPclModuleListInitialize: NvPclModule list[0]: imx290_front_DCTAFC position1
NvPclModuleListInitialize: NvPclModule list[1]: imx290_bottom_DCTAFC position0
NvPclHwScanExternalCameras -- adding video1 to discover list
NvPclHwScanExternalCameras -- adding video0 to discover list
NvPclHwScanExternalCameras -- adding video1 to discover list
NvPclHwScanExternalCameras -- adding video0 to discover list
initialize: /dev/video0
initialize: /dev/video1
NvPclHwScanExternalCameras -- adding video1 to discover list
NvPclHwScanExternalCameras -- adding video0 to discover list
initialize: /dev/video0
initialize: /dev/video1
NvPclHwScanExternalCameras: Failed to scan directory /sys/bus/pci_express/devices with error No such file or directory
NvPclHwGetPlugableModuleList: Failure ---------------------
NvPclModuleListInitialize: Failed getting plugable modules
getHotplugMonitor: Getting hotplug monitor instance
 initializeHotplug++
 hotPlugfunc ++ 
 addWatch: Watch added wd='1'
CheckProcDTExists: INFO: accessing /proc/device-tree/tegra-virtual-camera-platform/modules; No such file or directory
PCLHW_DTParser
setHotplugCallback: Registered new callback client
NvPclSetHotplugCallback: ----------------------
NvPclOpen: ++++++++++++++++++++++
NvPclStateControllerOpen: Found GUID 1 match at index[0]
NvPclHwInitializeModule: allocate overrides pathname @ 0x01059f10
NvPclHwInitializeModule: allocate overrides pathname @ 0x0104a290
NvPclHwInitializeModule: allocate overrides pathname @ 0x0104a320
NvPclHwInitializeModule: allocate overrides pathname @ 0x0104a3b0
NvPclHwInitializeModule: allocate overrides pathname @ 0x0105a1e0
NvPclHwInitializeModule: allocate overrides pathname @ 0x0105a270
LoadOverridesFile: looking for override file [/Calib/camera_override.isp] 1/16CheckOverridesPermissions: cannot stat file [/Calib/camera_override.isp]
LoadOverridesFile: looking for override file [/data/nvcam/settings/camera_overrides.isp] 2/16CheckOverridesPermissions: cannot stat file [/data/nvcam/settings/camera_overrides.isp]
LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/camera_overrides.isp] 3/16CheckOverridesPermissions: cannot stat file [/opt/nvidia/nvcam/settings/camera_overrides.isp]
LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/camera_overrides.isp] 4/16CheckOverridesPermissions: override file [/var/nvidia/nvcam/settings/camera_overrides.isp] has non-root:root owner or wrong permissions
LoadOverridesFile: looking for override file [/data/nvcam/camera_overrides.isp] 5/16CheckOverridesPermissions: cannot stat file [/data/nvcam/camera_overrides.isp]
LoadOverridesFile: looking for override file [/data/nvcam/settings/imx290_front_DCTAFC.isp] 6/16CheckOverridesPermissions: cannot stat file [/data/nvcam/settings/imx290_front_DCTAFC.isp]
LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/imx290_front_DCTAFC.isp] 7/16CheckOverridesPermissions: cannot stat file [/opt/nvidia/nvcam/settings/imx290_front_DCTAFC.isp]
LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/imx290_front_DCTAFC.isp] 8/16CheckOverridesPermissions: cannot stat file [/var/nvidia/nvcam/settings/imx290_front_DCTAFC.isp]
---- imager: No override file found. ----
Imager: looking for override file [/mnt/factory/camera/factory.bin] 1/16
Imager: looking for override file [/Calib/factory.bin] 2/16
Imager: looking for override file [/Calib/calibration.bin] 3/16
Imager: looking for override file [(null)] 4/16
Imager: looking for override file [(null)] 5/16
Imager: looking for override file [(null)] 6/16
Imager: looking for override file [(null)] 7/16
Imager: looking for override file [(null)] 8/16
Imager: looking for override file [(null)] 9/16
Imager: looking for override file [(null)] 10/16
Imager: looking for override file [(null)] 11/16
Imager: looking for override file [(null)] 12/16
Imager: looking for override file [(null)] 13/16
Imager: looking for override file [(null)] 14/16
Imager: looking for override file [(null)] 15/16
Imager: looking for override file [(null)] 16/16
Imager: looking for override file [/data/nvcam/settings/factory.bin] 1/16
Imager: looking for override file [/data/nvcam/settings/imx290_front_DCTAFC.bin] 2/16
Imager: looking for override file [/opt/nvidia/nvcam/settings/imx290_front_DCTAFC.bin] 3/16
Imager: looking for override file [/var/nvidia/nvcam/settings/imx290_front_DCTAFC.bin] 4/16
Imager: looking for override file [(null)] 5/16
Imager: looking for override file [(null)] 6/16
Imager: looking for override file [(null)] 7/16
Imager: looking for override file [(null)] 8/16
Imager: looking for override file [(null)] 9/16
Imager: looking for override file [(null)] 10/16
Imager: looking for override file [(null)] 11/16
Imager: looking for override file [(null)] 12/16
Imager: looking for override file [(null)] 13/16
Imager: looking for override file [(null)] 14/16
Imager: looking for override file [(null)] 15/16
Imager: looking for override file [(null)] 16/16
NvPclCreateDriver: Found NvPcl Driver Hal dev_name match (v4l2_sensor)
NvPclCreateDriver: Found a Driver name match (v4l2_sensor)
NvPclConnectDrivers: hImager was NULL, creating new imager
NvPclInitializeDrivers: v4l2_sensor ++++++++++++++++++
OFDPropertyGetString: could not read property [devnode-bus]
initialize: imx290 2-0032
OFDPropertyGetString: could not read property [delayed_gain]
OFDPropertyCopyToLong: could not read property [has-eeprom]
Control OTP Data not found
Control Frame Length not found
Control Coarse Time not found
Control Coarse Time Short not found
loadBinaryBlob: Binary read 0
OFDPropertyGetString: could not read property [mode0.x_start]
OFDPropertyGetString: could not read property [mode0.y_start]
OFDPropertyGetString: could not read property [mode0.x_end]
OFDPropertyGetString: could not read property [mode0.y_end]
OFDPropertyGetString: could not read property [mode0.h_scaling]
OFDPropertyGetString: could not read property [mode0.v_scaling]
initialize: Loaded Driver: 1 Modes Available--------------
NvPclInitializeDrivers: v4l2_sensor ------------------
NvPclOpen: ----------------------
LSC: LSC surface is not based on full res!
NvPclOpen: ++++++++++++++++++++++
NvPclStateControllerOpen: Found GUID 0 match at index[1]
NvPclHwInitializeModule: allocate overrides pathname @ 0x01052460
NvPclHwInitializeModule: allocate overrides pathname @ 0x010524f0
NvPclHwInitializeModule: allocate overrides pathname @ 0x01052580
NvPclHwInitializeModule: allocate overrides pathname @ 0x01052610
NvPclHwInitializeModule: allocate overrides pathname @ 0x010526a0
NvPclHwInitializeModule: allocate overrides pathname @ 0x01052730
LoadOverridesFile: looking for override file [/Calib/camera_override.isp] 1/16CheckOverridesPermissions: cannot stat file [/Calib/camera_override.isp]
LoadOverridesFile: looking for override file [/data/nvcam/settings/camera_overrides.isp] 2/16CheckOverridesPermissions: cannot stat file [/data/nvcam/settings/camera_overrides.isp]
LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/camera_overrides.isp] 3/16CheckOverridesPermissions: cannot stat file [/opt/nvidia/nvcam/settings/camera_overrides.isp]
LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/camera_overrides.isp] 4/16CheckOverridesPermissions: override file [/var/nvidia/nvcam/settings/camera_overrides.isp] has non-root:root owner or wrong permissions
LoadOverridesFile: looking for override file [/data/nvcam/camera_overrides.isp] 5/16CheckOverridesPermissions: cannot stat file [/data/nvcam/camera_overrides.isp]
LoadOverridesFile: looking for override file [/data/nvcam/settings/imx290_bottom_DCTAFC.isp] 6/16CheckOverridesPermissions: cannot stat file [/data/nvcam/settings/imx290_bottom_DCTAFC.isp]
LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/imx290_bottom_DCTAFC.isp] 7/16CheckOverridesPermissions: cannot stat file [/opt/nvidia/nvcam/settings/imx290_bottom_DCTAFC.isp]
LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/imx290_bottom_DCTAFC.isp] 8/16CheckOverridesPermissions: cannot stat file [/var/nvidia/nvcam/settings/imx290_bottom_DCTAFC.isp]
---- imager: No override file found. ----
Imager: looking for override file [/mnt/factory/camera/factory.bin] 1/16
Imager: looking for override file [/Calib/factory.bin] 2/16
Imager: looking for override file [/Calib/calibration.bin] 3/16
Imager: looking for override file [(null)] 4/16
Imager: looking for override file [(null)] 5/16
Imager: looking for override file [(null)] 6/16
Imager: looking for override file [(null)] 7/16
Imager: looking for override file [(null)] 8/16
Imager: looking for override file [(null)] 9/16
Imager: looking for override file [(null)] 10/16
Imager: looking for override file [(null)] 11/16
Imager: looking for override file [(null)] 12/16
Imager: looking for override file [(null)] 13/16
Imager: looking for override file [(null)] 14/16
Imager: looking for override file [(null)] 15/16
Imager: looking for override file [(null)] 16/16
Imager: looking for override file [/data/nvcam/settings/factory.bin] 1/16
Imager: looking for override file [/data/nvcam/settings/imx290_bottom_DCTAFC.bin] 2/16
Imager: looking for override file [/opt/nvidia/nvcam/settings/imx290_bottom_DCTAFC.bin] 3/16
Imager: looking for override file [/var/nvidia/nvcam/settings/imx290_bottom_DCTAFC.bin] 4/16
Imager: looking for override file [(null)] 5/16
Imager: looking for override file [(null)] 6/16
Imager: looking for override file [(null)] 7/16
Imager: looking for override file [(null)] 8/16
Imager: looking for override file [(null)] 9/16
Imager: looking for override file [(null)] 10/16
Imager: looking for override file [(null)] 11/16
Imager: looking for override file [(null)] 12/16
Imager: looking for override file [(null)] 13/16
Imager: looking for override file [(null)] 14/16
Imager: looking for override file [(null)] 15/16
Imager: looking for override file [(null)] 16/16
NvPclCreateDriver: Found NvPcl Driver Hal dev_name match (v4l2_sensor)
NvPclCreateDriver: Found a Driver name match (v4l2_sensor)
NvPclConnectDrivers: hImager was NULL, creating new imager
NvPclInitializeDrivers: v4l2_sensor ++++++++++++++++++
OFDPropertyGetString: could not read property [devnode-bus]
initialize: imx290 2-0031
OFDPropertyGetString: could not read property [delayed_gain]
OFDPropertyCopyToLong: could not read property [has-eeprom]
Control OTP Data not found
Control Frame Length not found
Control Coarse Time not found
Control Coarse Time Short not found
loadBinaryBlob: Binary read 0
OFDPropertyGetString: could not read property [mode0.x_start]
OFDPropertyGetString: could not read property [mode0.y_start]
OFDPropertyGetString: could not read property [mode0.x_end]
OFDPropertyGetString: could not read property [mode0.y_end]
OFDPropertyGetString: could not read property [mode0.h_scaling]
OFDPropertyGetString: could not read property [mode0.v_scaling]
initialize: Loaded Driver: 1 Modes Available--------------
NvPclInitializeDrivers: v4l2_sensor ------------------
NvPclOpen: ----------------------
LSC: LSC surface is not based on full res!
Camera devices found: 0

First strange thing is the line

... has non-root:root owner or wrong permissions

This file exists and has user/group set as root:root. Permissions 0664, 0644, 0444 and 0400 all don’t work, and I don’t know what’s expected here. Could you please clarify?

Next thing is that, apparently, two sensors are successfully loaded:

imx290_front_DCTAFC => initialize: Loaded Driver: 1 Modes Available
imx290_bottom_DCTAFC => initialize: Loaded Driver: 1 Modes Available

However, ICameraProvider::getCameraDevices() returns an empty list:

Camera devices found: 0

Can you see from the log what is wong?

linuxdev · May 6, 2020, 5:32pm

I know nothing about your particular case, but does “sudo ls” work?

JerryChang · February 29, 2020, 1:19am

hello stefan.zegenhagen,

there’s sensor probing during kernel initialization. please check did video node generated under /dev/ sysnode.
could you also try to launch camera sensor with v4l2-ctl, it’s another path to access the sensor driver.
for example.

v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=10 --stream-to=video0.raw

stefan.zegenhagen · February 1, 2020, 6:14am

@JerryChang:

The device nodes are created and I can capture with v4l2-ctl. The log clearly shows that libargus’ PCL layer successfully parsed the DTB entries and found /dev/video0 as well as /dev/video1.

Can you see anything from the above log that makes libargus reject the available camera devices?

JerryChang · February 29, 2020, 1:19am

hello stefan.zegenhagen,

need your confirmation about accessing the camera device with gstreamer,
are you able to launch camera to preview with below commands,

gst-launch-1.0 nvcamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1920, height=1080, framerate=30/1, format=NV12' ! nvoverlaysink -ev

also, there’s no much details about the failure with argus.
did you saw any suspect failure from kernel message?
could you please kill the argus service and launch the argus_deamon manually to enable background prints, thanks

ps aux | grep argus_daemon 
sudo kill # 
sudo /usr/sbin/argus_daemon

stefan.zegenhagen · February 1, 2020, 6:15am

@JerryChang:

Found the problem, the application was compiled against outdated libargus header files.

This indicates, however, a significant failure in the Tegra Multimedia API. Please keep in mind that I’m a programmer myself and can estimate the amount of failure here.

The API uses objects without functionality themselfs. Instead, the object must be queried for a certain (virtual) interface that implements the functionality. The wanted interface is identified by a UUID. So far so good.

What the Tegra Multimedia API implementation completely ignores is that such an interface (which is an ABI object as opposed to an API object, since the library is distributed in binary-only form) must also be versioned using the UUID. That means, whenever you add or remove virtual methods from such an interface specification, or just the calling conventions of an existing method, you must assign a new UUID to the new interface.

As in our case: we compiled against outdated libargus headers and linked against the new libargus library. The UUID for ICameraProvider has not changed, but the argus library has added 2 methods (getVersion() and getVendor()) to the virtual function table (VFT). This causes the compiler to layout the VFT completely different and makes the ICameraProvider interface binary incompatible. Our application ended up calling the wrong function from the virtual function table: everything could have happened, from a simple application crash to a complete system hang!

The correct way would have been to create a new version of ICameraProvider with a distinct UUID at the time the new functions were added. That is what the UUIDs are for in the first place, the only reason for using UUIDs at all! Applications compiling against the new headers would find a different UUID than applications compiled against older header files, and automatically use the newer interface version. It is a matter of taste if one keeps old versions of the interfaces supported for backward compatibility or removes them and prints a readable warning about the version mismatch.