Hello, currently facing an issue using nvargus-daemon along with nvarguscamerasrc.
My system is using Buildroot but is based on Jetpack L4T 32.7.3. I have added all of the libraries and binaries from nvidia-l4t-camera, nvidia-l4t-gstreamer, nvidia-l4t-core, nvidia-l4t-multimedia (Index).
I’m experiencing a weird issue where I run nvargus-daemon along with a simple gst-inspect-1.0 nvarguscamerasrc and I then see that nvargus-daemon hangs when it should be traversing the device tree entry etc.
I’m using an IMX219 RPI V2 camera and have tested using v4l2-ctl and I don’t have any issues grabbing frames so I think all is ok with the sensor integration itself.
Here are the snippets of what is happening;
root [ ~ ]$ gst-inspect-1.0 nvarguscamerasrc
(Argus) Error Timeout: (propagating from src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 137)
(Argus) Error Timeout: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
root [ ~ ]$ nvargus-daemon
=== NVIDIA Libargus Camera Service (0.98.3)=== Listening for connections...=== gst-inspect-1.0[2610]: Connection established (7F8769F160)
As you can see, a connection is established with the nvargus socket server from gstreamer but then it just hangs and eventually I get a timeout error on the gstreamer side.
Hi,
RPI camera V2 is supported on Jetson Nano developer kit with standard Jetpack release. Please set up Jetson Nano developer kit + RPI camera V2 and flash standard image. To make sure it works on this setup first.
Hi,
It looks like there is something wrong in device tree. Please run xxd command after booting, to check whether the device tree is correct. The reference device tree is
I looked at the link you posted. My sensor works fine, I have validated this with v4l2-ctl;
root [ / ]$ v4l2-ctl --stream-mmap
<<<<<<<<<<<<<<<<<<<<<<< 21.19 fps
<<<<<<<<<<<<<<<<<<<<< 21.19 fps
<<<<<<<<<<<<<<<<<<<<< 21.19 fps
<<<<<<<<<<<<<<<<<<<<< 21.19 fps
<<<<<<<<<<<<<<<<<<<<< 21.19 fps
<<<<<<<<<<<<<<<<<<<<<< 21.19 fps
<<<<<<<<<<<<<<<<<<<<< 21.19 fps
<<<<<<<<<<<<<<<<<<<<< 21.19 fps
<<<<<<<<<<<<<<<<<<<<< 21.19 fps
<<<<<<<<<<<<<<<<<<<<< 21.19 fps
<<<<<<<<<<<<<<<<<<<<<< 21.19 fps
root [ / ]$ dmesg | grep -iE 'imx219|cam'
[ 0.556683] GPIO line 151 (camera-control-output-low) hogged as output/low
[ 0.556717] GPIO line 152 (camera-control-output-low) hogged as output/low
[ 0.636200] camchar: rtcpu character device driver loaded
[ 0.758211] tegra_camera_platform tegra-camera-platform: tegra_camera_probe:camera_platform_driver probe
[ 0.758436] misc tegra_camera_ctrl: tegra_camera_isomgr_register isp_iso_bw=1500000, vi_iso_bw=1500000, max_bw=1500000
[ 4.354937] i2c-mux-gpio cam_i2cmux: 1 port mux on Tegra I2C adapter adapter
[ 4.355515] imx219 7-0010: tegracam sensor driver:imx219_v2.0.6
[ 4.491585] vi 54080000.vi: subdev imx219 7-0010 bound
root [ / ]$ v4l2-ctl --list-devices
vi-output, imx219 7-0010 (platform:54080000.vi:0):
/dev/video0
NVIDIA Tegra Video Input Device (platform:vi):
/dev/media0
root [ / ]$ media-ctl -p
Media controller API version 0.1.0
Media device information
------------------------
driver vi
model NVIDIA Tegra Video Input Device
serial
bus info
hw revision 0x3
driver version 0.0.0
Device topology
- entity 1: nvcsi--1 (2 pads, 2 links)
type V4L2 subdev subtype Unknown flags 0
device node name /dev/v4l-subdev0
pad0: Sink
<- "imx219 7-0010":0 [ENABLED]
pad1: Source
-> "vi-output, imx219 7-0010":0 [ENABLED]
- entity 4: imx219 7-0010 (1 pad, 1 link)
type V4L2 subdev subtype Sensor flags 0
device node name /dev/v4l-subdev1
pad0: Source
[fmt:SRGGB10_1X10/3264x2464 field:none colorspace:srgb]
-> "nvcsi--1":0 [ENABLED]
- entity 6: vi-output, imx219 7-0010 (1 pad, 1 link)
type Node subtype V4L flags 0
device node name /dev/video0
pad0: Sink
<- "nvcsi--1":1 [ENABLED]
So the issue seems to be purely with nvargus-daemon here.
I have even put my custom device tree on the devkit and nvargus-daemon works ok.
I thought it might have been an RPC issue so I wrote a test server and client to ensure RPC messages are getting across and it worked fine.
I have checked all of the dependencies with ldd, nothing is missing. I have run the daemon with LD_DEBUG=all, I have run with strace as well but nothing seems to come up with an error.
Are there any other ways to debug nvargus?
I am out of ideas to be honest! so anything you can suggest would be helpful.
Hi,
If the custom board is designed same as developer kit, and the same dtb works on developer kit, it looks to be an issue in signal quality. The signal may be worse on the custom board. Please adjust settle time on the custom board and see if you can find a good setting.
One difference between v4l2 command and Argus is auto exposure. Argus runs auto exposure and adjusts exposure time continuously, if the exposure time control is set and the received frame length is not expected, it may gets stuck. For further debugging, please add prints in exposure time control to check v4l2 command and Argus.
I understand your point … but from what I can see, even if no camera is connected we should get some output from nvargus-daemon even if it is just errors.
On my devkit, I have no camera connected and if I run nvargus-daemon I can see this; (when running nvargus_nvraw from another terminal)
nvidia@nvidia-desktop: $ nvargus-daemon
=== NVIDIA Libargus Camera Service (0.98.3)=== Listening for connections...=== nvargus_nvraw[8719]: Connection established (7F76E021D0)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
---- imager: No override file found. ----
(NvCamV4l2) Error ModuleNotPresent: V4L2Device not available (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function findDevice(), line 256)
(NvCamV4l2) Error ModuleNotPresent: (propagating from /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function initialize(), line 60)
(NvOdmDevice) Error ModuleNotPresent: (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function initialize(), line 107)
NvPclDriverInitializeData: Unable to initialize driver v4l2_sensor
NvPclInitializeDrivers: error: Failed to init camera sub module v4l2_sensor
NvPclStartPlatformDrivers: Failed to start module drivers
NvPclStateControllerOpen: Failed ImagerGUID 1. (error 0xA000E)
NvPclOpen: PCL Open Failed. Error: 0xf
SCF: Error BadParameter: Sensor could not be opened. (in src/services/capture/CaptureServiceDeviceSensor.cpp, function getSourceFromGuid(), line 593)
SCF: Error BadParameter: (propagating from src/services/capture/CaptureService.cpp, function addSourceByGuid(), line 437)
SCF: Error BadParameter: (propagating from src/api/CameraDriver.cpp, function addSourceByIndex(), line 305)
SCF: Error BadParameter: (propagating from src/api/CameraDriver.cpp, function getSource(), line 471)
Acquiring SCF Camera device source via index 0 has failed. === nvargus_nvraw[8719]: CameraProvider initialized (0x7f7094ded0)=== nvargus_nvraw[8719]: CameraProvider destroyed (0x7f7094ded0)=== nvargus_nvraw[8719]: Connection closed (7F76E021D0)=== nvargus_nvraw[8719]: Connection cleaned up (7F76E021D0)
On the devkit, we can see that the tegra-camera-platform is traversed and an initialisation is attempted. Whereas on my custom board, it simply stops at “Connection established” and then eventually times out.
So what needs to be understood, is why nvargus-daemon hangs after an RPC client such as nvargus_nvraw connects. There must be something wrong with the environment …
Hi,
It is more like an issue in hardware signal. Would suggest check the MIPI signal on the custom board. If the same Jetson Nano module works on developer kit and does not work on the custom board, the deviation looks to be signal quality. Software looks ready since the same dtb works on the developer kit.
Hi @DaneLLL , I was able to flash jetpack onto our custom board (not everything works but provides enough of a baseline for me to test nvargus-daemon) - it worked fine so it is not a hardware issue.
I will do some direct copy and comparison between the libraries on Jetpack RFS vs my custom Buildroot file system and see how it goes.
Hi @DaneLLL , I tried this on a similar system of ours using the Orin NX - which is based on a similar Buildroot based build.
What I noticed is that this actually fails in the same way, but gives a bit more information than the Jetson Nano does …
SCF: Error ResourceError: Unable to open BW Ioctl FD (in src/services/power/PowerServiceCore.cpp, function initialize(), line 125)
SCF: Error ResourceError: (propagating from src/services/power/PowerService.cpp, function startService(), line 52)
SCF: Error ResourceError: (propagating from src/components/ServiceHost.cpp, function startServices(), line 162)
This happens when the worker thread “worker thread PS handleRequests start” should be output.
I looked around and couldn’t figure out what this error meant. Clearly related to some sort of power control but it’s not clear.
Unable to open BW Ioctl FD
Suggests there is some sort of dev node that is attempted to be opened but cannot be reached. This is an indicator as to what is missing at least.
Hi,
Jetson Nano uses Jetpack 4 and Orin NX uses Jetpack 5 and 6. The software stacks are different in the releases, so it is not same conditions. Seems like you have your own rootfs. You may try default rootfs(with Ubuntu desktop) and see if the issue is present.
For testing more on Orin NX, you can also try this rootfs:
Yes I am aware that there are major differences and I have already tried simply copying the whole /usr/lib/aarch64-linux-gnu from Jetpack 4 to the Buildroot build on Jetson Nano and it works. But there are so many libraries in there it’s difficult to pinpoint which one is causing issues.
I would not say it’s not the same conditions. See the snippet below, this is the Jetson Nano Buildroot build which hangs just before worker thread PS handleRequests start is output. I know this is the next line because of my reference devkit running Jetpack.
On the Orin NX Buildroot build I have, this errors in exactly the same place except it prints some error messages. Suggesting to me that the code has been improved to print the errors in Jetpack 5/6.
So I really just want to know what Unable to open BW Ioctl FD means and what library or service or devnode it relates to. Please can you give more information about it?
Hi,
It is the device node for controlling the bandwidth:
/dev/tegra_camera_ctrl
To make sure the bandwidth of hardware engines is sufficient for the camera task. It should be same print for Jetpack 4, but you only see it on Jetpack 5. The condition looks different on Jetpack 4/Jetson Nano and Jetpack 5/Orin NX.