Intermittent Capture Failure on custom carrier board

[akhil.m.birlangi]

Hi,

We have a custom carrier board that we are using to capture from 4 IMX568 sensors via gstreamer. Occasionally when bringing up the 4 camera streams, one of the cameras fails to come online with the following error:

nvbuf_utils: dmabuf_fd -1 mapped entry NOT found

Looking at the nvargus-daemon logs we see an error regarding nvphsd

The error looks as follows:

Jun 13 22:57:57 looq nvargus-daemon[1790]: ---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ---- 
Jun 13 22:57:57 looq nvargus-daemon[1790]: CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res! 
Jun 13 22:57:57 looq nvargus-daemon[1790]: E/ libnvphs:socket: Error[2]: socket connection /var/lib/nvphs/nvphsd.ctl to PHS failed: No such file or directory 
Jun 13 22:57:57 looq nvargus-daemon[1790]: D/ libnvphs:socket: Warning: connecting to Power Hinting Service failed. Is PHS running? 
Jun 13 22:57:57 looq nvargus-daemon[1790]: === gst-launch-1.0
[1828]: CameraProvider initialized (0xffff7c5913a0)=== gst-launch-1.0
[1827]: CameraProvider initialized (0xffff74000c20)=== gst-launch-1.0
[1829]: CameraProvider initialized (0xffff6c000c20)=== gst-launch-1.0
[1830]: CameraProvider initialized (0xffff64000c20)SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)

This forum post suggested we disable the NVPHS service with has reduced the frequency of our error but not eliminated it.

https://forums.developer.nvidia.com/t/nvphs-error-on-custom-carrier-board-without-display/233441

Please let us know what information to provide to help debug this issue.

[JerryChang]

hello akhil.m.birlangi,

is this always the same camera device cannot output frames? or, it’s randomly?

may I confirm the Jetpack, L4T release version you’re now working with, you may check release tag, $ cat /etc/nv_tegra_release for confirmation.

assume you’re using rel-35 code-line, you could ignore NvPHSSendThroughputHints related messages since it should not effect camera use-case.

and…
please check developer guide, Applications Using V4L2 IOCTL Directly for using V4L2 IOCTL to verify basic camera functionality.

[akhil.m.birlangi]

Hi Jerry,

We are currently using release 35.1.0 and have seen this issue on previous versions as well.
The issue is intermittent and seems to occur randomly and not on the same device. We also have not encountered any issues with using V4L2 in trying to bring up a camera stream.

Thanks,
Akhil

[ShaneCCC]

Only failed while run 4 cameras simultaneously?
Try boost clocks to check if help on it.

sudo su
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee  /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate

[akhil.m.birlangi]

Hi Shane,

The clock boosting is currently used in our capture stack. We are still testing to reproduce the issue. It is very intermittent and mostly seems to occur when we try to start all 4 cameras simultaneously.

Is there anything we should use to capture diagnostic information when we encounter another failure?

Thanks,
Akhil