Camera disconnection in nvargus-daemon service

Robotics & Edge Computing Jetson Systems Jetson AGX Xavier
1

Hello, NV team

We encountered the problem of failure to obtain the camera video stream, the specific situation of synchronization, and the problem that may require nvidia support. The following describes the basic situation of the problem:
Platform: Jetson AGX Xavier
System version:32.7.2
Jetpack version:4.6
Number of cameras: 4
Camera sensor: imx334
Software solution: Use the software solution provided by NV, based on nvargus-daemon

Fault information introduction:
Basic description: The nargus-daemon reports an error when the camera reads the video stream, and the acquieBuffer interface returns an error (STATUS_TIMEOUT), causing the CV function to be abnormally interrupted.
Probability: It cannot be reproduced alone. With 200 cars, one car can appear once a day.
syslog and kern.log collection: refer to the attached compressed package (the internal picture is the fault abnormal time, and the abnormal print can be found in syslog through the time. The details can be explained in further detail).
Recovery method: Restart nvargus-daemon and private applications and then recover.

Need support:

  1. The nvargus-daemon source code nvidia is not open source. I saw some error messages in the log, but I cannot continue to troubleshoot, and there is no way to reproduce the abnormal phenomenon. I hope NV can provide troubleshooting ideas for this problem.
2

P_KRO_KRO_336_M3_004.zip (13.0 MB)
CAM3STMI011MI23N0209.zip (4.1 MB)
P_KRO_KRO_421_M3_010.zip (8.2 MB)
P_WF_SHOPRITE_199_M3_010A.zip (17.8 MB)

4

Get the daemon log and kernel log.

https://elinux.org/Jetson/l4t/Camera_BringUp#Steps_to_enable_more_debug_messages

5

We did not see kernel printing errors, only nvargus-daemon reported errors. Can you analyze the logs I uploaded?

6

Please get the nvargus-daemon log only from the link.

7

I execute the following command:
echo 2 > /sys/kernel/debug/camrtc/log-level
AcquireBuffer will appear 100%

May 13 12:02:03 caper-desktop kernel: [ 405.079094] rtcpu_trace_isp_event event id 20 cannot be found
May 13 12:02:03 caper-desktop kernel: [ 405.085009] rtcpu_trace_isp_event event id 20 cannot be found
May 13 12:02:03 caper-desktop kernel: [ 405.090803] rtcpu_trace_isp_event event id 20 cannot be found
May 13 12:02:03 caper-desktop kernel: [ 405.096306] rtcpu_trace_isp_event event id 20 cannot be found
May 13 12:02:03 caper-desktop nvargus-daemon[15336]: message repeated 3 times: [ (Argus) Error Timeout: (propagating from src/api/BufferOutputStreamImpl.cpp, function acquireBuffer(), line 265)]
May 13 12:02:03 caper-desktop nvargus-daemon: W/ libnvphs:socket: Warning (NvPHSSendThroughputHints): too many throughput hints given: buffer full, hints not sent!
May 13 12:02:03 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CpuFloorVmin, value=0, timeout_ms=1000]: queue_or_send() failed
May 13 12:02:03 caper-desktop nvargus-daemon[15336]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
May 13 12:02:03 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CameraFrameTimestamp, value=750398572, timeout_ms=1000]: queue_or_send() failed
May 13 12:02:03 caper-desktop nvargus-daemon[15336]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
May 13 12:02:03 caper-desktop kernel: [ 405.102213] rtcpu_trace_isp_event event id 20 cannot be found
May 13 12:02:03 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=FramerateTarget, value=30, timeout_ms=1000]: queue_or_send() failed
May 13 12:02:03 caper-desktop nvargus-daemon[15336]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
May 13 12:02:03 caper-desktop nvargus-daemon[15336]: message repeated 2 times: [ SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)]
May 13 12:02:03 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CpuFloorVmin, value=0, timeout_ms=1000]: queue_or_send() failed
May 13 12:02:03 caper-desktop nvargus-daemon[15336]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
May 13 12:02:03 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=FramerateTarget, value=30, timeout_ms=1000]: queue_or_send() failed
May 13 12:02:03 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CpuFloorVmin, value=0, timeout_ms=1000]: queue_or_send() failed
May 13 12:02:03 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CameraFrameTimestamp, value=750660722, timeout_ms=1000]: queue_or_send() failed
May 13 12:02:03 caper-desktop nvargus-daemon[15336]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
May 13 12:02:03 caper-desktop kernel: [ 405.107960] rtcpu_trace_isp_event event id 20 cannot be found
May 13 12:02:03 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CameraFrameTimestamp, value=751053938, timeout_ms=1000]: queue_or_send() failed
May 13 12:02:03 caper-desktop nvargus-daemon[15336]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
May 13 12:02:03 caper-desktop kernel: [ 405.113913] rtcpu_trace_isp_event event id 20 cannot be found
May 13 12:02:03 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=FramerateTarget, value=30, timeout_ms=1000]: queue_or_send() failed
May 13 12:02:03 caper-desktop nvargus-daemon[15336]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
May 13 12:02:03 caper-desktop nvargus-daemon[15336]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
May 13 12:02:03 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CpuFloorVmin, value=0, timeout_ms=1000]: queue_or_send() failed
May 13 12:02:03 caper-desktop nvargus-daemon[15336]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
May 13 12:02:03 caper-desktop nvargus-daemon[15336]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
May 13 12:02:03 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=FramerateTarget, value=30, timeout_ms=1000]: queue_or_send() failed
May 13 12:02:03 caper-desktop nvargus-daemon[15336]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
May 13 12:02:03 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CpuFloorVmin, value=0, timeout_ms=1000]: queue_or_send() failed
May 13 12:02:03 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CameraFrameTimestamp, value=751447160, timeout_ms=1000]: queue_or_send() failed

8

Which command can obtain the logs of nvargus-daemon alone?

10

Sorry, not 100% present 111

11

Do you know what this passage means? nvargus-daemon will print this during the capture process, but the acquireBuffer timeout does not appear directly.

May 13 12:37:15 caper-desktop nvargus-daemon: W/ libnvphs:socket: Warning (NvPHSSendThroughputHints): too many throughput hints given: buffer full, hints not sent!
May 13 12:37:15 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CameraFrameTimestamp, value=134481879, timeout_ms=1000]: queue_or_send() failed
May 13 12:37:15 caper-desktop nvargus-daemon[23631]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
May 13 12:37:15 caper-desktop nvargus-daemon[23631]: message repeated 6 times: [ SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)]
May 13 12:37:15 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=FramerateTarget, value=30, timeout_ms=1000]: queue_or_send() failed
May 13 12:37:15 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CpuFloorVmin, value=0, timeout_ms=1000]: queue_or_send() failed
May 13 12:37:15 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=FramerateTarget, value=30, timeout_ms=1000]: queue_or_send() failed
May 13 12:37:15 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CpuFloorVmin, value=0, timeout_ms=1000]: queue_or_send() failed
May 13 12:37:15 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=FramerateTarget, value=30, timeout_ms=1000]: queue_or_send() failed
May 13 12:37:15 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CpuFloorVmin, value=0, timeout_ms=1000]: queue_or_send() failed
May 13 12:37:15 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=FramerateTarget, value=30, timeout_ms=1000]: queue_or_send() failed
May 13 12:37:15 caper-desktop nvargus-daemon[23631]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
May 13 12:37:15 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CpuFloorVmin, value=0, timeout_ms=1000]: queue_or_send() failed
May 13 12:37:15 caper-desktop nvargus-daemon[23631]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
May 13 12:37:15 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CameraFrameTimestamp, value=135202786, timeout_ms=1000]: queue_or_send() failed
May 13 12:37:15 caper-desktop nvargus-daemon[23631]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
May 13 12:37:15 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=FramerateTarget, value=30, timeout_ms=1000]: queue_or_send() failed
May 13 12:37:15 caper-desktop nvargus-daemon[23631]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
May 13 12:37:15 caper-desktop nvargus-daemon[23631]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
May 13 12:37:15 caper-desktop nvargus-daemon: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CpuFloorVmin, value=0, timeout_ms=1000]: queue_or_send() failed

12

Run below command before running camera pipeline.

journalctl -u nvargus-daemon -f
13

The probability of this problem recurring is very low. Can you tell anything from the logs I uploaded?

14

5月 13 14:51:16 caper-desktop nvargus-daemon[6806]: W/ libnvphs:socket: Warning (NvPHSSendThroughputHints): too many throughput hints given: buffer full, hints not sent!
5月 13 14:51:16 caper-desktop nvargus-daemon[6806]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=FramerateTarget, value=30, timeout_ms=1000]: queue_or_send() failed
5月 13 14:51:16 caper-desktop nvargus-daemon[6806]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
5月 13 14:51:16 caper-desktop nvargus-daemon[6806]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CpuFloorVmin, value=0, timeout_ms=1000]: queue_or_send() failed
5月 13 14:51:16 caper-desktop nvargus-daemon[6806]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
5月 13 14:51:16 caper-desktop nvargus-daemon[6806]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CameraFrameTimestamp, value=1898475784, timeout_ms=1000]: queue_or_send() failed
5月 13 14:51:16 caper-desktop nvargus-daemon[6806]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=FramerateTarget, value=30, timeout_ms=1000]: queue_or_send() failed
5月 13 14:51:16 caper-desktop nvargus-daemon[6806]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CpuFloorVmin, value=0, timeout_ms=1000]: queue_or_send() failed
5月 13 14:51:16 caper-desktop nvargus-daemon[6806]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CameraFrameTimestamp, value=1898606853, timeout_ms=1000]: queue_or_send() failed
5月 13 14:51:16 caper-desktop nvargus-daemon[6806]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=FramerateTarget, value=30, timeout_ms=1000]: queue_or_send() failed
5月 13 14:51:16 caper-desktop nvargus-daemon[6806]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
5月 13 14:51:16 caper-desktop nvargus-daemon[6806]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CpuFloorVmin, value=0, timeout_ms=1000]: queue_or_send() failed
5月 13 14:51:16 caper-desktop nvargus-daemon[6806]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
5月 13 14:51:16 caper-desktop nvargus-daemon[6806]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=FramerateTarget, value=30, timeout_ms=1000]: queue_or_send() failed
5月 13 14:51:16 caper-desktop nvargus-daemon[6806]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CpuFloorVmin, value=0, timeout_ms=1000]: queue_or_send() failed
5月 13 14:51:16 caper-desktop nvargus-daemon[6806]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
5月 13 14:51:16 caper-desktop nvargus-daemon[6806]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CameraFrameTimestamp, value=1899065611, timeout_ms=1000]: queue_or_send() failed
5月 13 14:51:16 caper-desktop nvargus-daemon[6806]: SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
5月 13 14:51:16 caper-desktop nvargus-daemon[6806]: E/ libnvphs: Error: NvPHSSendThroughputHints[usecase=camera, hint=CameraFrameTimestamp, value=1899131146, timeout_ms=1000]: queue_or_send() failed

15

The message relative for the power hit is known harmless message.

Thanks

16

You can find the time of failure from the compressed package I uploaded, and then check the error information at the time of failure through syslog.
Did you find anything?

17

Which messages?

Thanks

18

these logs.

19

image
image1434×235 23.7 KB

20

Sorry, I can’t help to filter those messages.
Please get the nvargus-daemon log for analysis.

Thanks

21

OK，
What is the reason for the following nvargus-daemon error?
What happened during the process of fetching the stream:
May 6 15:57:48 caper-desktop nvargus-daemon[16064]: (Argus) Error Timeout: (propagating from src/api/BufferOutputStreamImpl.cpp, function acquireBuffer(), line 265)

Related logs:
May 6 15:57:40 caper-desktop nvargus-daemon[16064]: (Argus) Error InvalidState: Buffer was already acquired (in src/api/BufferImpl.cpp, function bufferAcquired(), line 87)
May 6 15:57:40 caper-desktop nvargus-daemon[16064]: (Argus) Error InvalidState: Buffer was not acquired (in src/api/BufferImpl.cpp, function bufferReleased(), line 100)
May 6 15:57:40 caper-desktop nvargus-daemon[16064]: message repeated 2 times: [ (Argus) Error InvalidState: Buffer was not acquired (in src/api/BufferImpl.cpp, function bufferReleased(), line 100)]
May 6 15:57:41 caper-desktop nvargus-daemon[16064]: (Argus) Error InvalidState: Buffer was already acquired (in src/api/BufferImpl.cpp, function bufferAcquired(), line 87)
May 6 15:57:41 caper-desktop nvargus-daemon[16064]: (Argus) Error InvalidState: Buffer was not acquired (in src/api/BufferImpl.cpp, function bufferReleased(), line 100)
May 6 15:57:43 caper-desktop nvargus-daemon[16064]: SCF: Error InvalidState: Buffer already assigned to capture (in src/api/Buffer.cpp, function assignToCapture(), line 365)
May 6 15:57:43 caper-desktop nvargus-daemon[16064]: SCF: Error InvalidState: (propagating from src/components/CaptureContainerImpl.cpp, function assignBuffer(), line 182)
May 6 15:57:43 caper-desktop nvargus-daemon[16064]: SCF: Error InvalidState: (propagating from src/components/CaptureSetupEngine.cpp, function setupCC(), line 234)
May 6 15:57:43 caper-desktop nvargus-daemon[16064]: SCF: Error InvalidState: (propagating from src/api/Session.cpp, function capture(), line 822)
May 6 15:57:45 caper-desktop nvargus-daemon[16064]: SCF: Error Timeout: (propagating from src/api/Buffer.cpp, function waitForUnlock(), line 635)
May 6 15:57:45 caper-desktop nvargus-daemon[16064]: SCF: Error Timeout: (propagating from src/components/CaptureContainerImpl.cpp, function returnBuffer(), line 365)
May 6 15:57:45 caper-desktop nvargus-daemon[16064]: SCF: Error Timeout: (propagating from src/components/CaptureContainerImpl.cpp, function returnAllBuffers(), line 535)
May 6 15:57:45 caper-desktop nvargus-daemon[16064]: (Argus) Error InvalidState: (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)
May 6 15:57:45 caper-desktop nvargus-daemon[16064]: PowerServiceCore:handleRequests: timePassed = 1489
May 6 15:57:45 caper-desktop nvargus-daemon[16064]: SCF: Error InvalidState: Buffer already assigned to capture (in src/api/Buffer.cpp, function assignToCapture(), line 365)
May 6 15:57:45 caper-desktop nvargus-daemon[16064]: SCF: Error InvalidState: (propagating from src/components/CaptureContainerImpl.cpp, function assignBuffer(), line 182)
May 6 15:57:45 caper-desktop nvargus-daemon[16064]: SCF: Error InvalidState: (propagating from src/components/CaptureSetupEngine.cpp, function setupCC(), line 234)
May 6 15:57:45 caper-desktop nvargus-daemon[16064]: SCF: Error InvalidState: (propagating from src/api/Session.cpp, function capture(), line 822)
May 6 15:57:45 caper-desktop nvargus-daemon[16064]: (Argus) Error InvalidState: (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)
May 6 15:57:45 caper-desktop nvargus-daemon[16064]: (Argus) Error InvalidState: Buffer was already acquired (in src/api/BufferImpl.cpp, function bufferAcquired(), line 87)
May 6 15:57:45 caper-desktop nvargus-daemon[16064]: SCF: Error InvalidState: Buffer is not locked (in src/api/Buffer.cpp, function putOutputFences(), line 564)
May 6 15:57:45 caper-desktop nvargus-daemon[16064]: SCF: Error InvalidState: (propagating from src/components/stages/CaptureStage.cpp, function fenceReadyEvent(), line 183)
May 6 15:57:45 caper-desktop nvargus-daemon[16064]: SCF: Error InvalidState: Buffer is not locked (in src/api/Buffer.cpp, function unlock(), line 454)
May 6 15:57:45 caper-desktop nvargus-daemon[16064]: SCF: Error InvalidState: (propagating from src/components/stages/CaptureStage.cpp, function fenceReadyEvent(), line 184)
May 6 15:57:45 caper-desktop nvargus-daemon[16064]: SCF: Error InvalidState: (propagating from src/services/capture/CaptureRecord.cpp, function issueCapture(), line 492)
May 6 15:57:45 caper-desktop nvargus-daemon[16064]: SCF: Error InvalidState: (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 1295)
May 6 15:57:45 caper-desktop nvargus-daemon[16064]: SCF: Error InvalidState: (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 1126)
May 6 15:57:45 caper-desktop nvargus-daemon[16064]: (Argus) Error InvalidState: Buffer was already acquired (in src/api/BufferImpl.cpp, function bufferAcquired(), line 87)
May 6 15:57:45 caper-desktop nvargus-daemon[16064]: SCF: Error InvalidState: (propagating from src/common/Utils.cpp, function

22

The error tell capture incomplete. Could be timeout or sensor output problem.

Try set infinite timeout mode to verify.

sudo service nvargus-daemon stop
sudo enableCamInfiniteTimeout=1 nvargus-daemon