Unscheduled reboots of Jetson Xavier AGX DevKit

Hi,

I am running Jetpack 32.4.3 on a Xavier DevKit which keeps rebooting randomly.

Attached the dmesg output and the syslog which show a CPU stall. I am using the D3 Engineering camera board (approved NVIDIA Jetson Camera Partner) and they have indicated its an issue related to NVIDIA HW.

Is this a known issue? Is there a patch for this L4T release?

Output of cat /etc/nv_tegra_release :

R32 (release), REVISION: 4.3, GCID: 21589087, BOARD: t186ref, EABI: aarch64, DATE: Fri Jun 26 04:34:27 UTC 2020

Thanks,
Sandip

dmesg_13Oct.txt (85.9 KB)
syslog_oct13 (2.2 MB)

hello sandiprmlc0,

could you please try remove the camera board to reproduce the issue,
besides, could you please also confirm you’re providing correct power-supply to the Xavier platform.

Hi,

could you please try remove the camera board to reproduce the issue,

Yes, but before I do that, do you need more logs, core dumps etc. from the system? Is there a livepatch that I can simply try with the current setup?

I am supplying power using the approved LITEON 19V adapter to the AGX.

Best,
Sandip

We have also been experiencing such random reboots using an existing 32.3.1 image we were flashing on new boards. This image worked well in the past but then started being unstable on new systems we were installing it on.

I ported our setup to 32.6.1 and have not seen any problems so far - keeping an eye on this.

Could it be that newer revisions of boards are not supported by older Jetpack versions?

1 Like

hello sandiprmlc0,

yes, please setup serial console to gather uart logs, or you may put terminal there to keep gathering kernel logs with $ dmesg --follow

1 Like

Random reboots seem to be quite the standard on the AGX.
I have noticed that having a ping to the gateway 192.168.0.1 (or similar) running in the background can keep the AGX running with up times of several months. As far as I can see there is some bug in either the network drivers or the networking hardware. Perhaps even power management related. I’ve tried to find the cause but only found this work-around. My suspicion for the network was triggered by intermittent network pauses. In all after network issues a reboot would be imminent. It’s like some interrupts don’t get noticed but the ping will cause the network stack to be nudged to keep processing pending packets.
There have been a multitude of reports on reboots and i have read many but there seems to be no knowledge of what causes it nor how to solve it. I guess I have learned to live with it. (Won’t be using any AGX in production anywhere here due to this issue.) For desktop use and development use it’s fine. Reboot is fast. Perhaps a future iteration with a newer processor will be rock stable, who knows. I have 2 jetson AGX systems for a full year now in testing and both have the same issue. One runs stock Ubuntu 18.04.5 LTS the other runs Debian Sid.

F.w.i.w.
[ 95.531425] [] el1_irq+0xe8/0x194
[ 95.531467] [] nf_conntrack_in+0x100/0x940 [nf_conntrack]
[ 95.531480] [] ipv4_conntrack_in+0x30/0x40 [nf_conntrack_ipv4]

Shows clearly that this is networking IRQ related and I have not read any sensible explanation anywhere in these forums nor fixes. I checked the changelogs for the NVidia kernels but can’t find anything that appears to be addressing it.

The only thing i recently saw was that they disable reboots on hung tasks in newer kernels… sounds more like a quick way to reduce reboots than a fix for the cause. (forgive my cynicism)
I hope the upcoming newer kernel will have a fix. The 5.x kernels are not yet available for testing but i hope to see them soon.

1 Like

Please check the /proc/interrupts and see if there is abnormal interrupt coming from bluedroid_pm.

It is a known issue on AGX Xavier. We have a patch on this forum too. Unfortunately it didn’t catch the last release.

The patch WayneWWW is refering to is this one: AGX Xavier freeze in MAXN mode - #22 by WayneWWW

1 Like

Thanks dkreutz for the link to the patch. Could I apply this to the JP 32.4.3? On your link, the patch seems to be for JP 4.4.1/4.5. Can it be applied to 4.3 as well? WayneWWW?

Thanks janrinze for your detailed reply! Yes, I have been seeing these intermittent network pauses as well! Ok, I will set up a background process to ping to the local gateway. Btw I am using VPN tunnel interfaces on the Jetson as well, should I instead ping on this tunnel interface?

This is an interesting question. WayneWWW any feedback?

I have successfully applied the patch to JP 4.4 and 4.5 and recently to JP4.6.
Don’t know if it works for 4.3. Maybe you simply look into the sources if there is the same/similar code as well.

1 Like

A quick way to validate whether this patch could resolve the problem is directly remove bluedroid_pm driver from the lsmod list.

If this issue is gone, after removing this driver, then that patch can work.

[quote=“BareMetalCoder, post:5, topic:191975”]
Could it be that newer revisions of boards are not supported by older Jetpack versions?
[/quote]

You can check the PCN list here and see if any of your Xavier gets affected by those.

Thank you. It’s good to discover the existence of these! That being said, it does not appear to be the case. The only significant one I see for AGX is the memory size change, but this required r32.1 which is older than any L4T we have used.

Hi @WayneWWW,
Disabling the bluedroid driver has not resolved the issue. Attached the syslog from the last reboot. Does it mean that the patch will not work in this case?

@janrinze,
I don’t think its an issue with the network stack not being active as the crash is observed when I send (and receive) a lot of traffic on the network interface.

I think the problem has multiple causes and I’m trying to narrow down the source but it seems there is different/inconsistent information in the syslogs/dmesg output everytime the system reboots…
syslog.1 (1.4 MB)

  1. Yes, if this issue is not related to bluedroid_pm, then that patch may not work. But please make sure you really disable it by checking “lsmod” command.

  2. I think the logs you shared so far do not help and they didn’t indicate the real cause of your reboot.
    For example, your dmesg in the first comment indeed has a kernel panic. But it still prints later usb log. It means the board didn’t reboot due to that panic.
    Also, in most of time, sudden reboot will not be recorded in syslog.

Thus, what you should do is remove “quiet” inside /boot/extlinux/extlinux.conf, and dump your log from the serial console instead of dmesg command and syslog. Use this console to monitor the error until it reboots.

Hi @WayneWWW ,

  1. I have confirmed that the bluedroid_pm driver is disabled.

  2. I am using a modified L4T kernel provided by D3 Engineering where the quiet flag was removed in /boot/extlinux/extlinux.conf. Attached the logdump from the serial console.

Do you have any idea how we could pinpoint the problem?

minicom_logs.txt (96.4 KB)

HI @sandiprmlc0,

Actually, the most important info here is just the first few lines. The rest of logs are already for the reboot, so not have the info we need.

[  485.258654] INFO: task ViCsiHw frameCo:10028 blocked for more than 120 seconds.
[  485.258838]       Not tainted 4.9.140+4.0.0-d3+ #2
[  485.258941] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  485.259236] Kernel panic - not syncing: hung_task: blocked tasks
[  485.259359] CPU: 5 PID: 717 Comm: khungtaskd Not tainted 4.9.140+4.0.0-d3+ #2
[  485.259492] Hardware name: Jetson-AGX (DT)
[  485.259580] Call trace:
[  485.259638] [<ffffff800808be10>] dump_backtrace+0x0/0x180
[  485.259743] [<ffffff800808c38c>] show_stack+0x24/0x30
[  485.259846] [<ffffff800847bc64>] dump_stack+0x94/0xb8
[  485.259950] [<ffffff80081e7b54>] panic+0x118/0x28c
[  485.260055] [<ffffff80081a8594>] watchdog+0x324/0x3d0
[  485.260156] [<ffffff8008103228>] kthread+0xd8/0xf0
[  485.260247] [<ffffff80080838a0>] ret_from_fork+0x10/0x30
[  485.260494] SMP: stopping secondary CPUs
[  485.260803] Kernel Offset: disabled
[  485.261069] Memory Limit: none

Do you have any log prior to these lines? Not sure if you cut most of them.

Based on current partial log, it says the reboot is due to " ViCsiHw frameCo:10028 blocked for more than 120 seconds".

Are you running some camera usecase?

Unfortunately not! The full output from the serial port is what I shared.

I can check dmesg and syslog to see if there is any other information.

Yes, I am running the D3 cameras with NvArgus streaming to multiple UDP ports using GStreamer.

Then I think that is the full log. No need to check dmesg and syslog. They probably doesn’t record any one since the system is already hang.
Please try to confirm if this application is indeed persistent to reproduce this error.

Attached also the syslog output before the reboot. The syslog actually shows an error in the nvargus_daemon related to the frame capture:

Oct 26 13:32:50 diesel nvargus-daemon[6779]: NvCaptureStatusErrorDecode Stream 0.0 failed: sof_ts 486922158208 eof_ts 486954174496 frame 225 error 2 data 0x000000a0
Oct 26 13:32:50 diesel nvargus-daemon[6779]: NvCaptureStatusErrorDecode Capture-Error: CSIMUX_FRAME (0x00000002)
Oct 26 13:32:50 diesel nvargus-daemon[6779]: CsimuxFrameError_Regular : 0x000000a0
Oct 26 13:32:50 diesel nvargus-daemon[6779]:     Stream ID                [ 2: 0]: 0
Oct 26 13:32:50 diesel nvargus-daemon[6779]:         
Oct 26 13:32:50 diesel nvargus-daemon[6779]:     VPR state from fuse block    [ 3]: 0
Oct 26 13:32:50 diesel nvargus-daemon[6779]:         
Oct 26 13:32:50 diesel nvargus-daemon[6779]:     Frame end (FE)              [ 5]: 1
Oct 26 13:32:50 diesel nvargus-daemon[6779]:         A frame end has been found on a regular mode stream.
Oct 26 13:32:50 diesel nvargus-daemon[6779]:     FS_FAULT                    [ 7]: 1
Oct 26 13:32:50 diesel nvargus-daemon[6779]:         A FS packet was found for a virtual channel that was already in frame.An errored FE packet was injected before FS was allowed through.
Oct 26 13:32:50 diesel nvargus-daemon[6779]:     Binary VC number [3:2]   [27:26]: 0
Oct 26 13:32:50 diesel nvargus-daemon[6779]:         To get full binary VC number, user need to concatenate VC[3:2] and VC[1:0] together.
Oct 26 13:32:50 diesel nvargus-daemon[6779]: SCF: Error InvalidState: Capture error with status 2 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 880)
Oct 26 13:32:52 diesel nvargus-daemon[6779]: === imp_sndstream[24891]: CameraProvider destroyed (0x7e8c0019f0)=== imp_sndstream[24891]: Connection closed (7EC8FF91D0)=== imp_sndstream[24891]: Connection cleaned up (7EC8FF91D0)=== imp_sndstream[25507]: Connection established (7EC8FF91D0)=== imp_sndstream[25507]: CameraProvider initialized (0x7e8c003370)CAM: serial no file already exists, skips storing againPowerServiceCore:handleRequests: timePassed = 1102
Oct 26 13:32:53 diesel kernel: [  473.323980] FAN rising trip_level:1 cur_temp:41100 trip_temps[2]:53000
Oct 26 13:32:54 diesel kernel: [  474.447676] FAN rising trip_level:1 cur_temp:42800 trip_temps[2]:53000
Oct 26 13:32:55 diesel nvargus-daemon[6779]: === imp_sndstream[26121]: Connection established (7E6B7FE1D0)=== imp_sndstream[26121]: CameraProvider initialized (0x7ef011ade0)CAM: serial no file already exists, skips storing againPowerServiceCore:handleRequests: timePassed = 987
Oct 26 13:32:56 diesel kernel: [  476.687651] FAN rising trip_level:1 cur_temp:41150 trip_temps[2]:53000
Oct 26 13:32:58 diesel nvargus-daemon[6779]: NvCaptureStatusErrorDecode Stream 0.0 failed: sof_ts 0 eof_ts 494735877408 frame 0 error 2 data 0x000000a0
Oct 26 13:32:58 diesel nvargus-daemon[6779]: NvCaptureStatusErrorDecode Capture-Error: CSIMUX_FRAME (0x00000002)
Oct 26 13:32:58 diesel nvargus-daemon[6779]: CsimuxFrameError_Regular : 0x000000a0
Oct 26 13:32:58 diesel nvargus-daemon[6779]:     Stream ID                [ 2: 0]: 0
Oct 26 13:32:58 diesel nvargus-daemon[6779]:         
Oct 26 13:32:58 diesel nvargus-daemon[6779]:     VPR state from fuse block    [ 3]: 0
Oct 26 13:32:58 diesel nvargus-daemon[6779]:         
Oct 26 13:32:58 diesel nvargus-daemon[6779]:     Frame end (FE)              [ 5]: 1
Oct 26 13:32:58 diesel nvargus-daemon[6779]:         A frame end has been found on a regular mode stream.
Oct 26 13:32:58 diesel nvargus-daemon[6779]:     FS_FAULT                    [ 7]: 1
Oct 26 13:32:58 diesel nvargus-daemon[6779]:         A FS packet was found for a virtual channel that was already in frame.An errored FE packet was injected before FS was allowed through.
Oct 26 13:32:58 diesel nvargus-daemon[6779]:     Binary VC number [3:2]   [27:26]: 0
Oct 26 13:32:58 diesel nvargus-daemon[6779]:         To get full binary VC number, user need to concatenate VC[3:2] and VC[1:0] together.
Oct 26 13:32:58 diesel nvargus-daemon[6779]: SCF: Error InvalidState: Capture error with status 2 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 880)
Oct 26 13:32:58 diesel nvargus-daemon[6779]: NvCaptureStatusErrorDecode Stream 0.2 failed: sof_ts 0 eof_ts 494739843072 frame 0 error 2 data 0x000000a0
Oct 26 13:32:58 diesel nvargus-daemon[6779]: NvCaptureStatusErrorDecode Capture-Error: CSIMUX_FRAME (0x00000002)
Oct 26 13:32:58 diesel nvargus-daemon[6779]: CsimuxFrameError_Regular : 0x000000a0
Oct 26 13:32:58 diesel nvargus-daemon[6779]:     Stream ID                [ 2: 0]: 0
Oct 26 13:32:58 diesel nvargus-daemon[6779]:         
Oct 26 13:32:58 diesel nvargus-daemon[6779]:     VPR state from fuse block    [ 3]: 0
Oct 26 13:32:58 diesel nvargus-daemon[6779]:         
Oct 26 13:32:58 diesel nvargus-daemon[6779]:     Frame end (FE)              [ 5]: 1
Oct 26 13:32:58 diesel nvargus-daemon[6779]:         A frame end has been found on a regular mode stream.
Oct 26 13:32:58 diesel nvargus-daemon[6779]:     FS_FAULT                    [ 7]: 1
Oct 26 13:32:58 diesel nvargus-daemon[6779]:         A FS packet was found for a virtual channel that was already in frame.An errored FE packet was injected before FS was allowed through.
Oct 26 13:32:58 diesel nvargus-daemon[6779]:     Binary VC number [3:2]   [27:26]: 0
Oct 26 13:32:58 diesel nvargus-daemon[6779]:         To get full binary VC number, user need to concatenate VC[3:2] and VC[1:0] together.
Oct 26 13:32:58 diesel nvargus-daemon[6779]: SCF: Error InvalidState: Capture error with status 2 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 880)
Oct 26 13:32:58 diesel kernel: [  478.923259] FAN rising trip_level:1 cur_temp:41300 trip_temps[2]:53000
Oct 26 13:32:59 diesel kernel: [  480.043119] FAN rising trip_level:1 cur_temp:42200 trip_temps[2]:53000
Oct 26 13:33:01 diesel kernel: [  481.163180] FAN rising trip_level:1 cur_temp:42500 trip_temps[2]:53000
Oct 26 13:33:01 diesel nvargus-daemon[6779]: === imp_sndstream[23040]: CameraProvider destroyed (0x7eb8000be0)=== imp_sndstream[23040]: Connection closed (7ED17FA1D0)=== imp_sndstream[23040]: Connection cleaned up (7ED17FA1D0)NvCaptureStatusErrorDecode Stream 0.2 failed: sof_ts 0 eof_ts 498274031744 frame 0 error 2 data 0x000000a0
Oct 26 13:33:01 diesel nvargus-daemon[6779]: NvCaptureStatusErrorDecode Capture-Error: CSIMUX_FRAME (0x00000002)
Oct 26 13:33:01 diesel nvargus-daemon[6779]: CsimuxFrameError_Regular : 0x000000a0
Oct 26 13:33:01 diesel nvargus-daemon[6779]:     Stream ID                [ 2: 0]: 0
Oct 26 13:33:01 diesel nvargus-daemon[6779]:         
Oct 26 13:33:01 diesel nvargus-daemon[6779]:     VPR state from fuse block    [ 3]: 0
Oct 26 13:33:01 diesel nvargus-daemon[6779]:         
Oct 26 13:33:01 diesel nvargus-daemon[6779]:     Frame end (FE)              [ 5]: 1
Oct 26 13:33:01 diesel nvargus-daemon[6779]:         A frame end has been found on a regular mode stream.
Oct 26 13:33:01 diesel nvargus-daemon[6779]:     FS_FAULT                    [ 7]: 1
Oct 26 13:33:01 diesel nvargus-daemon[6779]:         A FS packet was found for a virtual channel that was already in frame.An errored FE packet was injected before FS was allowed through.
Oct 26 13:33:01 diesel nvargus-daemon[6779]:     Binary VC number [3:2]   [27:26]: 0
Oct 26 13:33:01 diesel nvargus-daemon[6779]:         To get full binary VC number, user need to concatenate VC[3:2] and VC[1:0] together.
Oct 26 13:33:01 diesel nvargus-daemon[6779]: SCF: Error InvalidState: Capture error with status 2 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 880)
Oct 26 13:33:01 diesel nvargus-daemon[6779]: NvCaptureStatusErrorDecode Stream 0.1 failed: sof_ts 0 eof_ts 498287637888 frame 0 error 2 data 0x000000a0
Oct 26 13:33:01 diesel nvargus-daemon[6779]: NvCaptureStatusErrorDecode Capture-Error: CSIMUX_FRAME (0x00000002)
Oct 26 13:33:01 diesel nvargus-daemon[6779]: CsimuxFrameError_Regular : 0x000000a0
Oct 26 13:33:01 diesel nvargus-daemon[6779]:     Stream ID                [ 2: 0]: 0
Oct 26 13:33:01 diesel nvargus-daemon[6779]:         
Oct 26 13:33:01 diesel nvargus-daemon[6779]:     VPR state from fuse block    [ 3]: 0
Oct 26 13:33:01 diesel nvargus-daemon[6779]:         
Oct 26 13:33:01 diesel nvargus-daemon[6779]:     Frame end (FE)              [ 5]: 1
Oct 26 13:33:01 diesel nvargus-daemon[6779]:         A frame end has been found on a regular mode stream.
Oct 26 13:33:01 diesel nvargus-daemon[6779]:     FS_FAULT                    [ 7]: 1
Oct 26 13:33:01 diesel nvargus-daemon[6779]:         A FS packet was found for a virtual channel that was already in frame.An errored FE packet was injected before FS was allowed through.
Oct 26 13:33:01 diesel nvargus-daemon[6779]:     Binary VC number [3:2]   [27:26]: 0
Oct 26 13:33:01 diesel nvargus-daemon[6779]:         To get full binary VC number, user need to concatenate VC[3:2] and VC[1:0] together.
Oct 26 13:33:01 diesel nvargus-daemon[6779]: SCF: Error InvalidState: Capture error with status 2 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 880)
Oct 26 13:33:03 diesel kernel: [  483.403089] FAN rising trip_level:1 cur_temp:41450 trip_temps[2]:53000
Oct 26 13:33:04 diesel nvargus-daemon[6779]: === imp_sndstream[23860]: CameraProvider destroyed (0x7ea800c9c0)=== imp_sndstream[23860]: Connection closed (7E6AFFD1D0)=== imp_sndstream[23860]: Connection cleaned up (7E6AFFD1D0)=== imp_sndstream[27494]: Connection established (7E6AFFD1D0)=== imp_sndstream[27494]: CameraProvider initialized (0x7ea8014340)CAM: serial no file already exists, skips storing againPowerServiceCore:handleRequests: timePassed = 1004
Oct 26 13:33:05 diesel kernel: [  485.642881] FAN rising trip_level:1 cur_temp:41450 trip_temps[2]:53000
Oct 26 13:33:07 diesel nvargus-daemon[6779]: === imp_sndstream[27994]: Connection established (7F46FFD1D0)=== imp_sndstream[27994]: CameraProvider initialized (0x7f34000b20)CAM: serial no file already exists, skips storing againPowerServiceCore:handleRequests: timePassed = 1091
Oct 26 13:33:07 diesel kernel: [  487.883285] FAN rising trip_level:1 cur_temp:41450 trip_temps[2]:53000
Oct 26 13:33:08 diesel kernel: [  489.002689] FAN rising trip_level:1 cur_temp:42200 trip_temps[2]:53000
Oct 26 13:33:11 diesel kernel: [  491.242535] FAN rising trip_level:1 cur_temp:42700 trip_temps[2]:53000
Oct 26 13:33:13 diesel kernel: [  493.482264] FAN rising trip_level:1 cur_temp:42400 trip_temps[2]:53000
Oct 26 13:33:16 diesel kernel: [  496.842261] FAN rising trip_level:1 cur_temp:42700 trip_temps[2]:53000
Oct 26 13:33:18 diesel kernel: [  499.082147] FAN rising trip_level:1 cur_temp:42700 trip_temps[2]:53000
Oct 26 13:33:21 diesel kernel: [  501.321972] FAN rising trip_level:1 cur_temp:42700 trip_temps[2]:53000
Oct 26 13:33:24 diesel kernel: [  504.681796] FAN rising trip_level:1 cur_temp:43000 trip_temps[2]:53000
Oct 26 13:33:26 diesel kernel: [  506.925684] FAN rising trip_level:1 cur_temp:42700 trip_temps[2]:53000
Oct 26 13:33:35 diesel kernel: [  515.880890] FAN rising trip_level:1 cur_temp:42700 trip_temps[2]:53000
Oct 26 13:33:39 diesel kernel: [  519.240678] FAN rising trip_level:1 cur_temp:43200 trip_temps[2]:53000
Oct 26 13:33:41 diesel kernel: [  521.480338] FAN rising trip_level:1 cur_temp:43000 trip_temps[2]:53000
Oct 26 13:33:44 diesel ovpn-client[6196]: TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Oct 26 13:33:44 diesel ovpn-client[6196]: TLS Error: TLS handshake failed
Oct 26 13:33:44 diesel ovpn-client[6196]: SIGUSR1[soft,tls-error] received, process restarting
Oct 26 13:33:44 diesel ovpn-client[6196]: Restart pause, 80 second(s)
Oct 26 13:33:44 diesel kernel: [  524.840591] FAN rising trip_level:1 cur_temp:43200 trip_temps[2]:53000
Oct 26 13:33:46 diesel kernel: [  527.080268] FAN rising trip_level:1 cur_temp:43200 trip_temps[2]:53000
Oct 26 13:33:49 diesel kernel: [  529.319972] FAN rising trip_level:1 cur_temp:43200 trip_temps[2]:53000
Oct 26 13:33:54 diesel kernel: [  534.919573] FAN rising trip_level:1 cur_temp:43200 trip_temps[2]:53000
Oct 26 13:33:58 diesel kernel: [  538.279374] FAN rising trip_level:1 cur_temp:43500 trip_temps[2]:53000
Oct 26 13:34:01 diesel kernel: [  541.639253] FAN rising trip_level:1 cur_temp:43200 trip_temps[2]:53000
Oct 26 13:34:03 diesel kernel: [  543.878855] FAN rising trip_level:1 cur_temp:43500 trip_temps[2]:53000
Oct 26 13:34:13 diesel kernel: [  553.958471] FAN rising trip_level:1 cur_temp:43500 trip_temps[2]:53000
Oct 26 13:34:17 diesel kernel: [  557.318121] FAN rising trip_level:1 cur_temp:43500 trip_temps[2]:53000
Oct 26 13:34:19 diesel nvargus-daemon[6779]: NvCaptureStatusErrorDecode Stream 0.2 failed: sof_ts 0 eof_ts 576285525248 frame 0 error 2 data 0x000000a0
Oct 26 13:34:19 diesel nvargus-daemon[6779]: NvCaptureStatusErrorDecode Capture-Error: CSIMUX_FRAME (0x00000002)
Oct 26 13:34:19 diesel nvargus-daemon[6779]: CsimuxFrameError_Regular : 0x000000a0
Oct 26 13:34:19 diesel nvargus-daemon[6779]:     Stream ID                [ 2: 0]: 0
Oct 26 13:34:19 diesel nvargus-daemon[6779]:         
Oct 26 13:34:19 diesel nvargus-daemon[6779]:     VPR state from fuse block    [ 3]: 0
Oct 26 13:34:19 diesel nvargus-daemon[6779]:         
Oct 26 13:34:19 diesel nvargus-daemon[6779]:     Frame end (FE)              [ 5]: 1
Oct 26 13:34:19 diesel nvargus-daemon[6779]:         A frame end has been found on a regular mode stream.
Oct 26 13:34:19 diesel nvargus-daemon[6779]:     FS_FAULT                    [ 7]: 1
Oct 26 13:34:19 diesel nvargus-daemon[6779]:         A FS packet was found for a virtual channel that was already in frame.An errored FE packet was injected before FS was allowed through.
Oct 26 13:34:19 diesel nvargus-daemon[6779]:     Binary VC number [3:2]   [27:26]: 0
Oct 26 13:34:19 diesel nvargus-daemon[6779]:         To get full binary VC number, user need to concatenate VC[3:2] and VC[1:0] together.
Oct 26 13:34:19 diesel nvargus-daemon[6779]: SCF: Error InvalidState: Capture error with status 2 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 880)
Oct 26 13:34:19 diesel nvargus-daemon[6779]: NvCaptureStatusErrorDecode Stream 0.0 failed: sof_ts 0 eof_ts 576288314432 frame 0 error 2 data 0x000000a0
Oct 26 13:34:19 diesel nvargus-daemon[6779]: NvCaptureStatusErrorDecode Capture-Error: CSIMUX_FRAME (0x00000002)
Oct 26 13:34:19 diesel nvargus-daemon[6779]: CsimuxFrameError_Regular : 0x000000a0
Oct 26 13:34:19 diesel nvargus-daemon[6779]:     Stream ID                [ 2: 0]: 0
Oct 26 13:34:19 diesel nvargus-daemon[6779]:         
Oct 26 13:34:19 diesel nvargus-daemon[6779]:     VPR state from fuse block    [ 3]: 0
Oct 26 13:34:19 diesel nvargus-daemon[6779]:         
Oct 26 13:34:19 diesel nvargus-daemon[6779]:     Frame end (FE)              [ 5]: 1
Oct 26 13:34:19 diesel nvargus-daemon[6779]:         A frame end has been found on a regular mode stream.
Oct 26 13:34:19 diesel nvargus-daemon[6779]:     FS_FAULT                    [ 7]: 1
Oct 26 13:34:19 diesel nvargus-daemon[6779]:         A FS packet was found for a virtual channel that was already in frame.An errored FE packet was injected before FS was allowed through.
Oct 26 13:34:19 diesel nvargus-daemon[6779]:     Binary VC number [3:2]   [27:26]: 0
Oct 26 13:34:19 diesel nvargus-daemon[6779]:         To get full binary VC number, user need to concatenate VC[3:2] and VC[1:0] together.
Oct 26 13:34:19 diesel nvargus-daemon[6779]: SCF: Error InvalidState: Capture error with status 2 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 880)
Oct 26 13:34:21 diesel kernel: [  561.797968] FAN rising trip_level:1 cur_temp:43200 trip_temps[2]:53000
Oct 26 13:34:22 diesel nvargus-daemon[6779]: === imp_sndstream[27494]: CameraProvider destroyed (0x7ea8014340)=== imp_sndstream[27494]: Connection closed (7E6AFFD1D0)=== imp_sndstream[27494]: Connection cleaned up (7E6AFFD1D0)NvCaptureStatusErrorDecode Stream 0.1 failed: sof_ts 578925860768 eof_ts 578957876608 frame 2670 error 2 data 0x000000a0
Oct 26 13:34:22 diesel nvargus-daemon[6779]: NvCaptureStatusErrorDecode Capture-Error: CSIMUX_FRAME (0x00000002)
Oct 26 13:34:22 diesel nvargus-daemon[6779]: CsimuxFrameError_Regular : 0x000000a0
Oct 26 13:34:22 diesel nvargus-daemon[6779]:     Stream ID                [ 2: 0]: 0
Oct 26 13:34:22 diesel nvargus-daemon[6779]:         
Oct 26 13:34:22 diesel nvargus-daemon[6779]:     VPR state from fuse block    [ 3]: 0
Oct 26 13:34:22 diesel nvargus-daemon[6779]:         
Oct 26 13:34:22 diesel nvargus-daemon[6779]:     Frame end (FE)              [ 5]: 1
Oct 26 13:34:22 diesel nvargus-daemon[6779]:         A frame end has been found on a regular mode stream.
Oct 26 13:34:22 diesel nvargus-daemon[6779]:     FS_FAULT                    [ 7]: 1
Oct 26 13:34:22 diesel nvargus-daemon[6779]:         A FS packet was found for a virtual channel that was already in frame.An errored FE packet was injected before FS was allowed through.
Oct 26 13:34:22 diesel nvargus-daemon[6779]:     Binary VC number [3:2]   [27:26]: 0
Oct 26 13:34:22 diesel nvargus-daemon[6779]:         To get full binary VC number, user need to concatenate VC[3:2] and VC[1:0] together.
Oct 26 13:34:22 diesel nvargus-daemon[6779]: SCF: Error InvalidState: Capture error with status 2 (channel 0) (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 880)
Oct 26 13:34:23 diesel nvargus-daemon[6779]: === imp_sndstream[27994]: CameraProvider destroyed (0x7f34000b20)=== imp_sndstream[27994]: Connection closed (7F46FFD1D0)=== imp_sndstream[27994]: Connection cleaned up (7F46FFD1D0)NvCaptureStatusErrorDecode Stream 0.1 failed: sof_ts 0 eof_ts 580060283296 frame 0 error 2 data 0x000000a0
Oct 26 13:34:23 diesel nvargus-daemon[6779]: NvCaptureStatusErrorDecode Capture-Error: CSIMUX_FRAME (0x00000002)
Oct 26 13:34:23 diesel nvargus-daemon[6779]: CsimuxFrameError_Regular : 0x000000a0
Oct 26 13:34:23 diesel nvargus-daemon[6779]:     Stream ID                [ 2: 0]: 0
Oct 26 13:34:23 diesel nvargus-daemon[6779]:         
Oct 26 13:34:23 diesel nvargus-daemon[6779]:     VPR state from fuse block    [ 3]: 0
Oct 26 13:34:23 diesel nvargus-daemon[6779]:         
Oct 26 13:34:23 diesel nvargus-daemon[6779]:     Frame end (FE)              [ 5]: 1
Oct 26 13:34:23 diesel nvargus-daemon[6779]:         A frame end has been found on a regular mode stream.
Oct 26 13:34:23 diesel nvargus-daemon[6779]:     FS_FAULT                    [ 7]: 1
Oct 26 13:34:23 diesel nvargus-daemon[6779]:         A FS packet was found for a virtual channel that was already in frame.An errored FE packet was injected before FS was allowed through.
Oct 26 13:34:23 diesel nvargus-daemon[6779]:     Binary VC number [3:2]   [27:26]: 0
Oct 26 13:34:23 diesel nvargus-daemon[6779]:         To get full binary VC number, user need to concatenate VC[3:2] and VC[1:0] together.

Yes I am running the nvarguscamerasrc GStreamer plugin application for streaming live video from the cameras over UDP ports.

syslog (2.7 MB)

1 Like

I am curious if you see “quiet” in the output of “cat /proc/cmdline”? If so, then remove “quiet” from “/boot/extlinux/extlinux.conf” and you might see more useful logs.