Jetson TX2 argus crash on camera initialization after upgrade from L4T 32.5.1 to 32.7.6

SOM: Jetson TX2 PN 900-83310-0001-000 BV
Carrier: LI-TX1-CB-6CAM
Cameras on MIPI ports A-E: LI-IMX290-MIPI
Camera on MIPI port F: LI-LMH0341-MPI-DSER

We have recently updated our system from L4T 32.5.1 to 32.7.6 to support the last revision of jetson TX2’s that have PCN211181 memory.

We have been running fine for ~6 years on 32.5.1 but after the update from 32.5.1 to 32.7.6 we are seeing an nvidia argus daemon crash when initializing the cameras.

Our system uses the Jetson TX2 with a Leopard Imaging LI-TX1-CB-6CAM carrier board and L4T is pulled into our custom buildroot image. We also run a custom video pipeline that uses argus via it’s library interface as well as the provided v4l extension.

We use 6 total cameras, 5 LI-IMX290-MIPI cameras on ports A-E, and on port F there is a LI-LMH0341-MPI-DSER .

Were there any updates in Argus between those versions that might be affecting camera initialization/timing or possibly changes in what is expected in the device tree?

Feb 02 15:29:59 [5152]: NVARGUS STARTED, PID 5153, 0 CPUS
Feb 02 15:29:59 [5152]: STARTED, PID 5154
Feb 02 15:29:59 [5154]: executing on 4 cpus
Feb 02 15:29:59 [5153]: executing nvargus-daemon on 2 cpus
Feb 02 15:30:00 [5153]: === NVIDIA Libargus Camera Service (0.98.3)=== Listening for connections...=== [5154]: Connection established (7F76BB8130)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
Feb 02 15:30:00 [5153]: OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
Feb 02 15:30:00 [5153]: OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
Feb 02 15:30:00 [5153]: OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module3
Feb 02 15:30:00 [5153]: OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module4
Feb 02 15:30:00 [5153]: OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module5
Feb 02 15:30:00 [5153]: NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
Feb 02 15:30:00 [5153]: NvPclHwGetModuleList: No module data found
Feb 02 15:30:00 [5153]: NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
Feb 02 15:30:00 [5153]: NvPclHwGetModuleList: No module data found
Feb 02 15:30:00 [5153]: NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
Feb 02 15:30:00 [5153]: NvPclHwGetModuleList: No module data found
Feb 02 15:30:00 [5153]: NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
Feb 02 15:30:00 [5153]: NvPclHwGetModuleList: No module data found
Feb 02 15:30:00 [5153]: NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
Feb 02 15:30:00 [5153]: NvPclHwGetModuleList: No module data found
Feb 02 15:30:00 [5153]: NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
Feb 02 15:30:00 [5153]: NvPclHwGetModuleList: No module data found
Feb 02 15:30:01 [5153]: OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
Feb 02 15:30:01 [5153]: ---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
Feb 02 15:30:01 [5153]: ---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
Feb 02 15:30:01 [5153]: ---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
Feb 02 15:30:01 [5153]: ---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
Feb 02 15:30:01 [5153]: ---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
Feb 02 15:30:01 [5153]: ---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
Feb 02 15:30:01 [5153]: === [5154]: CameraProvider initialized (0x7f70d16170)
Feb 02 15:30:01 [5154]: 6 cameras available
Feb 02 15:30:01 [5154]: output stream 0xc0d8f80 0xc0a4f70 0xc12f401
Feb 02 15:30:01 [5154]: CONNECTED!
Feb 02 15:30:02 [5154]: output stream 0xc0d8c30 0xc0a2e10 0xc0a2a51
Feb 02 15:30:02 [5154]: CONNECTED!
Feb 02 15:30:02 [5154]: output stream 0xc0a3dc0 0xc0a4800 0xc0a3061
Feb 02 15:30:02 [5154]: CONNECTED!
Feb 02 15:30:02 [5154]: output stream 0xc0e0800 0xc0dadd0 0xc0e06a1
Feb 02 15:30:02 [5154]: CONNECTED!
Feb 02 15:30:02 [5154]: output stream 0xc0dbab0 0xc0dc4f0 0xc0db571
Feb 02 15:30:02 [5154]: CONNECTED!
Feb 02 15:30:03 [5153]: PowerServiceCore:handleRequests: timePassed = 636
Feb 02 15:30:03 [5153]: PowerServiceCore:handleRequests: timePassed = 618
Feb 02 15:30:04 [5153]: PowerServiceCore:handleRequests: timePassed = 613
Feb 02 15:30:05 kernel: sdi3841a 35-001a: sdi3841a_power_on: power on
Feb 02 15:30:05 systemd[1]: Started video streaming server.
Feb 02 15:30:05 kernel: sdi3841a 35-001a: [sdi3841a_set_mode]
Feb 02 15:30:05 kernel: sdi3841a 35-001a: [sdi3841a_start_streaming] dummy.
Feb 02 15:30:05 [5153]: PowerServiceCore:handleRequests: timePassed = 609
Feb 02 15:30:05 [5152]: SERVICE STARTED!
Feb 02 15:30:05 [5952]: NvMMLiteOpen : Block : BlockType = 4
Feb 02 15:30:05 [5952]: ===== NVMEDIA: NVENC =====
Feb 02 15:30:05 [5952]: NvMMLiteBlockCreate : Block : BlockType = 4
Feb 02 15:30:05 [5952]: Requested 8 buffers, got 8
Feb 02 15:30:05 [5952]: H264: Profile = 66, Level = 31
Feb 02 15:30:11 systemd[1]: Started NVIDIA Late Init Script.
Feb 02 15:30:11 systemd[1]: Startup finished in 9.857s (kernel) + 24.086s (userspace) = 33.944s.
Feb 02 15:30:11 systemd[1]: nv-late-init.service: Succeeded.
Feb 02 15:30:12 kernel: spmic-ldo0: disabling
Feb 02 15:30:12 kernel: spmic-ldo1: disabling
Feb 02 15:30:12 kernel: en-vdd-sd: disabling
Feb 02 15:30:12 kernel: vdd-usb0-5v: disabling
Feb 02 15:30:12 kernel: vdd-usb1-5v: disabling
Feb 02 15:30:12 kernel: vdd-fan: disabling
Feb 02 15:30:12 kernel: vdd-3v3: disabling
Feb 02 15:30:12 kernel: en-vdd-vcm-2v8: disabling
Feb 02 15:30:12 kernel: vdd-usb2-5v: disabling
Feb 02 15:30:12 kernel: vdd-sys-bl: disabling
Feb 02 15:31:52 [5153]: corrupted double-linked list
Feb 02 15:31:52 audit[5153]: ANOM_ABEND auid=4294967295 uid=1000 gid=1000 ses=4294967295 pid=5153 comm=53434620457865637574696F6E exe="/usr/sbin/nvargus-daemon" sig=6
Feb 02 15:31:52 kernel: audit_printk_skb: 15 callbacks suppressed
Feb 02 15:31:52 kernel: audit: type=1701 audit(1612279912.132:7): auid=4294967295 uid=1000 gid=1000 ses=4294967295 pid=5153 comm=53434620457865637574696F6E exe="/usr/sbin/nvargus-daemon" sig=6
Feb 02 15:31:52 [5154]: (Argus) Error FileOperationFailed: Failed socket read: Connection reset by peer (in src/rpc/socket/common/SocketUtils.cpp, function readSocket(), line 79)
Feb 02 15:31:52 [5154]: (Argus) Error FileOperationFailed: Unexpected error in reading socket (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 266)
Feb 02 15:31:52 [5154]: (Argus) Error FileOperationFailed: Receive worker failure, notifying 1 waiting threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 340)
Feb 02 15:31:52 [5154]: (Argus) Error InvalidState: Argus client is exiting with 1 outstanding client threads (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadCore(), line 357)
Feb 02 15:31:52 [5154]: (Argus) Error FileOperationFailed: Receiving thread terminated with error (in src/rpc/socket/client/ClientSocketManager.cpp, function recvThreadWrapper(), line 368)
Feb 02 15:31:52 [5154]: (Argus) Error FileOperationFailed: Client thread received an error from socket (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 145)
Feb 02 15:31:52 [5154]: (Argus) Error FileOperationFailed: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
Feb 02 15:31:52 [5154]: Failed to acquire ISourceSettings interface!
Feb 02 15:31:52 [5154]: (Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
Feb 02 15:31:52 [5154]: (Argus) Error InvalidState: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
Feb 02 15:31:52 [5154]: Failed to acquire ISourceSettings interface!
Feb 02 15:31:52 [5154]: (Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
Feb 02 15:31:52 [5154]: (Argus) Error InvalidState: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
Feb 02 15:31:52 [5154]: Failed to acquire ISourceSettings interface!
Feb 02 15:31:52 [5154]: (Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
Feb 02 15:31:52 [5154]: (Argus) Error InvalidState: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
Feb 02 15:31:52 [5154]: Failed to acquire ISourceSettings interface!
Feb 02 15:31:52 [5154]: (Argus) Error InvalidState: Receive thread is not running cannot send. (in src/rpc/socket/client/ClientSocketManager.cpp, function send(), line 96)
Feb 02 15:31:52 [5154]: (Argus) Error InvalidState: (propagating from src/rpc/socket/client/SocketClientDispatch.cpp, function dispatch(), line 91)
Feb 02 15:31:52 [5154]: Failed to acquire ISourceSettings interface!
Feb 02 15:31:52 [5152]: NVARGUS DIED, STATUS 6
Feb 02 15:31:52 [5152]: KILLING 5154
Feb 02 15:31:52 kernel: sdi3841a 35-001a: [sdi3841a_stop_streaming] dummy
Feb 02 15:31:52 kernel: nvcsi 150c0000.nvcsi: csi4_cil_check_status (5) CILA_INTR_STATUS 0x00000044
Feb 02 15:31:52 kernel: nvcsi 150c0000.nvcsi: csi4_cil_check_status (5) CILA_ERR_INTR_STATUS 0x00000044
Feb 02 15:31:52 kernel: sdi3841a 35-001a: sdi3841a_power_off: power off

This is most likely a sensor/driver/CSI signal issue causing Argus to hit a bad error-recovery path.

Do you verify the sensor functionality by v4l2-ctl?

If v4l2-ctl also failed you may need to check the sensor driver or device tree.

Thanks

I will look into that.

Can you point me to any documentation showing the changes in how the device tree is handled in 32.5.1 vs 32.7.6?

To be clear, when using our custom video pipeline the cameras do stream for a while, the crash seems to happen when setting exposure settings and requesting frames from the cameras.

As far as v4l2 with nvargus-daemon stopped v4l2-ctl is working fine.

Starting live stream tests on /dev/video0 through /dev/video5...

Live Camera Framerates (Press Ctrl+C to stop)

Camera 0: 30.00 fps
Camera 1: 30.09 fps
Camera 2: 30.09 fps
Camera 3: 30.09 fps
Camera 4: 30.00 fps
Camera 5: 30.00 fps

Live Camera Framerates (Press Ctrl+C to stop)

Camera 0: 30.00 fps
Camera 1: 30.00 fps
Camera 2: 30.04 fps
Camera 3: 30.00 fps
Camera 4: 30.04 fps
Camera 5: 30.00 fps

Live Camera Framerates (Press Ctrl+C to stop)

Camera 0: 30.00 fps
Camera 1: 30.00 fps
Camera 2: 30.03 fps
Camera 3: 30.00 fps
Camera 4: 30.00 fps
Camera 5: 30.00 fps

Live Camera Framerates (Press Ctrl+C to stop)

Camera 0: 30.00 fps
Camera 1: 30.00 fps
Camera 2: 30.02 fps
Camera 3: 30.00 fps
Camera 4: 30.02 fps
Camera 5: 30.00 fps

Live Camera Framerates (Press Ctrl+C to stop)

Camera 0: 30.00 fps
Camera 1: 30.00 fps
Camera 2: 30.01 fps
Camera 3: 30.00 fps
Camera 4: 30.01 fps
Camera 5: 30.00 fps

Live Camera Framerates (Press Ctrl+C to stop)

Camera 0: 30.00 fps
Camera 1: 30.00 fps
Camera 2: 30.01 fps
Camera 3: 30.00 fps
Camera 4: 30.00 fps
Camera 5: 30.00 fps

This is the script I am running

#!/bin/bash
Ensure Argus isn't holding the nodes

sudo systemctl stop nvargus-daemon 2>/dev/null

echo "Starting live stream tests on /dev/video0 through /dev/video5..."

declare -a PIDS

Cleanup function to kill background processes on exit

cleanup() {
echo -e "\nStopping cameras and cleaning up..."
for pid in "${PIDS[@]}"; do
kill $pid 2>/dev/null
done
rm -f /tmp/cam*_live.log
exit 0
}

Trap Ctrl+C (SIGINT) to ensure clean shutdown

trap cleanup SIGINT SIGTERM

Launch cameras in the background indefinitely

for i in {0..5}
do
# Clear out any old log data first
> /tmp/cam${i}_live.log

if [ $i -eq 5 ]; then
    # Camera 5: The SDI-to-MIPI YUV stream
    ./v4l2-ctl -d /dev/video$i \
        --set-fmt-video=width=1920,height=1080,pixelformat=YUYV \
        --stream-mmap > /tmp/cam${i}_live.log 2>&1 &
else
    # Cameras 0-4: The RAW 12-bit Bayer streams
    # Note: RG12 is the standard V4L2 fourcc for 12-bit RGGB
    ./v4l2-ctl -d /dev/video$i \
       --set-fmt-video=width=1948,height=1096,pixelformat=RG12 \
       --set-ctrl sensor_mode=0 \
       --set-ctrl bypass_mode=0 \
       --stream-mmap > /tmp/cam${i}_live.log 2>&1 &
fi
    
# Store the Process ID so we can kill it later
PIDS+=($!)

done

Give the Tegra hardware a second to lock onto the streams

sleep 2

Live monitoring loop

while true
do
#printf "\033c"
echo "==============================================="
echo " Live Camera Framerates (Press Ctrl+C to stop) "
echo "==============================================="

for i in {0..5}
do
    # Grab the very last 'fps' line printed to the log
    FPS=$(grep -i "fps" /tmp/cam${i}_live.log | tail -n 1 | grep -oE '[0-9.]+ fps' | cut -d' ' -f1)
    
    # Check if the process is actually still running
    if kill -0 "${PIDS[$i]}" 2>/dev/null; then
        if [ -z "$FPS" ]; then
            echo "Camera $i: Negotiating stream..."
        else
            echo "Camera $i: $FPS fps"
        fi
    else
        echo "Camera $i: OFFLINE / FAILED"
    fi
done

# Refresh rate of the dashboard
sleep 1

done

Attached is a snippet of the camera device tree cameras.txt (42.2 KB)

Do you test by single camera?

Do you verify by sample app like argus_camera?