Jetson AGX JP4.3 + LeopardImaging imx334 sometimes make self reboot when capture video

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1

We make video capture app that capture 4k@60 video from 3 imx334 cameras, make fisheye correction on cuda and store result to ssd.
Configuration is: Jetson AGX DevKit with LeopardImaging 3 imx334 camera kit.

And system is very unstable.
We often get argus api error after 3-5 minutes capture.
And also sometime os make self reboot by some watchdog in camera api.

Argus api error like this:

(NvCapture) Error InvalidState: Free request list is empty! (in /dvs/git/dirty/git-master_linux/camera/capture/nvcapture/capture.c, function NvCaptureGetRequest(), line 706)

SCF: Error InvalidState:  (propagating from src/services/capture/NvCaptureViCsiHw.cpp, function startCaptureInternal(), line 597)

SCF: Error InvalidState:  (propagating from src/services/capture/CaptureRecord.cpp, function doCSItoMemCapture(), line 517)

SCF: Error InvalidState:  (propagating from src/services/capture/CaptureRecord.cpp, function issueCapture(), line 454)

SCF: Error InvalidState:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 1276)

SCF: Error InvalidState:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueBubbleFillCapturesIfNeeded(), line 676)

SCF: Error InvalidState:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 1118)

SCF: Error InvalidState:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)

SCF: Error InvalidState: Worker thread CaptureScheduler frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)

SCF: Error InvalidState: Capture Scheduler not running (in src/services/capture/CaptureServiceDevice.cpp, function addNewItemToSchedule(), line 908)

SCF: Error InvalidState:  (propagating from src/services/capture/CaptureService.cpp, function addRequest(), line 395)

SCF: Error InvalidState:  (propagating from src/components/stages/MemoryToISPCaptureStage.cpp, function doHandleRequest(), line 137)

SCF: Error InvalidState:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)

SCF: Error InvalidState: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 990)

SCF: Error InvalidState: Capture Scheduler not running (in src/services/capture/CaptureServiceDevice.cpp, function addNewItemToSchedule(), line 908)

SCF: Error InvalidState:  (propagating from src/services/capture/CaptureService.cpp, function addRequest(), line 395)

SCF: Error InvalidState:  (propagating from src/components/stages/MemoryToISPCaptureStage.cpp, function doHandleRequest(), line 137)

SCF: Error InvalidState:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)

SCF: Error InvalidState: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 990)

SCF: Error InvalidState: Capture Scheduler not running (in src/services/capture/CaptureServiceDevice.cpp, function addNewItemToSchedule(), line 908)

SCF: Error InvalidState:  (propagating from src/services/capture/CaptureService.cpp, function addRequest(), line 395)

SCF: Error InvalidState:  (propagating from src/components/stages/MemoryToISPCaptureStage.cpp, function doHandleRequest(), line 137)

SCF: Error InvalidState:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)

SCF: Error InvalidState: Sending critical error event (in src/api/Session.cpp, function sendErrorEvent(), line 990)

SCF: Error InvalidState: Capture Scheduler not running (in src/services/capture/CaptureServiceDevice.cpp, function addNewItemToSchedule(), line 908)

SCF: Error InvalidState:  (propagating from src/services/capture/CaptureService.cpp, function addRequest(), line 395)

SCF: Error InvalidState:  (propagating from src/components/stages/SensorCaptureStage.cpp, function doHandleRequest(), line 87)

SCF: Error InvalidState:  (propagating from src/components/stages/OrderedStage.cpp, function doExecute(), line 158)

SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 667)

(Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)

SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 667)

(Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)

SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 667)

(Argus) Error InvalidState:  (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)

SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 667)

And system reboot like this:

before reboot:

nov 09 13:46:13 jetson-j1-32gb-cam kernel: tegra194-vi5 15c10000.vi: no reply from camera processor
nov 09 13:46:13 jetson-j1-32gb-cam kernel: tegra194-vi5 15c10000.vi: csi_stream_release: failed to close nvcsi stream 4
nov 09 13:46:14 jetson-j1-32gb-cam kernel: tegra194-vi5 15c10000.vi: no reply from camera processor
nov 09 13:46:15 jetson-j1-32gb-cam kernel: tegra194-vi5 15c10000.vi: no reply from camera processor
nov 09 13:46:16 jetson-j1-32gb-cam kernel: tegra194-vi5 15c10000.vi: no reply from camera processor
nov 09 13:46:16 jetson-j1-32gb-cam kernel: tegra194-vi5 15c10000.vi: csi_stream_release: failed to close nvcsi stream 2

nov 09 13:46:21 jetson-j1-32gb-cam kernel: channel context at 4 is busy
nov 09 13:46:21 jetson-j1-32gb-cam kernel: ------------[ cut here ]------------
nov 09 13:46:21 jetson-j1-32gb-cam kernel: WARNING: CPU: 3 PID: 15868 at /home/simon/R32.3.1/kernel/nvidia/drivers/platform/tegra/rtcpu/capture-ivc.c:224 tegra_capture_ivc_notify_chan_id+0x184/0x1a0
nov 09 13:46:21 jetson-j1-32gb-cam kernel: Modules linked in: fuse bnep zram overlay spidev nvgpu bluedroid_pm ip_tables x_tables
nov 09 13:46:21 jetson-j1-32gb-cam kernel: 
nov 09 13:46:21 jetson-j1-32gb-cam kernel: CPU: 3 PID: 15868 Comm: camera-handler Not tainted 4.9.140-tegra #1
nov 09 13:46:21 jetson-j1-32gb-cam kernel: Hardware name: Jetson-AGX (DT)
nov 09 13:46:21 jetson-j1-32gb-cam kernel: task: ffffffc7d1214600 task.stack: ffffffc753f7c000
nov 09 13:46:21 jetson-j1-32gb-cam kernel: PC is at tegra_capture_ivc_notify_chan_id+0x184/0x1a0
nov 09 13:46:21 jetson-j1-32gb-cam kernel: LR is at tegra_capture_ivc_notify_chan_id+0x184/0x1a0
nov 09 13:46:21 jetson-j1-32gb-cam kernel: pc : [<ffffff8008c93724>] lr : [<ffffff8008c93724>] pstate: 60400045
nov 09 13:46:21 jetson-j1-32gb-cam kernel: sp : ffffffc753f7fba0
nov 09 13:46:21 jetson-j1-32gb-cam kernel: x29: ffffffc753f7fba0 x28: ffffffc7d1214600 
nov 09 13:46:21 jetson-j1-32gb-cam kernel: x27: ffffff8008f62000 x26: 000000000000001d 
nov 09 13:46:21 jetson-j1-32gb-cam kernel: x25: 0000000000000043 x24: ffffffc7d5658020 
nov 09 13:46:21 jetson-j1-32gb-cam kernel: x23: 0000000000000004 x22: ffffffc7d5658cd8 
nov 09 13:46:21 jetson-j1-32gb-cam kernel: x21: 0000000000000066 x20: 0000000000000cc0 
nov 09 13:46:21 jetson-j1-32gb-cam kernel: x19: ffffffc7d5658018 x18: 0000000000000000 
nov 09 13:46:21 jetson-j1-32gb-cam kernel: x17: 0000007fa680bb00 x16: 0000000000000000 
nov 09 13:46:21 jetson-j1-32gb-cam kernel: x15: ffffffffffffffff x14: ffffff800a128260 
nov 09 13:46:21 jetson-j1-32gb-cam kernel: x13: ffffff800a127e94 x12: 0000000000000000 
nov 09 13:46:21 jetson-j1-32gb-cam kernel: x11: 00000000003637a5 x10: 0000000000000512 
nov 09 13:46:21 jetson-j1-32gb-cam kernel: x9 : ffffffc753f7f9a0 x8 : 20747865746e6f63 
nov 09 13:46:21 jetson-j1-32gb-cam kernel: x7 : 206c656e6e616863 x6 : ffffff800a12713c 
nov 09 13:46:21 jetson-j1-32gb-cam kernel: x5 : 0000000000000000 x4 : 0000000000000000 
nov 09 13:46:21 jetson-j1-32gb-cam kernel: x3 : ffffffffffffffff x2 : 00000047f6511000 
nov 09 13:46:21 jetson-j1-32gb-cam kernel: x1 : ffffffc7d1214600 x0 : 000000000000001c 
nov 09 13:46:21 jetson-j1-32gb-cam kernel: 
nov 09 13:46:21 jetson-j1-32gb-cam kernel: ---[ end trace 55d8e3ad7d808e94 ]---
nov 09 13:46:21 jetson-j1-32gb-cam kernel: Call trace:
nov 09 13:46:21 jetson-j1-32gb-cam kernel: [<ffffff8008c93724>] tegra_capture_ivc_notify_chan_id+0x184/0x1a0
nov 09 13:46:21 jetson-j1-32gb-cam kernel: [<ffffff8008b3f4e0>] vi_capture_setup+0x340/0x4d0
nov 09 13:46:21 jetson-j1-32gb-cam kernel: [<ffffff8008b40518>] vi_channel_ioctl+0x278/0x5e8
nov 09 13:46:21 jetson-j1-32gb-cam kernel: [<ffffff8008273188>] do_vfs_ioctl+0xb0/0x8d8
nov 09 13:46:21 jetson-j1-32gb-cam kernel: [<ffffff8008273a3c>] SyS_ioctl+0x8c/0xa8
nov 09 13:46:21 jetson-j1-32gb-cam kernel: [<ffffff80080838c0>] el0_svc_naked+0x34/0x38
nov 09 13:46:21 jetson-j1-32gb-cam kernel: tegra194-vi5 15c10000.vi: failed to update control callback
nov 09 13:46:22 jetson-j1-32gb-cam kernel: tegra194-vi5 15c10000.vi: no reply from camera processor
nov 09 13:46:22 jetson-j1-32gb-cam kernel: tegra194-vi5 15c10000.vi: vi capture setup failed
nov 09 13:46:23 jetson-j1-32gb-cam kernel: tegra194-vi5 15c10000.vi: no reply from camera processor
nov 09 13:46:24 jetson-j1-32gb-cam kernel: tegra194-vi5 15c10000.vi: no reply from camera processor
nov 09 13:46:24 jetson-j1-32gb-cam kernel: Unable to handle kernel paging request at virtual address dead000000000200
nov 09 13:46:24 jetson-j1-32gb-cam kernel: Mem abort info:
nov 09 13:46:24 jetson-j1-32gb-cam kernel:   ESR = 0x96000004
nov 09 13:46:24 jetson-j1-32gb-cam kernel:   Exception class = DABT (current EL), IL = 32 bits
nov 09 13:46:24 jetson-j1-32gb-cam kernel:   SET = 0, FnV = 0
nov 09 13:46:24 jetson-j1-32gb-cam kernel:   EA = 0, S1PTW = 0
nov 09 13:46:24 jetson-j1-32gb-cam kernel: Data abort info:
nov 09 13:46:24 jetson-j1-32gb-cam kernel:   ISV = 0, ISS = 0x00000004
nov 09 13:46:24 jetson-j1-32gb-cam kernel:   CM = 0, WnR = 0
nov 09 13:46:24 jetson-j1-32gb-cam kernel: [dead000000000200] address between user and kernel address ranges
nov 09 13:46:24 jetson-j1-32gb-cam kernel: Internal error: Oops: 96000004 [#1] PREEMPT SMP
nov 09 13:46:24 jetson-j1-32gb-cam kernel: Modules linked in: fuse bnep zram overlay spidev nvgpu bluedroid_pm ip_tables x_tables
nov 09 13:46:24 jetson-j1-32gb-cam kernel: CPU: 3 PID: 15868 Comm: camera-handler Tainted: G        W       4.9.140-tegra #1
nov 09 13:46:24 jetson-j1-32gb-cam kernel: Hardware name: Jetson-AGX (DT)
nov 09 13:46:24 jetson-j1-32gb-cam kernel: task: ffffffc7d1214600 task.stack: ffffffc753f7c000
nov 09 13:46:24 jetson-j1-32gb-cam kernel: PC is at dma_buf_unmap_attachment+0x28/0x140
nov 09 13:46:24 jetson-j1-32gb-cam kernel: LR is at capture_common_unpin_memory+0x30/0x60
nov 09 13:46:24 jetson-j1-32gb-cam kernel: pc : [<ffffff80087d3a00>] lr : [<ffffff8008b4c4b0>] pstate: 20400045
nov 09 13:46:24 jetson-j1-32gb-cam kernel: sp : ffffffc753f7fcd0
nov 09 13:46:24 jetson-j1-32gb-cam kernel: x29: ffffffc753f7fcd0 x28: ffffffc7d1214600 
nov 09 13:46:24 jetson-j1-32gb-cam kernel: x27: ffffff8008f62000 x26: 0000000000000039 
nov 09 13:46:24 jetson-j1-32gb-cam kernel: x25: 0000000000000020 x24: 0000000000000008 
nov 09 13:46:24 jetson-j1-32gb-cam kernel: x23: ffffffc65e3b0400 x22: 0000000000000000 
nov 09 13:46:24 jetson-j1-32gb-cam kernel: x21: 0000000000000000 x20: 0000238c00000000 
nov 09 13:46:24 jetson-j1-32gb-cam kernel: x19: dead000000000200 x18: 0000000000000010 
nov 09 13:46:24 jetson-j1-32gb-cam kernel: x17: 0000007fa680bb00 x16: 0000000000000000 
nov 09 13:46:24 jetson-j1-32gb-cam kernel: x15: ffffffffffffffff x14: 0a726f737365636f 
nov 09 13:46:24 jetson-j1-32gb-cam kernel: x13: 7270206172656d61 x12: 63206d6f72662079 
nov 09 13:46:24 jetson-j1-32gb-cam kernel: x11: 6c706572206f6e20 x10: 000000000000053b 
nov 09 13:46:24 jetson-j1-32gb-cam kernel: x9 : 3163353120356976 x8 : ffffff80083d47b0 
nov 09 13:46:24 jetson-j1-32gb-cam kernel: x7 : ffffff8009e94358 x6 : ffffffc7ffd1abf0 
nov 09 13:46:24 jetson-j1-32gb-cam kernel: x5 : ffffffc7ffd1abf0 x4 : 0000000000000000 
nov 09 13:46:24 jetson-j1-32gb-cam kernel: x3 : 0000238c00000000 x2 : 0000000000000000 
nov 09 13:46:24 jetson-j1-32gb-cam kernel: x1 : 0000238c00000000 x0 : ffffff8008b4c4b0 
nov 09 13:46:24 jetson-j1-32gb-cam kernel: 
nov 09 13:46:24 jetson-j1-32gb-cam kernel: Process camera-handler (pid: 15868, stack limit = 0xffffffc753f7c000)
nov 09 13:46:24 jetson-j1-32gb-cam kernel: Call trace:
nov 09 13:46:24 jetson-j1-32gb-cam kernel: [<ffffff80087d3a00>] dma_buf_unmap_attachment+0x28/0x140
nov 09 13:46:24 jetson-j1-32gb-cam kernel: [<ffffff8008b4c4b0>] capture_common_unpin_memory+0x30/0x60
nov 09 13:46:24 jetson-j1-32gb-cam kernel: [<ffffff8008b3ff60>] vi_capture_request_unpin+0x78/0xb8
nov 09 13:46:24 jetson-j1-32gb-cam kernel: [<ffffff8008b3f134>] vi_capture_shutdown+0xac/0x118
nov 09 13:46:24 jetson-j1-32gb-cam kernel: [<ffffff8008b40044>] vi_channel_close_ex+0x2c/0x88
nov 09 13:46:24 jetson-j1-32gb-cam kernel: [<ffffff8008b400cc>] vi_channel_release+0x2c/0x40
nov 09 13:46:24 jetson-j1-32gb-cam kernel: [<ffffff800825fc00>] __fput+0x90/0x1d0
nov 09 13:46:24 jetson-j1-32gb-cam kernel: [<ffffff800825fdb8>] ____fput+0x20/0x30
nov 09 13:46:24 jetson-j1-32gb-cam kernel: [<ffffff80080d9bf4>] task_work_run+0xbc/0xd8
nov 09 13:46:24 jetson-j1-32gb-cam kernel: [<ffffff800808b6ac>] do_notify_resume+0xa4/0xb0
nov 09 13:46:24 jetson-j1-32gb-cam kernel: [<ffffff8008083754>] work_pending+0x8/0x10
nov 09 13:46:24 jetson-j1-32gb-cam kernel: ---[ end trace 55d8e3ad7d808e95 ]---
nov 09 13:46:24 jetson-j1-32gb-cam systemd[1]: camera-handler.service: Main process exited, code=killed, status=11/SEGV
nov 09 13:46:24 jetson-j1-32gb-cam systemd[1]: camera-handler.service: Failed with result 'signal'.
nov 09 13:46:25 jetson-j1-32gb-cam systemd[1]: camera-handler.service: Service hold-off time over, scheduling restart.
nov 09 13:46:25 jetson-j1-32gb-cam systemd[1]: camera-handler.service: Scheduled restart job, restart counter is at 42.
nov 09 13:46:25 jetson-j1-32gb-cam systemd[1]: Stopped Camera Handler.
nov 09 13:46:25 jetson-j1-32gb-cam systemd[1]: Starting Camera Handler...
nov 09 13:46:26 jetson-j1-32gb-cam systemd[1]: Started Camera Handler.
nov 09 13:46:28 jetson-j1-32gb-cam kernel: channel context at 4 is busy
nov 09 13:46:28 jetson-j1-32gb-cam kernel: ------------[ cut here ]------------
nov 09 13:46:28 jetson-j1-32gb-cam kernel: WARNING: CPU: 4 PID: 15988 at /home/simon/R32.3.1/kernel/nvidia/drivers/platform/tegra/rtcpu/capture-ivc.c:224 tegra_capture_ivc_notify_chan_id+0x184/0x1a0
nov 09 13:46:28 jetson-j1-32gb-cam kernel: Modules linked in: fuse bnep zram overlay spidev nvgpu bluedroid_pm ip_tables x_tables
nov 09 13:46:28 jetson-j1-32gb-cam kernel: 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: CPU: 4 PID: 15988 Comm: camera-handler Tainted: G      D W       4.9.140-tegra #1
nov 09 13:46:28 jetson-j1-32gb-cam kernel: Hardware name: Jetson-AGX (DT)
nov 09 13:46:28 jetson-j1-32gb-cam kernel: task: ffffffc7d1217000 task.stack: ffffffc6c916c000
nov 09 13:46:28 jetson-j1-32gb-cam kernel: PC is at tegra_capture_ivc_notify_chan_id+0x184/0x1a0
nov 09 13:46:28 jetson-j1-32gb-cam kernel: LR is at tegra_capture_ivc_notify_chan_id+0x184/0x1a0
nov 09 13:46:28 jetson-j1-32gb-cam kernel: pc : [<ffffff8008c93724>] lr : [<ffffff8008c93724>] pstate: 60400045
nov 09 13:46:28 jetson-j1-32gb-cam kernel: sp : ffffffc6c916fba0
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x29: ffffffc6c916fba0 x28: ffffffc7d1217000 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x27: ffffff8008f62000 x26: 000000000000001d 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x25: 0000000000000043 x24: ffffffc7d5658020 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x23: 0000000000000004 x22: ffffffc7d5658d98 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x21: 000000000000006c x20: 0000000000000d80 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x19: ffffffc7d5658018 x18: 0000000000000000 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x17: 0000007f892bab00 x16: ffffff80082739b0 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x15: ffffffffffffffff x14: ffffffc74916f4df 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x13: ffffffc6c916f4e2 x12: 071c71c71c71c71c 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x11: ffffffc6c916f4a0 x10: 000000000000056c 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x9 : 00000000ffffffd0 x8 : 6220736920342074 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x7 : 6120747865746e6f x6 : ffffff80083d3ba0 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x5 : 0000000000000000 x4 : 0000000000000000 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x3 : ffffffffffffffff x2 : 00000047f652a000 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x1 : ffffffc7d1217000 x0 : 000000000000001c 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: ---[ end trace 55d8e3ad7d808e96 ]---
nov 09 13:46:28 jetson-j1-32gb-cam kernel: Call trace:
nov 09 13:46:28 jetson-j1-32gb-cam kernel: [<ffffff8008c93724>] tegra_capture_ivc_notify_chan_id+0x184/0x1a0
nov 09 13:46:28 jetson-j1-32gb-cam kernel: [<ffffff8008b3f4e0>] vi_capture_setup+0x340/0x4d0
nov 09 13:46:28 jetson-j1-32gb-cam kernel: [<ffffff8008b40518>] vi_channel_ioctl+0x278/0x5e8
nov 09 13:46:28 jetson-j1-32gb-cam kernel: [<ffffff8008273188>] do_vfs_ioctl+0xb0/0x8d8
nov 09 13:46:28 jetson-j1-32gb-cam kernel: [<ffffff8008273a3c>] SyS_ioctl+0x8c/0xa8
nov 09 13:46:28 jetson-j1-32gb-cam kernel: [<ffffff80080838c0>] el0_svc_naked+0x34/0x38
nov 09 13:46:28 jetson-j1-32gb-cam kernel: tegra194-vi5 15c10000.vi: failed to update control callback
nov 09 13:46:28 jetson-j1-32gb-cam kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000
nov 09 13:46:28 jetson-j1-32gb-cam kernel: Mem abort info:
nov 09 13:46:28 jetson-j1-32gb-cam kernel:   ESR = 0x96000005
nov 09 13:46:28 jetson-j1-32gb-cam kernel:   Exception class = DABT (current EL), IL = 32 bits
nov 09 13:46:28 jetson-j1-32gb-cam kernel:   SET = 0, FnV = 0
nov 09 13:46:28 jetson-j1-32gb-cam kernel:   EA = 0, S1PTW = 0
nov 09 13:46:28 jetson-j1-32gb-cam kernel: Data abort info:
nov 09 13:46:28 jetson-j1-32gb-cam kernel:   ISV = 0, ISS = 0x00000005
nov 09 13:46:28 jetson-j1-32gb-cam kernel:   CM = 0, WnR = 0
nov 09 13:46:28 jetson-j1-32gb-cam kernel: user pgtable: 4k pages, 39-bit VAs, pgd = ffffffc71aed5000
nov 09 13:46:28 jetson-j1-32gb-cam kernel: [0000000000000000] *pgd=0000000000000000, *pud=0000000000000000
nov 09 13:46:28 jetson-j1-32gb-cam kernel: Internal error: Oops: 96000005 [#2] PREEMPT SMP
nov 09 13:46:28 jetson-j1-32gb-cam kernel: Modules linked in: fuse bnep zram overlay spidev nvgpu bluedroid_pm ip_tables x_tables
nov 09 13:46:28 jetson-j1-32gb-cam kernel: CPU: 3 PID: 14377 Comm: kworker/3:3 Tainted: G      D W       4.9.140-tegra #1
nov 09 13:46:28 jetson-j1-32gb-cam kernel: Hardware name: Jetson-AGX (DT)
nov 09 13:46:28 jetson-j1-32gb-cam kernel: Workqueue: events tegra_capture_ivc_worker
nov 09 13:46:28 jetson-j1-32gb-cam kernel: task: ffffffc7da07c600 task.stack: ffffffc705138000
nov 09 13:46:28 jetson-j1-32gb-cam kernel: PC is at __wake_up_common+0x38/0xa0
nov 09 13:46:28 jetson-j1-32gb-cam kernel: LR is at __wake_up_locked+0x3c/0x50
nov 09 13:46:28 jetson-j1-32gb-cam kernel: pc : [<ffffff8008111ad0>] lr : [<ffffff8008111be4>] pstate: a0c000c5
nov 09 13:46:28 jetson-j1-32gb-cam kernel: sp : ffffffc70513bc60
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x29: ffffffc70513bc60 x28: 0000000000000000 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x27: 0000000000000000 x26: ffffffc7ffd23540 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x25: ffffff800a079000 x24: 0000000000000001 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x23: 0000000000000003 x22: 0000000000000000 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x21: 0000000000000000 x20: ffffffc6e61f44e0 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x19: ffffffc6e61f44d8 x18: 0000000000000b7d 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x17: 0000007f8ce072e8 x16: 0000000000000000 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x15: 000000000001c9f2 x14: 0000000000000000 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x13: 0000000000000000 x12: 0000000000000000 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x11: 0000000000000000 x10: 0000000000000000 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x9 : 0000000000000000 x8 : 0000000000000000 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x7 : 0000000000000000 x6 : ffffffc6e61f4640 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x5 : ffffff80080a0db8 x4 : 0000000000000000 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x3 : 0000000000000000 x2 : 0000000000000001 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: x1 : 0000000000000003 x0 : 0000000000000000 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: 
nov 09 13:46:28 jetson-j1-32gb-cam kernel: Process kworker/3:3 (pid: 14377, stack limit = 0xffffffc705138000)
nov 09 13:46:28 jetson-j1-32gb-cam kernel: Call trace:
nov 09 13:46:28 jetson-j1-32gb-cam kernel: [<ffffff8008111ad0>] __wake_up_common+0x38/0xa0
nov 09 13:46:28 jetson-j1-32gb-cam kernel: [<ffffff8008111be4>] __wake_up_locked+0x3c/0x50
nov 09 13:46:28 jetson-j1-32gb-cam kernel: [<ffffff8008112838>] complete+0x48/0x68
nov 09 13:46:28 jetson-j1-32gb-cam kernel: [<ffffff8008b3e400>] vi_capture_ivc_control_callback+0x90/0xe8
nov 09 13:46:28 jetson-j1-32gb-cam kernel: [<ffffff8008c93d14>] tegra_capture_ivc_worker+0x84/0xe8
nov 09 13:46:28 jetson-j1-32gb-cam kernel: [<ffffff80080d4f3c>] process_one_work+0x1e4/0x4b0
nov 09 13:46:28 jetson-j1-32gb-cam kernel: [<ffffff80080d5258>] worker_thread+0x50/0x4c8
nov 09 13:46:28 jetson-j1-32gb-cam kernel: [<ffffff80080dbee4>] kthread+0xec/0xf0
nov 09 13:46:28 jetson-j1-32gb-cam kernel: [<ffffff8008083850>] ret_from_fork+0x10/0x40
nov 09 13:46:28 jetson-j1-32gb-cam kernel: ---[ end trace 55d8e3ad7d808e97 ]---
nov 09 13:46:28 jetson-j1-32gb-cam kernel: note: kworker/3:3[14377] exited with preempt_count 1

after reboot:

[    0.931596] tegra-pmc: ### PMC reset source: TEGRA_SOFTWARE_RESET
[    0.931610] tegra-pmc: ### PMC reset level: TEGRA_RESET_LEVEL_L1
[    0.931617] tegra-pmc: ### PMC reset status reg: 0x2d

Can anyone help with this ?

3

Hi,

We are experiencing similar issues on JP4.3 & JP4.4 with imx577 cameras from Leopard Imaging. We’re running 4x4k@50 and 4x4k@60. We’re not using Argus though.

Frans

4

It’s better consult with Leopard to confirm the sensor driver.

5

Hello,

  1. As I known Argus api is non open source so can you say what "(NvCapture) Error InvalidState: Free request list is empty! " error mean and how to fix it ?
  2. L4T as I suppose have have some watchdog that make this sporadic TEGRA_SOFTWARE_RESET. Is there a way to find what and why system make it ? More often reset happen without any tracelog.
  3. Can you confirm that backtrace above with “Unable to handle kernel paging request at virtual address dead000000000200” happened because of bugs in camera driver ?
6

Those problem could be the sensor signal or something wrong cause the NVCSI/VI capture failed and timeout.

7

Hi villy21,
Have you tried saving the videos to Xavier eMMC instead of SSD? Do you see the same issue?

8

Hi fransklaver,
Could you provide a little more details about the issue? Which driver are you using (the name of the driver guide)? How many cameras are you using in the video recording? Do you save the videos to SSD or Xavier eMMC?

9

@villy21
Could you confirm the issue relate with stored the image to SSD. Could you try if problem with preview only?

10

Store on ssd only speedup the bug.
We reproduce the same problem without store any data at all.
When store to ssd video from 3 camera then self reboot can happen after 1 minute from start.
When we only grab data from 3 camera via argus api and did not do anything else self reboot happen after 1 hour.
Even if we grab data from only 1 camera we get self reboot after 19 hours.

And after reboot in dmesg we always get TEGRA_SOFTWARE_RESET.

Is there a way to check that this is driver or hardware hangup?

11

What’s the version? J4.4?

12

No, we use JP4.3 32.3.1
There is no imx334 driver for 32.4.4

13

@SimonZhu
Could you share the driver for r32.4.3 to verify.

14

Hi ShaneCCC,
We don’t have R32.4.3 driver yet. The current IMX334 Xavier driver is based on R32.3.1 (JP4.3).
Please download the driver binary and patch files below.

We actually recorded the videos from 3 cameras simultaneously and saved them to Xavier eMMC for 50 minutes, but we could not reproduce the error or self reboot issue.

15

Problem happen not only with our app but with regular argus_camera too.
Here is the command to run argus_camera and some logs after this.

jetson_imx334_3argus_camera_log.txt (424.2 KB) argus_camera_record.sh.txt (934 Bytes)

16

@ShaneCCC @SimonZhu
Did someone solve this problem?
It still happens.

17

@chaekyunghwa11 Which driver are you using? Is it the same IMX334 Xavier JP4.3 driver?

18

@SimonZhu
I’m using LI-IMX412-MIPI-CS in Xavier. But I don’t know exact installed version in Ubuntu18.04.

19

@chaekyunghwa11
Do you still have our driver guide? If so, please download the OS image from the driver guide and re-flash it to Xavier, then re-install the driver to see if the issue is still there.
We could not reproduce this issue on our side when we tested the Xavier IMX334 driver before.

20

@SimonZhu
Can I know when the driver for IMX412 is updated?
We installed the driver in Xavier at a year ago.

21

@chaekyunghwa11
We have different adapter boards for IMX412 and Xavier? Which one are you using (4CAM or 6CAM)? If possible, could you take a picture for us about the whole camera kit?