Hi,
Currently we are encountering an issue with a Jetson Nano with two Raspberry Pi Camera Module v2 cameras connected. When running the deepstream-app in an environment with temperatures between 0 - 10°C we get following errors
Jan 05 22:02:06 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:06 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:06 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:06 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:06 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:06 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:06 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:06 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:06 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:06 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:06 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
...
Almost 1000 messages in the same second, and
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 639 kernel messages
Jan 05 22:02:05 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 33 kernel messages
Jan 05 22:02:05 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 39 kernel messages
Jan 05 22:02:05 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 34 kernel messages
Jan 05 22:02:05 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 36 kernel messages
Jan 05 22:02:05 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 40 kernel messages
Jan 05 22:02:05 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 35 kernel messages
Jan 05 22:02:05 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 36 kernel messages
Jan 05 22:02:05 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 38 kernel messages
Jan 05 22:02:05 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 45 kernel messages
Jan 05 22:02:05 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 44 kernel messages
Jan 05 22:02:05 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 33 kernel messages
Jan 05 22:02:05 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 40 kernel messages
Jan 05 22:02:05 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 36 kernel messages
Jan 05 22:02:05 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 33 kernel messages
Jan 05 22:02:05 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 40 kernel messages
Jan 05 22:02:05 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 42 kernel messages
Jan 05 22:02:05 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 120 kernel messages
Jan 05 22:02:05 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 39 kernel messages
Jan 05 22:02:05 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 37 kernel messages
Jan 05 22:02:05 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 32 kernel messages
Jan 05 22:02:05 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 30 kernel messages
Jan 05 22:02:05 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 36 kernel messages
...
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 36 kernel messages
Jan 05 22:02:05 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 64 kernel messages
Jan 05 22:02:05 5498c6a kernel: host1x 50000000.host1x: syncpt_thresh_cascade_isr(): syncpoint id 0 incremented
Jan 05 22:02:05 5498c6a systemd-journald[2376]: Missed 622 kernel messages
More than 2000 messages in the previous second
And following error:
06.01.21 15:08:36 (+0100) deepstream SCF: Error Timeout: ISP port 0 timed out! (in src/services/capture/NvIspHw.cpp, function waitIspFrameEnd(), line 478)
06.01.21 15:08:36 (+0100) deepstream SCF: Error Timeout: (propagating from src/services/capture/NvIspHw.cpp, function waitIspFrameEnd(), line 519)
06.01.21 15:08:36 (+0100) deepstream SCF: Error Timeout: (propagating from src/common/Utils.cpp, function workerThread(), line 116)
06.01.21 15:08:36 (+0100) deepstream SCF: Error Timeout: Worker thread IspHw frameComplete failed (in src/common/Utils.cpp, function workerThread(), line 133)
06.01.21 15:08:36 (+0100) deepstream Error: waitCsiFrameStart timeout guid 1
06.01.21 15:08:36 (+0100) deepstream ************VI/CSI Debug Registers**********
06.01.21 15:08:36 (+0100) deepstream SCF: Error Timeout: ISP Stats timed out! (in src/services/capture/NvIspHw.cpp, function waitIspStatsFinished(), line 561)
06.01.21 15:08:36 (+0100) deepstream VI_CFG_INTERRUPT_MASK_0 = 0x00000000
06.01.21 15:08:36 (+0100) deepstream VI_CFG_INTERRUPT_STATUS_0 = 0x00000000
06.01.21 15:08:36 (+0100) deepstream VI_CSI_0_ERROR_STATUS_0 = 0x00000001
06.01.21 15:08:36 (+0100) deepstream VI_CSI_0_ERROR_INT_MASK_0 = 0x0000001f
Between the errors reboots are happened but it’s all on the same hardware, we have other devices running the same software without this issue. It looks like the issue happens more frequently in colder temperatures. Above 10 °C we didn’t encounter the issue a lot.
Does anybody has an idea what could be wrong and how to debug this issue?
Kind regards,
Clint