Rtcpu_trace_isp_event event id 20 cannot be found in tracing

Hi,

I am debugging my camera drivers, by enabling tracing like so:

echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 2 > /sys/kernel/debug/camrtc/log-level
echo 1 > /sys/kernel/debug/tracing/events/camera_common/enable
echo > /sys/kernel/debug/tracing/trace

When I have my camera feed running, this outputs the following in dmesg constantly:

May 18 08:23:11 nvidia-desktop kernel: [  407.032651] rtcpu_trace_isp_event event id 20 cannot be found
May 18 08:23:11 nvidia-desktop kernel: [  407.038426] rtcpu_trace_isp_event event id 20 cannot be found
May 18 08:23:11 nvidia-desktop kernel: [  407.044205] rtcpu_trace_isp_event event id 20 cannot be found
May 18 08:23:11 nvidia-desktop kernel: [  407.049741] rtcpu_trace_isp_event event id 20 cannot be found
May 18 08:23:11 nvidia-desktop kernel: [  407.055498] rtcpu_trace_isp_event event id 20 cannot be found
May 18 08:23:11 nvidia-desktop kernel: [  407.061290] rtcpu_trace_isp_event event id 20 cannot be found
...

The actual tracing buffer seems pretty normal

kworker/1:2-1721  [001] ....   280.300565: rtos_queue_send_from_isr_failed: tstamp:8892240445 queue:0x0bcba5e0
kworker/1:2-1721  [001] ....   280.300566: rtos_queue_send_from_isr_failed: tstamp:8892240603 queue:0x0bcbb3a0
kworker/1:2-1721  [001] ....   280.300568: rtos_queue_send_from_isr_failed: tstamp:8892240757 queue:0x0bcbc160
kworker/1:2-1721  [001] ....   280.300569: rtcpu_vinotify_event: tstamp:8892261473 tag:FS channel:0x01 frame:2394 vi_tstamp:8892071068 data:0x00000012
kworker/1:2-1721  [001] ....   280.300570: rtcpu_vinotify_event: tstamp:8892261631 tag:ATOMP_FS channel:0x00 frame:2394 vi_tstamp:8892071068 data:0x20000000
kworker/1:2-1721  [001] ....   280.300572: rtcpu_vinotify_event: tstamp:8892261810 tag:CHANSEL_PXL_SOF channel:0x1d frame:2394 vi_tstamp:8892082980 data:0x00000001
kworker/1:2-1721  [001] ....   280.300573: rtcpu_vinotify_event: tstamp:8892261980 tag:RESERVED_19 channel:0x1d frame:90 vi_tstamp:9668748960 data:0x08020897
kworker/1:2-1721  [001] ....   280.300574: rtcpu_vinotify_event: tstamp:8892262151 tag:RESERVED_18 channel:0x1d frame:0 vi_tstamp:9668807392 data:0x10000000
kworker/1:2-1721  [001] ....   280.300576: rtcpu_vinotify_event: tstamp:8892262297 tag:RESERVED_18 channel:0x1d frame:0 vi_tstamp:9668811744 data:0x31000898
kworker/1:2-1721  [001] ....   280.300577: rtcpu_vinotify_event: tstamp:8892262468 tag:CHANSEL_PXL_EOF channel:0x21 frame:2409 vi_tstamp:8892183038 data:0x031f0002
kworker/1:2-1721  [001] ....   280.300578: rtcpu_vinotify_event: tstamp:8892262613 tag:ATOMP_FRAME_DONE channel:0x21 frame:2409 vi_tstamp:8892183054 data:0x00000000
kworker/1:2-1721  [001] ....   280.300580: rtcpu_vinotify_event: tstamp:8892262782 tag:RESERVED_19 channel:0x21 frame:105 vi_tstamp:9671951232 data:0x0202089a
kworker/1:2-1721  [001] ....   280.300581: rtcpu_vinotify_event: tstamp:8892262929 tag:FE channel:0x03 frame:2409 vi_tstamp:8892183141 data:0x00000020
kworker/1:2-1721  [001] ....   280.300582: rtcpu_vinotify_event: tstamp:8892263103 tag:ATOMP_FE channel:0x00 frame:2409 vi_tstamp:8892183141 data:0x00000000
kworker/1:2-1721  [001] ....   280.300583: rtcpu_vinotify_event: tstamp:8892263247 tag:RESERVED_19 channel:0x21 frame:105 vi_tstamp:9671955296 data:0x0002089a
kworker/1:2-1721  [001] ....   280.300585: rtcpu_vinotify_event: tstamp:8892263416 tag:RESERVED_19 channel:0x21 frame:0 vi_tstamp:9671958080 data:0x0702089b
kworker/1:2-1721  [001] ....   280.300586: rtcpu_vinotify_event: tstamp:8892263562 tag:CHANSEL_PXL_EOF channel:0x20 frame:2346 vi_tstamp:8892260456 data:0x031f0002
kworker/1:2-1721  [001] ....   280.300587: rtcpu_vinotify_event: tstamp:8892263737 tag:ATOMP_FRAME_DONE channel:0x20 frame:2346 vi_tstamp:8892260471 data:0x00000000
kworker/1:2-1721  [001] ....   280.300589: rtcpu_vinotify_event: tstamp:8892263881 tag:RESERVED_19 channel:0x20 frame:42 vi_tstamp:9674428576 data:0x0202089e
kworker/1:2-1721  [001] ....   280.300591: rtos_queue_send_from_isr_failed: tstamp:8892301060 queue:0x0bcb41f8
kworker/1:2-1721  [001] ....   280.300592: rtos_queue_send_from_isr_failed: tstamp:8892301224 queue:0x0bcb8a60
kworker/1:2-1721  [001] ....   280.300593: rtos_queue_send_from_isr_failed: tstamp:8892301386 queue:0x0bcba5e0
kworker/1:2-1721  [001] ....   280.300595: rtos_queue_send_from_isr_failed: tstamp:8892301542 queue:0x0bcbb3a0

If the fps of the video stream is high enough, enabling tracing even halts the videostream, seeming that some resources are drained to to enabling this.

My questions:

  • Am I enabling the correct traces?
  • Should I be worried about the rtcpu trace event id 20? Or should I just disable rtcpu tracing?
  • Is it normal that enabling tracing can kill a video stream?

Thanks for your feedback

  1. That’s correct to enable the trace.
  2. It’s just a warning message it could be harmless.
  3. Didn’t have this kind of experience. Have you try to boost the system and vi/nvcsi/isp clock to try.

https://elinux.org/Jetson_TX2_Camera_BringUp

Hi,

Regarding the clocks, still have the issue that enabling traces freezes the stream. I do this

sudo su
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee  /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate

If I lower the fps, enabling the traces does not halt the stream, or it takes longer to do so.

I’ve seen this come by when the stream failed (but not sure if it is a result or gives more info about the cause)

(NvCapture) Error InvalidState: Free request list is empty! (in /dvs/git/dirty/git-master_linux/camera/capture/nvcapture/capture.c, function NvCaptureGetRequest(), line 706)
SCF: Error InvalidState:  (propagating from src/services/capture/NvCaptureViCsiHw.cpp, function startCaptureInternal(), line 597)
SCF: Error InvalidState:  (propagating from src/services/capture/CaptureRecord.cpp, function doCSItoMemCapture(), line 517)
SCF: Error InvalidState:  (propagating from src/services/capture/CaptureRecord.cpp, function issueCapture(), line 454)
SCF: Error InvalidState:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 1276)
SCF: Error InvalidState:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueBubbleFillCapturesIfNeeded(), line 676)
SCF: Error InvalidState:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 1118)
SCF: Error InvalidState:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)
SCF: Error InvalidState: Worker thread CaptureScheduler frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)

What’s the resolution and frame rate?

8x 1280x800, 30fps, 10bit

Do you mean launch 8 sensors?

Yes

Was there a resolution to tracing printing the following msg so fast it stalls the whole system? We’re running Xavier with 8 cameras and JetPack 4.6 and need to enable tracing for debugging.

rtcpu_trace_isp_event event id 20 cannot be found

Don’t struggle on this message. Have a check the wiki to check the trace log.

Even if the msg isn’t important, it is printed so fast that it takes up a lot of cpu and will make the system unresponsive when the combination of number of cameras and frame rate is high. It’s the:

echo 2 > /sys/kernel/debug/camrtc/log-level

specifically that causes it. Leaving that at 0 or 1 stops the msg from printing. What is the difference between log-level 1 and log-level 2? What output is lost going to 1?

Enable this log for sensor bring up only. Usually disable it in normal case.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.