[kernel panic] failed to update control callback

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
1

Hi guys,

Our customized AGX Orin product supports 16 GMSL2 cameras.
I encounter an issue that kernel panic sometimes happens while I start stream and then at the same time stop stream quickly.

  • Env: Orin 64G with L4T R35.4.1

  • RCE firmware:
    Topic268833_JP-512_rce-fw.7z
    or
    Topic268833_JP-512_rce-fw_Debug.7z

  • Camera device nodes:

    NVIDIA Tegra Video Input Device (platform:tegra-camrtc-ca):
        /dev/media0

vi-output, gmsl2_sensor 2-001b (platform:tegra-capture-vi:0):
        /dev/video0
        /dev/video1
        /dev/video2
        /dev/video3

vi-output, gmsl2_sensor 2-002b (platform:tegra-capture-vi:2):
        /dev/video4
        /dev/video5
        /dev/video6
        /dev/video7

vi-output, gmsl2_sensor 7-003b (platform:tegra-capture-vi:4):
        /dev/video8
        /dev/video9
        /dev/video10
        /dev/video11

vi-output, gmsl2_sensor 7-005b (platform:tegra-capture-vi:5):
        /dev/video12
        /dev/video13
        /dev/video14
        /dev/video15

  • /dev/videoN format:

    ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture

        [0]: 'UYVY' (UYVY 4:2:2)
                Size: Discrete 2880x1860
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 3840x2160
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 1920x1536
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 1600x1300
                        Interval: Discrete 0.033s (30.000 fps)
        [1]: 'NV16' (Y/CbCr 4:2:2)
                Size: Discrete 2880x1860
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 3840x2160
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 1920x1536
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 1600x1300
                        Interval: Discrete 0.033s (30.000 fps)
        [2]: 'UYVY' (UYVY 4:2:2)
                Size: Discrete 2880x1860
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 3840x2160
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 1920x1536
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 1600x1300
                        Interval: Discrete 0.033s (30.000 fps)

  • Reproduce SOP:

    • Run start_test.sh
      start_test.zip (651 Bytes)
    • Durng the camera streams of all the /dev/videoN are opening, immediately close or press Ctrl-C to stop the opened terminals.
    • Loop this SOP fews times, the kernel panic happens.

Note: Kernel panic happens no matter the camera are acctually connected or not.

3

hello ting.chang,

would like to narrow down the issue,
is it the only approach to reproduce kernel panic by terminated the process during camera initialization?
if yes, you may give it a try with race condition fix as mentioned in Topic 258971.
thanks

4

Hi Jerry,

I’ve tried the semaphore fix, but it gets worse…
The exception happens everytime I start the stream.

The attachment is the log that I first opened /dev/video0 and then opened /dev/video1 after 130 seconds.
dmesg_kernel_panic_new1.txt (7.5 KB)

5

hello ting.chang,

am still wondering your repo steps,
for instance, as mentioned in previous comments, is it the only approach to reproduce kernel panic by terminated the process during camera initialization?

6

Yes, currently this is the only way we reproduce kernel panic, I don’t know if there are other approches to reproduce this issue.

7

hello ting.chang,

here’s change to remove redundant vi_capture_release, and also an updated API to close VI channel,
please also check whether this is included in the r35.4.1 release branch, and please give it a try,
for instance, 0001-camera-vi5-fix-stream-on-off-memory-leakage.patch (5.7 KB)

8

Hi Jerry,

I’ve tried the memory-leakage.patch, but it gets worse, too.

Can you reproduce this issue by using my simple driver and test scripts?
I can reproduce kernel panic by running video_kill_open.sh without inserting any cameras.
debug_kernel_panic.zip (8.2 KB)

9

hello ting.chang,

actually, what’s the real use-case to terminated the process during camera initialization?

10

It’s not a real use-case, I just encounter this issue sometimes during camera driver development.

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.