We use the GMSL camera in the Thor EA version, and a kernel panic problem caused by VI timeout occurs during the open close test.
This phenomenon has also been repeated many times in Orin：
We use a GMSL camera. It is possible that the camera hardware is not in good contact, vi reports a timeout error, and triggering revovery fails. However, it should not cause a kernel panic.
Jul 31 03:18:31 mi-desktop kernel: [22961.465435] tegra194-vi5 13e40000.host1x:vi1@14c00000: capture control message timed out
Jul 31 03:18:31 mi-desktop kernel: [22961.465668] tegra194-vi5 13e40000.host1x:vi1@14c00000: vi_capture_release: release channel IVC failed
Jul 31 03:18:31 mi-desktop kernel: [2296…
[98680.434211] ox05b1s 2-001b: ox05b1s_stop_streaming#9
uart_dmesg.log (9.6 MB)
Another log entry shows that even though the VI reset was successful, it still caused a kernel panic.
thor_log_er.zip (773.7 KB)
hello deeptalkcamera,
please moving to JP-7.0/r38.2 for development.
We are pleased to announce the production release of JetPack 7.0. JetPack 7.0 is a major upgrade in the JetPack series, supporting the NVIDIA Jetson AGX Thor Developer Kit and the Thor based T5000 module. With JetPack 7, Jetson software aligns with the Server Base System Architecture (SBSA), positioning Jetson Thor alongside industry-standard ARM server design. JetPack 7.0 packages Jetson Linux 38.2 with Linux Kernel 6.8 and Ubuntu 24.04 LTS based root file system.
What’s new in JetPack 7.0
NO…
panic_GA_0828.tar.gz (17.3 MB)
We also reproduced it on Thor GA version, the version number is: 38.2.0-gcid-41844464 08/22/2025
[57360.788822] tegra194-vi5 8181200000.host1x:vi1@8188c00000: capture control message timed out#1
hello deeptalkcamera,
may I double check your reproduce steps?
This test is to open and close the camera in a loop with an interval of 15 seconds.
Error recovery should be triggered by itself after VI timeout？
hello deeptalkcamera,
we cannot reproduce the issue,
if there’s camera issue, error recovery will trigger to reset/restart the VI channel.0001-vi5-fix-vi5_channel_error_recover-memory-leak.patch (1.8 KB)
We simultaneously enabled four GMSL cameras, turning them on and off at 15-second intervals.
Does the patch “0001-vi5-fix-vi5_channel_error_recover-memory-leak.patch“ help with fix the kernel panic issue?
hello deeptalkcamera,
as you can see.. it fixes the error recovery memory leak.
kernel.log (12.1 MB)
uart_dmesg.log (8.4 MB)
The problem reappeared after using 0001-vi5-fix-vi5_channel_error_recover-memory-leak.patch. Please help to check the log.
hello deeptalkcamera,
according to the error logs.. refcount_t: addition on 0; use-after-free
please give it another try to apply this patch, 0001-rtcpu-capture-ivc-fix-multi-cam-race-condition.patch (2.2 KB)Topic 337341 for reference.
hi,JerryChang
seems there are two issues may cause refcount_t problem.
dma memory leak, cause the tegra_channel_kthread_capture_dequeue() thread exit abnormally.
“vi_capture_release: control failed, errno 1”. errno 1 means CAPTURE_ERROR_INVALID_PARAMETER, it also cause tegra_channel_kthread_capture_dequeue() thread exit abnormally.
maybe we can add some protect to prevent kernelpanic happen.
add “chan->kthread_capture_dequeue = NULL;”,and then when vi5_channel_stop_streaming() call kthread_stop, it will not stop the tegra_channel_kthread_capture_dequeue() thread that already not exist, maybe it can prevent refcount_t issue happen?
HI JerryChang ：
I used the following patches: 0001-vi5-fix-vi5_channel_error_recover-memory-leak.patch +
The panic issue reappeared, but the error stack trace was different:
Please help analyze the log again. Thank you.
0903_dma_null_race.zip (4.0 MB)
hello deeptalkcamera,
we may debug into vi5_power_off() to find the root cause,
static void vi5_power_off(struct tegra_channel *chan)
{
...
vi5_unit_get_device_handle(vi->ndev, chan->port[0], &dev);
hi,JerryChang
2025-08-29T03:55:29.364779+08:00 tegra-ubuntu kernel: tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms#1 #1 ] PREEMPT SMPclient_loop: send disconnect: Broken pipe
error_recovery.zip (1.8 KB)
hi all,
let me double check how you reproduce this issue? or, please share the pipeline for reference.
hi, Jerry
test with 4 camera.(2 camera in one channel)
plug-out one camera, and then plug-in, to let vi driver timeout, err recover fuction will reset the capture channel all the time.
open stream, after 15s, stop stream && close fd.
after about 10 hours, the err recover funtion will meet fatal error like “errno 1“ and “vi capture setup failed“, the dequeue thread will exit. and then it will meet kernel panic issue.
tegra_channel_kthread_capture_dequeue() thread meet “fatal: error recovery failed“ is the condition to reproduce kernel panic issue.
hello liutee,
this is the difference.. we don’t have this kind of test-case.
liuting11:
plug-out one camera, and then plug-in, to let vi driver timeout, err recover fuction will reset the capture channel all the time.
may I know what’s actual use-case?before stream-on.
hi, Jerry
by the way, if timed out happen, error recover will be called all the time, if reset channel successfully, it will not crash. but if reset channel failed(err no1, or vi capture setup failed), it will crash.