[r36.4.4] vi5_fops.c: wrong address freed in tegra_channel_capture_setup() error path → Bad page state → kernel panic

Hi NVIDIA team,

On AGX Orin with r36.4.4 (kernel 5.15.148-tegra), we found a bug in the VI5 error path that corrupts kernel memory and leads to a system reset.

Root cause

In vi5_fops.c, tegra_channel_capture_setup() allocates the descriptor ring into chan->request[vi_port], but when vi_capture_setup() fails it frees chan->request instead. That is the address of the pointer array inside struct tegra_channel, not the DMA buffer.

c

err = vi_capture_setup(chan->tegra_vi_channel[vi_port], &setup);
if (err) {
        dev_err(chan->vi->dev, "vi capture setup failed\n");
        dma_free_coherent(chan->tegra_vi_channel[vi_port]->rtcpu_dev,
                        setup.queue_depth * setup.request_size,
                        chan->request, setup.iova);   /* should be chan->request[vi_port] */
        return err;
}

As a result, a live kernel page is returned to the allocator, the real DMA buffer is leaked, and the pointer is left dangling.

Evidence

The path is easily hit when cameras are disconnected and several channels retry VIDIOC_STREAMON at the same time. Then channel context at N is busy makes vi_capture_setup() fail:

[2247.727] channel context at 0 is busy  (WARN, capture-ivc.c:195, Comm: ai_framework)
[2248.733] tegra194-vi5 ...vi0@15c00000: failed to update control callback
[2248.733] tegra-camrtc-capture-vi tegra-capture-vi: vi capture setup failed   <- this error path
[2249.396] BUG: Bad page state in process ai_framework ... page dumped because: non-NULL mapping
-> IVC control submit failed / MBOX_TX_QUEUE_LEN -> Oops -> panic -> reset

Across the resets we checked, the number of “vi capture setup failed” messages matches the “Bad page state” reports (e.g. 6 → 5 in one boot), and every reset was preceded by this sequence. The same busy → “failed to update control callback” → Bad page state pattern also appears in Topic 317977.

Proposed fix

c

dma_free_coherent(chan->tegra_vi_channel[vi_port]->rtcpu_dev,
                setup.queue_depth * setup.request_size,
                chan->request[vi_port], setup.iova);
chan->request[vi_port] = NULL;

Related leaks in the same area: the csi_chan == NULL (-EINVAL) branch in the same function, and the err_set_stream error path of vi5_channel_start_streaming(), both never free chan->request[vi_port].

Already applied: the capture-ivc multi-cam race fix (337341), the error_recover leak fix (337609), and the infinite capture timeout.

Questions

  1. Can you confirm this bug and provide an official patch for r36.x?
  2. Is it fixed in a later release?

Thanks.

hello LeeJ,

it should not fully resolved in the latest JP-6 public release, the issue remains in JP-6.2.3/r36.5.2
per checking the code, clearing the pointer immediately after the setup-error free is still needed.
for instance,

--- a/drivers/media/platform/tegra/camera/vi/vi5_fops.c
+++ b/drivers/media/platform/tegra/camera/vi/vi5_fops.c
@@ -375,9 +375,10 @@ static int tegra_channel_capture_setup(struct tegra_channel *chan, unsigned int vi_port)
        err = vi_capture_setup(chan->tegra_vi_channel[vi_port], &setup);
        if (err) {
                dev_err(chan->vi->dev, "vi capture setup failed\n");
                dma_free_coherent(chan->tegra_vi_channel[vi_port]->rtcpu_dev,
                                        setup.queue_depth * setup.request_size,
-                                     chan->request, setup.iova);
+                                     chan->request[vi_port], setup.iova);
+             chan->request[vi_port] = NULL;
                return err;

Hi JerryChang,

Thank you for confirming the issue and the fix.

We also checked the latest release, r39.2, and the fix is not applied there either.

1. Official fix schedule
Is there a plan or a target release for the official fix in r36.x and r39.x?

2. Error paths in tegra_channel_capture_setup()
chan->request[vi_port] is allocated at the top of the function, before both error returns below.

  • (a) csi_chan == NULL branch: this branch returns -EINVAL after the allocation, without freeing the buffer or clearing the pointer.
    Could you confirm whether this branch also needs both dma_free_coherent() and chan->request[vi_port] = NULL, the same as in (b)?
  • (b) vi_capture_setup() failure branch: the fix you provided.

For reference, on r36.4.4 this is what we have in mind:

 static int tegra_channel_capture_setup(struct tegra_channel *chan, unsigned int vi_port)
 {
 	...
 	chan->request[vi_port] = dma_alloc_coherent(chan->tegra_vi_channel[vi_port]->rtcpu_dev,
 					setup.queue_depth * setup.request_size,
 					&setup.iova, GFP_KERNEL);
 	chan->request_iova[vi_port] = setup.iova;
 	...
 	/* Set CSI port info */
 	if (chan->pg_mode) {
 		setup.csi_port = NVCSI_PORT_UNSPECIFIED;
 	} else {
 		struct tegra_csi_channel *csi_chan = find_linked_csi_channel(chan);
 
 		if (csi_chan == NULL)
 		{
 			dev_err(chan->vi->dev, "csi_chan not found");
+			dma_free_coherent(chan->tegra_vi_channel[vi_port]->rtcpu_dev,
+					setup.queue_depth * setup.request_size,
+					chan->request[vi_port], setup.iova);
+			chan->request[vi_port] = NULL;
 			return -EINVAL;
 		}
 
 		setup.csi_port = csi_chan->ports[vi_port].csi_port;
 	}
 
 	if (chan->fmtinfo->fourcc == V4L2_PIX_FMT_NV16)
 		setup.channel_flags |= CAPTURE_CHANNEL_FLAG_SEMI_PLANAR;
 
 	err = vi_capture_setup(chan->tegra_vi_channel[vi_port], &setup);
 	if (err) {
 		dev_err(chan->vi->dev, "vi capture setup failed\n");
 		dma_free_coherent(chan->tegra_vi_channel[vi_port]->rtcpu_dev,
 				setup.queue_depth * setup.request_size,
-				chan->request, setup.iova);
+				chan->request[vi_port], setup.iova);
+		chan->request[vi_port] = NULL;
 		return err;
 	}
 
 	return 0;
 }

We will apply the fix and verify it under the same conditions (cameras disconnected, concurrent STREAMON on multiple channels), and share the result.

Thanks.

Hi JerryChang,

We applied the fix and tested it under the same conditions as before. Unfortunately, the kernel panic still occurs, and in our test it happened even sooner than without the fix.

Test setup

  • AGX Orin, r36.4.4, kernel 5.15.148-tegra
  • Patched tegra_channel_capture_setup() as discussed (both csi_chan == NULL and vi_capture_setup() failure paths: free chan->request[vi_port] and set it to NULL)
  • Confirmed the patched module is loaded (version string printed at boot from the patched vi5_fops.c)
  • Already applied: capture-ivc multi-cam race fix (337341), error_recover leak fix (337609), infinite capture timeout
  • Cameras disconnected, application restarting STREAMON on multiple channels concurrently

Logs after the fix

  1. channel context busy still occurs in the STREAMON path:
39644 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849410] pc : tegra_capture_ivc_notify_chan_id+0x1a0/0x1c0 [capture_ivc]
39645 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849414] lr : tegra_capture_ivc_notify_chan_id+0x1a0/0x1c0 [capture_ivc]
39646 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849417] sp : ffff800023b7b820
39647 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849419] x29: ffff800023b7b820 x28: ffff000da3daec58 x27: ffff000085f2a420
39648 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849424] x26: 0000000000000158 x25: 0000000000001043 x24: ffff000088aac088
39649 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849429] x23: 00000000000000a5 x22: ffff000088aad520 x21: ffff000088aac080
39650 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849434] x20: 0000000000000001 x19: 00000000000000a5 x18: ffffffffffffffff
39651 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849439] x17: 0000000000000000 x16: ffffbc7ca6dbdbe0 x15: ffffbc7ca8361215
39652 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849445] x14: ffffffffffffffff x13: ffffbc7ca836120b x12: 6e6f63206c656e6e
39653 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849451] x11: ffffffffffffffff x10: ffffbc7ca7f8a078 x9 : ffff800023b7b820
39654 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849456] x8 : ffff800023b7b820 x7 : 7973756220736920 x6 : 7973756220736920
39655 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849460] x5 : ffff000da3bd79f0 x4 : 0000000100005292 x3 : ffffbc7ca7fd7bf8
39656 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849468] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff00022d5d3e00
39657 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849473] Call trace:
39658 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849475]  tegra_capture_ivc_notify_chan_id+0x1a0/0x1c0 [capture_ivc]
39659 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849478]  vi_capture_setup+0x334/0x680 [tegra_camera]
39660 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849502]  tegra_channel_capture_setup+0x10c/0x2bc [tegra_camera]
39661 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849516]  vi5_channel_start_streaming+0x6c/0x410 [tegra_camera]
39662 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849528]  tegra_channel_start_streaming+0x58/0x84 [tegra_camera]
39663 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849541]  vb2_start_streaming+0x74/0x160 [videobuf2_common]
39664 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849548]  vb2_core_streamon+0xa4/0x1b0 [videobuf2_common]
39665 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849552]  vb2_ioctl_streamon+0x6c/0xc0 [videobuf2_v4l2]
39666 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849558]  v4l_streamon+0x44/0x60 [videodev]
39667 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849573]  __video_do_ioctl+0x190/0x400 [videodev]
39668 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849585]  video_usercopy+0x278/0x660 [videodev]
39669 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849596]  video_ioctl2+0x44/0x80 [videodev]
39670 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849606]  v4l2_ioctl+0x6c/0xa0 [videodev]
39671 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849617]  __arm64_sys_ioctl+0xb4/0x100
39672 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849625]  invoke_syscall+0x5c/0x130
39673 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849632]  el0_svc_common.constprop.0+0x64/0x110
39674 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849636]  do_el0_svc+0x74/0xa0
39675 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849639]  el0_svc+0x28/0x80
39676 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849645]  el0t_64_sync_handler+0xa4/0x130
39677 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849649]  el0t_64_sync+0x1a4/0x1a8
39678 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.849652] ---[ end trace 857891deb939ed08 ]---
39679 Nov  5 16:10:36 nvidia-desktop kernel: [ 2400.857257] thcv242 cam1_14-0065: thcv242_s_stream - enable : 0
39680 Nov  5 16:10:36 nvidia-desktop kernel: [ 2401.291819] thcv242 cam4_9-0065: thcv242_s_stream - enable : 0
39681 Nov  5 16:10:36 nvidia-desktop kernel: [ 2401.295161] thcv242 cam5_11-0065: thcv242_s_stream - enable : 0
  1. In another run, the camera RTCPU stops responding and its recovery is refused because the device is runtime-suspended:
[ 1238.900069] tegra186-cam-rtcpu bc00000.rtcpu: Alert: Camera RTCPU gone bad! restoring it immediately!!
[ 1238.900082] tegra186-cam-rtcpu bc00000.rtcpu: Alert: Camera RTCPU gone bad! restoring it immediately!!
[ 1238.900097] tegra186-cam-rtcpu bc00000.rtcpu: Alert: Camera RTCPU gone bad! restoring it immediately!!
[ 1238.900108] tegra186-cam-rtcpu bc00000.rtcpu: Alert: Camera RTCPU gone bad! restoring it immediately!!
[ 1238.900127] tegra186-cam-rtcpu bc00000.rtcpu: Alert: Camera RTCPU gone bad! restoring it immediately!!
[ 1238.900137] tegra186-cam-rtcpu bc00000.rtcpu: Alert: Camera RTCPU gone bad! restoring it immediately!!
[ 1238.900148] tegra186-cam-rtcpu bc00000.rtcpu: cannot reboot while suspended
[ 1238.900157] tegra186-cam-rtcpu bc00000.rtcpu: Alert: Camera RTCPU gone bad! restoring it immediately!!
[ 1238.900173] tegra186-cam-rtcpu bc00000.rtcpu: Alert: Camera RTCPU gone bad! restoring it immediately!!

This looks the same as the runtime PM issue reported in Topic 369303.

Questions

  1. With the wrong-address free fixed, the panic still happens (and sooner). Is there another free or cleanup path that touches the same channel or buffer after vi_capture_setup() fails, or a known additional fix for this scenario?
  2. Is there a fix for the “cannot reboot while suspended” state, so the camera RTCPU can be recovered without rebooting the whole system?
  3. Is there a fix that prevents “channel context at N is busy” when multiple channels STREAMON concurrently, with the race-condition fix already applied?

Thanks.

hello LeeJ,

may I have more details of your reproduce steps.
for instance, how many cameras in your system? did you remove the camera device physically for testing?

Hi JerryChang,

Here are the detailed reproduction steps.

System

  • Jetson AGX Orin on a custom carrier board, r36.4.4 (kernel 5.15.148-tegra)
  • 6 CSI cameras
  • Capture through V4L2 (/dev/videoN), one thread per camera in a single process
  • Capture timeout set to infinite (override_capture_timeout_ms = -1)

Steps

  1. Boot the system with all 6 cameras connected; all 6 video nodes are registered normally.
  2. Physically disconnect all 6 cameras (remove the camera modules from the SerDes links).
  3. Run the application. Each of the 6 camera threads restarts its stream every 200 ms, and all 6 threads do this at the same time:
    VIDIOC_STREAMOFF → VIDIOC_STREAMON (repeated)
    Since the cameras are disconnected, the sensor s_stream fails, so STREAMON fails and the thread retries after 200 ms.
  4. Within some tens of minutes, channel context at N is busy appears in the vi_capture_setup() path, followed by vi capture setup failed.
    • Without the fix: BUG: Bad page state follows within about 1 s, then Oops, panic and reset (observed after 26 to 69 min in our runs).
    • With the fix: the panic still occurs, and sooner. In another run we also saw Camera RTCPU gone bad! with cannot reboot while suspended.

Key conditions

  • Multiple channels (6) restarting concurrently, with a short interval (200 ms)
  • Cameras disconnected, so every STREAMON attempt fails and is retried
  • With a single camera, or with the restarts serialized, we have not reproduced it so far.

Please let us know if you need any other information or logs.

Thanks.

hello LeeJ,

honestly, we only test error recovery mechanism with the software-simulated stream loss,
we’ve never test with singe camera or even six cameras being physically disconnected scenario.

may I have more details for repeated concurrent STREAMOFF/STREAMON retries while all camera modules remain physically disconnected?
what’s the actual use-case?
as a temporary workaround, you should check sensor/link availability before retrying and serialize restart attempts.

anyways..
here’re couple of kernel patches
(1) media: camera: fix a negative refcount

diff --git a/drivers/media/platform/tegra/camera/tegracam_v4l2.c b/drivers/media/platform/tegra/camera/tegracam_v4l2.c
index a5748c2..7169a65 100644
--- a/drivers/media/platform/tegra/camera/tegracam_v4l2.c
+++ b/drivers/media/platform/tegra/camera/tegracam_v4l2.c
@@ -83,7 +83,7 @@
 		err = sensor_ops->stop_streaming(tc_dev);
 		if (err) {
 			dev_err(&client->dev, "Error turning off streaming\n");
-			goto error;
+			return err;
 		}

(2) vi5: fix channel status inconsistency

diff --git a/drivers/media/platform/tegra/camera/vi/channel.c b/drivers/media/platform/tegra/camera/vi/channel.c
index e818f61..37c5383 100644
--- a/drivers/media/platform/tegra/camera/vi/channel.c
+++ b/drivers/media/platform/tegra/camera/vi/channel.c
@@ -1019,6 +1019,7 @@
 
 	if (vi->fops) {
 		vi->fops->vi_stop_streaming(vq);
+		atomic_set(&chan->is_streaming, DISABLE);
 		vi->fops->vi_power_off(chan);
 	}

besides..
please also have a try, here’re steps to disable RCE suspend (rtcpu auto-suspend).
# echo -1 > /sys/devices/platform/*rtcpu/power/autosuspend_delay_ms
# echo on > /sys/devices/platform/*rtcpu/power/control

Hi JerryChang,

Actual use-case
In the field, a camera link can be lost temporarily. When that happens, our application tries to recover the camera automatically by restarting the stream (STREAMOFF → STREAMON). Each camera thread runs this recovery independently, so several channels can restart at the same time.
Physically disconnecting all 6 cameras in our test is a way to reproduce this “link lost + automatic restart” condition quickly and repeatedly.

Test result
We applied the tegra_channel_capture_setup() free fix and the two patches you provided ((1) negative refcount, (2) channel status inconsistency), and repeated the same test.

  • Bad page state no longer appears.
  • The time to panic increased (about 110 min in this run).
  • However, the kernel panic still occurs, at a different location: a NULL pointer dereference in vi_capture_request_unpin().

The kernel log of the panic is attached. Key lines:

261110_kern_panic_sanitized.log (21.8 KB)

[ 6597.201134] channel context at 1 is busy          (tegra_capture_ivc_notify_chan_id <- vi_capture_setup, STREAMON path)
[ 6598.207070] failed to update control callback
[ 6598.207147] vi capture setup failed
[ 6599.227840] capture control message timed out     (several channels)
[ 6600.255819] csi_stream_release: failed to disable nvcsi tpg on stream 2 virtual channel 0
[ 6601.275816] vi_capture_release: release channel IVC failed
               WARNING at capture-vi.c:954 vi_capture_release  <- vi_capture_shutdown <- vi_channel_close_ex <- vi5_channel_start_streaming
[ 6602.281343] ivc channel driver missing / Try increasing MBOX_TX_QUEUE_LEN
[ 6602.325177] capture channel 1 is idle             (tegra_capture_ivc_unregister_capture_cb <- vi_capture_release)
[ 6603.330663] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
               pc : vi_capture_request_unpin+0x48/0xb0
               vi_capture_request_unpin <- vi_capture_shutdown <- vi_channel_close_ex <- vi5_channel_start_streaming+0x1a8
[ 6603.939404] Kernel panic - not syncing: Oops: Fatal exception

Analysis (r36.4.4 capture-vi.c / capture-ivc.c)

  1. In vi_capture_setup(), capture->channel_id is set from the RCE response before tegra_capture_ivc_notify_chan_id() is called. When notify fails (busy), the cb_fail path frees unpins_list (and sets it to NULL) and requests_memoryinfo, but leaves channel_id, stream_id and queue_depth set.
  2. vi5_channel_start_streaming() then goes to err_setup → vi_channel_close_ex() → vi_capture_shutdown(). Because channel_id is still valid, shutdown handles this channel as if setup had completed:
    • It sends reset, CSI stream release and channel release requests to the RCE. The release fails, and vi_capture_release() calls tegra_camrtc_reboot(), which affects all channels.
    • tegra_capture_ivc_unregister_capture_cb(1) warns “capture channel 1 is idle”, because the capture callback was never registered.
    • tegra_capture_ivc_unregister_control_cb(1) clears slot 1 without checking priv_context. No “control channel 1 is idle” warning is printed, so slot 1 still held a callback at that point. Since this channel’s own temporary control callback was already unregistered in cb_fail and never moved to slot 1, the callback that gets cleared belongs to another channel. That channel then stops receiving RCE responses, which matches the “capture control message timed out” messages on several channels.
    • It loops vi_capture_request_unpin(i) up to queue_depth while unpins_list is NULL, which causes the NULL pointer dereference.

Questions

  1. Is there an official fix for the cb_fail path of vi_capture_setup()? In this case the RCE has already allocated the channel, so it seems the RCE channel should be released without touching a kernel callback slot owned by another channel, and the capture state (channel_id, stream_id, queue_depth) should be reset.
  2. tegra_capture_ivc_unregister_control_cb() and unregister_capture_cb() clear a slot without checking priv_context, so a channel whose setup failed can remove another channel’s callback. Should these functions check the owner before clearing a slot?
  3. We are also planning development on r39.2. Is r39.2 affected by the same paths?

Thanks.

unfortunately,
STREAMOFF → STREAMON restarts the capture channel; it does not guarantee complete VI/RCE recovery.
please give it a try by restarting the complete V4L2 capture session instead of repeatedly issuing STREAMOFF → STREAMON on the existing file descriptors.