Kernel crash on continuous stream ON and stream OFF using gstreamer

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

Hi everyone,

I am facing an issue while performing stream ON and stream OFF continuously using gstreamer.

Setup used :

  1. L4T version - 28.2.1
  2. Jetson TX2 development kit + ar1335 sensor

For continuous stream ON and stream OFF I am using the following script.,

#! /bin/bash
while [ 1 ]
do
gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=100 ! "video/x-raw, width=1920, height=1080, format=(string)UYVY" ! fakseink
done

On running the above script, kernel crash occurs and dmesg has the following information.,

[ 9137.533519] Unable to handle kernel paging request at virtual address ffffffc400c76340
[ 9137.541563] pgd = ffffffc19d6f7000
[ 9137.545055] [ffffffc400c76340] *pgd=0000000000000000, *pud=0000000000000000
[ 9137.552239] Internal error: Oops: 96000005 [#1] PREEMPT SMP
[ 9137.557809] Modules linked in: nvhost_vi4 ar1335(O) fuse ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack nf_nat br_netfilter overlay bcmdhd pci_tegra bluedroid_pm
[ 9137.578080] CPU: 3 PID: 21318 Comm: v4l2src0:src Tainted: G           O    4.4.38-g #1
[ 9137.585984] Hardware name: quill (DT)
[ 9137.589643] task: ffffffc1e18e7080 ti: ffffffc1a38a4000 task.ti: ffffffc1a38a4000
[ 9137.597129] PC is at camera_common_try_fmt+0x94/0x2d0
[ 9137.602176] LR is at camera_common_try_fmt+0x68/0x2d0
[ 9137.607222] pc : [<ffffffc0007da41c>] lr : [<ffffffc0007da3f0>] pstate: a0000145
[ 9137.614606] sp : ffffffc1a38a7ae0
[ 9137.617915] x29: ffffffc1a38a7ae0 x28: ffffffc1ae3f8e00 
[ 9137.623245] x27: ffffffc1e174be50 x26: ffffffc00147e000 
[ 9137.628572] x25: ffffffc1e174be50 x24: ffffffc1e10c8018 
[ 9137.633898] x23: ffffffc000c76000 x22: 0000000000000000 
[ 9137.639222] x21: ffffffc1e174be18 x20: ffffffc1a38a7ba0 
[ 9137.644545] x19: ffffffbffcf135a0 x18: 0000000017220107 
[ 9137.649868] x17: 0000007f7c333b20 x16: ffffffc0001ee0a0 
[ 9137.655192] x15: 0000000000000028 x14: 0000000000000000 
[ 9137.660517] x13: 0000000000000000 x12: 0000000000000000 
[ 9137.665840] x11: 0000000000000000 x10: 0000000000000000 
[ 9137.671165] x9 : 0000000000000000 x8 : 0000000000000000 
[ 9137.676491] x7 : 0000000000000000 x6 : 0000000000000000 
[ 9137.681814] x5 : 0000000000000000 x4 : ffffffc073fc3458 
[ 9137.687135] x3 : ffffffc000c760d0 x2 : 000000007fffffff 
[ 9137.692458] x1 : ffffffc000c76348 x0 : 0000000000000438 

[ 9137.699271] Process v4l2src0:src (pid: 21318, stack limit = 0xffffffc1a38a4020)
[ 9137.706568] Call trace:
[ 9137.709016] [<ffffffc0007da41c>] camera_common_try_fmt+0x94/0x2d0
[ 9137.715116] [<ffffffbffcf103f0>] ar1335_set_fmt+0x28/0x30 [ar1335]
[ 9137.721290] [<ffffffc0007cf9cc>] __tegra_channel_try_format+0x8c/0x180
[ 9137.727806] [<ffffffc0007cfadc>] tegra_channel_try_format+0x1c/0x28
[ 9137.734067] [<ffffffc0007b6f54>] v4l_try_fmt+0x1a4/0x4a8
[ 9137.739373] [<ffffffc0007b63d4>] __video_do_ioctl+0x264/0x2d0
[ 9137.745111] [<ffffffc0007b5e54>] video_usercopy+0x29c/0x598
[ 9137.750675] [<ffffffc0007b6164>] video_ioctl2+0x14/0x20
[ 9137.755894] [<ffffffc0007b1210>] v4l2_ioctl+0xe8/0x118
[ 9137.761032] [<ffffffc0001eddec>] do_vfs_ioctl+0x33c/0x5f0
[ 9137.766422] [<ffffffc0001ee12c>] SyS_ioctl+0x8c/0xa0
[ 9137.771383] [<ffffffc000084ff0>] el0_svc_naked+0x24/0x28
[ 9137.777328] ---[ end trace 5e3bfdca625958e5 ]---

On further debugging, the kernel crash occur at camera_common_try_fmt function. Also the issue occurs in gstreamer and not in v4l2-ctl. I have listed the sequence of calls to start stream in both application.

Sequence of calls in v4l2-ctl :
1. Camera_common_try_fmt
2. Camera_common_set_fmt
3. Camera_common_try_fmt
4. Stream_ON

Sequence of calls in gstreamer :
1.       Camera_commn_try_fmt
2.       Camera_commn_try_fmt
3.       Camera_commn_try_fmt
          .
          .
          .
          .
          .
          .
N.       Camera_commn_try_fmt
N+1.     Camera common_set_fmt
N+2.     Camera_common_try_fmt
N+3.     Stream_ON

From the above sequence while using gstreamer, the camera_common_try_fmt function is called N number of times. At some point during these calls kernel crash occurs.

Questions :

  1. Why camera_common_try_fmt is called so many times while opening gstreamer ?
  2. What is the reason for kernel crash ? Is there any fix for this ?
2

hello arunmuthuganesh.r,

we had fix some known failures with l4t-r28.2/TX2 kernel drivers,
please check below forum topics and also apply the patch for verification.

  1. Fix FE syncpt wait (observe first 3 green/corruption frames)
    [url]Data Error of CSI Camera Capture on Jetson TX2 - Jetson TX2 - NVIDIA Developer Forums
  2. fix missing timestamp for vi4
    [url]https://devtalk.nvidia.com/default/topic/1020202/jetson-tx1/v4l2-time-stamp-does-not-work-for-l4t-28-1-jetpack-3-1-/post/5274687/#5274687[/url]

BTW, here’s discussion thread to add error handling mechanism for T186 (recovery mechanism), you may also check Topic 1037809 for patches.
thanks