Kernel panic when memory mapping, reading, and writing peripheral device registers

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
1

Hello,

I have been writing a very simple test I2S kernel module for kernel-5.10 where I memory map an I2S base register for the Jetson AGX Xavier I2S module into the kernel address space. I obtained the base register addresses (see i2s_driver_test/i2s.h · master · Peterson Yuhala / jetson-xavier · GitLab) by reading the corresponding device tree file for my device. The code for the simple kernel module in question can be found here:
i2s_driver_test/i2s_driver_test.c · master · Peterson Yuhala / jetson-xavier · GitLab

Once I load the kernel module (using sudo insmod module_name.ko), my Jetson AGX Xavier kit freezes and reboots. Below is part of the error log obtained after the crash happens:

Ubuntu 20.04.5 LTS jetson-desktop ttyTCU0

jetson-desktop login: [ 1509.076800] CPU:0, Error: cbb-noc@2300000, irq=15
[ 1509.076942] **************************************
[ 1509.077056] CPU:0, Error:cbb-noc
[ 1509.077132] 	Error Logger		: 0
[ 1509.077211] 	ErrLog0			: 0x80030008
[ 1509.077294] 	  Transaction Type	: WR  - Write, Incrementing
[ 1509.077424] 	  Error Code		: SLV
[ 1509.077494] 	  Error Source		: Target
[ 1509.077570] 	  Error Description	: Target error detected by CBB slave
[ 1509.077704] 	  AXI2APB_1 bridge error: SFIFONE - Status FIFO Not Empty interrupt
[ 1509.077849] 	  AXI2APB_1 bridge error: SLV - SLVERR interrupt
[ 1509.077981] 	  Packet header Lock	: 0
[ 1509.078052] 	  Packet header Len1	: 3
[ 1509.078126] 	  NOC protocol version	: version >= 2.7
[ 1509.078223] 	ErrLog1			: 0x320013
[ 1509.078287] 	ErrLog2			: 0x0
[ 1509.078343] 	  RouteId		: 0x320013
[ 1509.078409] 	  InitFlow		: ccroc_p2ps/I/ccroc_p2ps
[ 1509.078505] 	  Targflow		: axis_satellite_grout/T/axis_satellite_grout
[ 1509.078978] 	  TargSubRange		: 0
[ 1509.079212] 	  SeqId			: 0
[ 1509.079412] 	ErrLog3			: 0x2901120
[ 1509.079965] 	ErrLog4			: 0x0
[ 1509.082600] 	  Address accessed	: 0x2901120
[ 1509.087062] 	ErrLog5			: 0x489f851
[ 1509.090120] 	  Non-Modify		: 0x1
[ 1509.093788] 	  AXI ID		: 0x9
[ 1509.096676] 	  Master ID		: CCPLEX
[ 1509.099656] 	  Security Group(GRPSEC): 0x7e
[ 1509.103857] 	  Cache			: 0x1 -- Bufferable 
[ 1509.108315] 	  Protection		: 0x2 -- Unprivileged, Non-Secure, Data Access
[ 1509.115139] 	  FALCONSEC		: 0x0
[ 1509.118032] 	  Virtual Queuing Channel(VQC): 0x0
[ 1509.122929] 	**************************************
[ 1509.127971] kernel BUG at drivers/soc/tegra/cbb/tegra194-cbb.c:2057!
[ 1509.134304] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[ 1509.139815] Modules linked in: i2s_driver_test(OE) fuse(E) lzo_rle(E) lzo_compress(E) zram(E) snd_soc_tegra186_asrc(E) snd_soc_tegra210_ope(E) snd_soc_tegra186_dspk(E) snd_soc_tegra186_arad(E) snd_soc_tegra210_admaif(E) snd_soc_tegra210_dmic(E) snd_soc_tegra210_mvc(E) snd_soc_tegra210_iqc(E) snd_soc_tegra_pcm(E) snd_soc_tegra210_adx(E) snd_soc_tegra210_afc(E) snd_soc_tegra210_mixer(E) snd_soc_tegra210_i2s(E) snd_soc_tegra210_amx(E) snd_soc_tegra210_sfc(E) aes_ce_blk(E) crypto_simd(E) cryptd(E) aes_ce_cipher(E) ghash_ce(E) sha2_ce(E) sha256_arm64(E) sha1_ce(E) snd_soc_spdif_tx(E) snd_soc_tegra_machine_driver(E) input_leds(E) loop(E) snd_hda_codec_hdmi(E) snd_soc_tegra210_adsp(E) snd_hda_tegra(E) ucsi_ccg(E) snd_soc_rt5659(E) snd_hda_codec(E) snd_soc_tegra_utils(E) typec_ucsi(E) snd_soc_simple_card_utils(E) typec(E) max77620_thermal(E) nct1008(E) nvadsp(E) snd_soc_tegra210_ahub(E) snd_soc_rl6231(E) tegra210_adma(E) tegra_bpmp_thermal(E) userspace_alert(E) bluedroid_pm(E) snd_hda_core(E)
[ 1509.140048]  ahci(E) libahci(E) libata(E) spi_tegra114(E) binfmt_misc(E) ina3221(E) pwm_fan(E) nvgpu(E) nvmap(E) ip_tables(E) x_tables(E) [last unloaded: i2s_driver_test]
[ 1509.242369] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           OE     5.10.104-tegra #3
[ 1509.250243] Hardware name: Unknown Jetson-AGX/Jetson-AGX, BIOS 2.1-32413640 01/24/2023
[ 1509.258385] pstate: 60400089 (nZCv daIf +PAN -UAO -TCO BTYPE=--)
[ 1509.264426] pc : tegra194_cbb_err_isr+0x19c/0x1b0
[ 1509.269146] lr : tegra194_cbb_err_isr+0x11c/0x1b0
[ 1509.273883] sp : ffff800010003df0
[ 1509.277540] x29: ffff800010003df0 x28: 0000000000000001 
[ 1509.282793] x27: 0000000000000080 x26: ffffde9fd3533f70 
[ 1509.288305] x25: ffffde9fd426ae10 x24: 0000000000000001 
[ 1509.293988] x23: ffffde9fd3817000 x22: ffffde9fd3faea00 
[ 1509.299500] x21: 000000000000000f x20: 0000000000000005 
[ 1509.305014] x19: ffffde9fd3fae9f0 x18: 0000000000000010 
[ 1509.310350] x17: 0000000000000000 x16: 0000000000000000 
[ 1509.315606] x15: ffffde9fd3b12bf0 x14: ffffffffffffffff 
[ 1509.321289] x13: ffff8000900039e7 x12: ffff8000100039ef 
[ 1509.326367] x11: 0000000000000038 x10: 0101010101010101 
[ 1509.332138] x9 : ffff800010003d00 x8 : 2a2a2a2a2a2a2a2a 
[ 1509.337650] x7 : 2a2a2a2a2a2a2a09 x6 : c0000000ffff7fff 
[ 1509.342907] x5 : ffff612f3fd1c958 x4 : ffffde9fd3b27968 
[ 1509.348587] x3 : 0000000000000001 x2 : ffffde9fd1fce170 
[ 1509.353668] x1 : ffffde9fd3b12680 x0 : 0000000100010001 
[ 1509.359006] Call trace:
[ 1509.361732]  tegra194_cbb_err_isr+0x19c/0x1b0
[ 1509.365749]  __handle_irq_event_percpu+0x68/0x2a0
[ 1509.370552]  handle_irq_event_percpu+0x40/0xa0
[ 1509.375014]  handle_irq_event+0x50/0xf0
[ 1509.378520]  handle_fasteoi_irq+0xc0/0x170
[ 1509.382815]  generic_handle_irq+0x40/0x60
[ 1509.386828]  __handle_domain_irq+0x70/0xd0
[ 1509.391048]  efi_header_end+0xb0/0xf0
[ 1509.394527]  el1_irq+0xd0/0x180
[ 1509.397505]  cpuidle_enter_state+0xb8/0x410
[ 1509.401444]  cpuidle_enter+0x40/0x60
[ 1509.405221]  call_cpuidle+0x44/0x80
[ 1509.408445]  do_idle+0x208/0x270
[ 1509.412114]  cpu_startup_entry+0x30/0x70
[ 1509.415882]  rest_init+0xdc/0xe8
[ 1509.418872]  arch_call_rest_init+0x18/0x20
[ 1509.423057]  start_kernel+0x514/0x54c
[ 1509.426388] Code: a9446bf9 a94573fb a8c67bfd d65f03c0 (d4210000) 
[ 1509.432616] ---[ end trace 98fda04170279c03 ]---
[ 1509.437576] Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt
[ 1509.445108] SMP: stopping secondary CPUs
[ 1509.448981] Kernel Offset: 0x5e9fc1e10000 from 0xffff800010000000
[ 1509.455077] PHYS_OFFSET: 0xffff9ed940000000
[ 1509.458937] CPU features: 0x8240002,03802a30
[ 1509.463410] Memory Limit: none
[ 1509.466552] ---[ end Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt ]---

I would be happy if any one could help pinpoint what causes this.

4

Hi petersonyuhala,
There is already an I2S ALSA driver as part of jetpack release. You could use that as a base for your work. I2S Controller Register access would require APE power domain to be ungated and certain clocks to be on which your custom test driver is not ensuring hence the crash during driver insmod.

5

Thanks for your response. I will update my code to take that into account.

Indeed, I am aware there is already an I2S driver included in the JetPack release. I am trying to do a very simple implementation for an OPTEE environment, and the JetPack version is a bit large and complex, and can’t function within OPTEE without major changes. That’s why I decided to build one from the ground up.

In the meantime, I would be happy to get access to an I2S solution that works in OPTEE.

6

Hi petersonyuhala,
Please note our support will be limited to I2S ALSA driver here. We would happy to give pointers for your development outside of this.

If you can able to handle the APE power domain and Audio clocks enable/disable from your test app that would help you avoid the crash.

Also think of, if you can port ALSA to OPTEE which can leverage the existing driver for your usage.

7

Thanks very much for your support. We will consider porting the driver from Jetpack.

8

Hello @mkumard , following your advice, I have been trying to implement the clock enable/disable. When I try retrieving the i2s and sync_input clocks (same clock names in the DT), it always fails. The code snippet I use is as follows:

struct clk* bit_clock = clk_get(NULL, "i2s");

struct clk* sync_clock = clk_get(NULL, "sync_input");

I confirm both instructions fail to retrieve the clocks with if (IS_ERR(bit_clock)) etc.

The Jetpack driver uses the devm_clk_get API with an associated platform_device as parameter. Could this be the problem with my code ?

Otherwise, I would greatly appreciate if you could provide some pointers on how I could go about ungating the APE power domain correctly and setting the right clocks.

9

Hi petersonyuhala,
Am not expert on clock apis, but I believe when we don’t provide device as an argument to clk_get api, it won’t take the name from DT. so “i2s” may not work. Can you just give a try with clk_get(NULL, “i2s1”); which shall use bpmp fw clock name.

10

Ok, thanks. I’d try that out.

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.