TX2 halt occurs during write operation to the PCIe switch interrupt register

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

HI,

IDT PCIe switch connected to jetson tx2.
The IDT switch is connected to the board that drives the NTB developed by us.
The transfer of data from the jetson through the NTB port is usually done.
However, there is a problem with the interrupt.
The system hangs when a write operation is performed on the register associated with the IDT switch interrupt.
Unlike a simple interrupt congestion problem, when an interrupt is generated, isr is executed, but the time at which isr is executed is significantly slower(4~5sec) than the interrupt. Also, various kernel error logs are printed before and after isr is executed.

Below is the dmesg log.

[ 134.938548] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 134.942874] 0-…: (1 ticks this GP) idle=caf/2/0 softirq=12410/12410 fqs=0
[ 134.950147] (detected by 1, t=5252 jiffies, g=767, c=766, q=14)
[ 134.956242] Task dump for CPU 0:
[ 134.956266] swapper/0 R running task 0 0 0 0x00000002
[ 134.956322] Call trace:
[ 134.956456] [] __switch_to+0x9c/0xc0
[ 134.956508] [] cpuidle_enter_state+0xa0/0x380
[ 134.956598] [] cpuidle_enter+0x34/0x48
[ 134.956629] [] call_cpuidle+0x44/0x70
[ 134.956657] [] cpu_startup_entry+0x1b0/0x200
[ 134.956691] [] rest_init+0x84/0x90
[ 134.956722] [] start_kernel+0x370/0x384
[ 134.956757] [] __primary_switched+0x80/0x94
[ 134.956875] rcu_sched kthread starved for 5252 jiffies! g767 c766 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
[ 134.966130] rcu_sched S 0 8 2 0x00000000
[ 134.966169] Call trace:
[ 134.966269] [] __switch_to+0x9c/0xc0
[ 134.966300] [] __schedule+0x270/0x780
[ 134.966325] [] schedule+0x40/0xa8
[ 134.966383] [] schedule_timeout+0x88/0x420
[ 134.966415] [] rcu_gp_kthread+0x4a4/0x7d8
[ 134.966442] [] kthread+0xec/0xf0
[ 134.966467] [] ret_from_fork+0x10/0x40
[ 134.966732] INFO: rcu_preempt detected stalls on CPUs/tasks:
[ 134.972489] 0-…: (2 GPs behind) idle=caf/2/0 softirq=12410/12410 fqs=2539
[ 134.979663] (detected by 1, t=5257 jiffies, g=3826, c=3825, q=97)
[ 134.985919] Task dump for CPU 0:
[ 134.985936] swapper/0 R running task 0 0 0 0x00000002
[ 134.985984] Call trace:
[ 134.986053] [] __switch_to+0x9c/0xc0
[ 134.986174] [] cpuidle_enter_state+0xa0/0x380
[ 134.986199] [] cpuidle_enter+0x34/0x48
[ 134.986223] [] call_cpuidle+0x44/0x70
[ 134.986247] [] cpu_startup_entry+0x1b0/0x200
[ 134.986273] [] rest_init+0x84/0x90
[ 134.986295] [] start_kernel+0x370/0x384
[ 134.986324] [] __primary_switched+0x80/0x94

3

Could you please help us understand the issue better by answering the following queries?
Where is this register to which a write is happening present? Is it part of the BAR of the upstream port of the switch?
Is the issue happening because of write to this register or is it happening because of the ISR execution getting delayed? Curious to know as to how did you find out that the ISR execution is getting delayed by 4~5 seconds?
Just for the sake of experiment, have to tried writing to this register in a normal scenario? i.e. post booting of the system through a tool like busybox?
Honestly we don’t see any reason at this point why TX2 should hang unless there there is a malfunctioning element on the other side of the PCIe link.
Could you also confirm if this setup works when it is connected to another host system (like x86)?