System Error using Intel 10G Ethernet X710DA2 adapter w/ TX1

Autonomous Machines Jetson & Embedded Systems Jetson TX1
1

Hello!

I am trying to use this card here Amazon.com on the Nivida TX1 dev board. The Intel board is by default configured to use x8 PCIe but they have sent me a flash image for x4 configuration. I have programmed the board with this image and tested it on a x86 Linux system and it works perfectly.

When I tried to use the board on the TX1 dev kit with L4T 24.2 I found that that the system hangs upon loading the Intel i40e driver. Narrowing down the problem in the driver I found that the problem occurs in the pci_enable_device_mem() call.

Then I looked further in the pci subsystem and narrowed it down there to a specific line in function pcibios_enable_device() in /arch/arm64/kernel/bios32.c.
The problem occurs in the following bit of code when pci_write_config_word() is called. The cmd value written is 0x142, whereas the old_cmd value was 0x140.

if (cmd != old_cmd) {
                              printk("PCI: enabling device %s (%04x -> %04x)\n",
                                     pci_name(dev), old_cmd, cmd);
                              pci_write_config_word(dev, PCI_COMMAND, cmd);
               }

After this happens the system outputs some trace info and locks up. Following is a snippet of beginning of trace.

[  319.049506] INFO: rcu_preempt self-detected stall on CPU[  319.049538] INFO:g
[  319.064742] Bad mode in Error handler detected, code 0xbf000002
[  319.070664] CPU: 3 PID: 1297 Comm: rtkit-daemon Not tainted 3.10.96+ #1
[  319.077266] task: ffffffc0d6ccc140 ti: ffffffc0d6cd4000 task.ti: ffffffc0d6c0
[  319.084738] PC is at 0x7fa25750e4
[  319.088044] LR is at 0x7fa25750c4
root@teg[  319.091349] pc : [<0000007fa25750e4>] lr : [<0000007fa25750c4>] psta0
ra-ubunt[  319.099415] sp : ffffffc0d6cd7ff0

I tried to debug the issue further by looking at the error handler and found that the code 0xbf000002 seems to be the value of the Exception syndrome register. Decoding the values it seems like the exception class is 0x2F which is a System Error Interrupt.
The value in the syndrome information field is 0x1000002 which is defined as “implementation specific” syndrome information.

The issue can be reproduced very simply without i40e driver just by using setpci as follows:

#setpci -d 8086:1572 04.W=0x0142

I have tried using other PCIe boards in the TX1 dev kit and they work fine. I also probed the PCIe reset on the Intel card while the hang was happening and it looks OK. Attached is the lspci output.

I would like to know why is using this board with the TX1 dev kit causing a system error and what can be done to resolve the issue ?

2

Full lspci output

00:01.0 PCI bridge [0604]: NVIDIA Corporation Device [10de:0fae] (rev a1) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 130
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: fff00000-000fffff
	Prefetchable memory behind bridge: 0000000020000000-00000000213fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity+ SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Subsystem: NVIDIA Corporation Device [10de:0000]
	Capabilities: [48] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable- Count=1/2 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
		Mapping Address Base: 00000000fee00000
	Capabilities: [80] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0
			ExtTag+ RBE+
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
			ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Off, PwrInd On, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet+ LinkState+
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [140 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=30us PortTPowerOnTime=70us
	Kernel driver in use: pcieport

01:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)
	Subsystem: Intel Corporation Ethernet Converged Network Adapter X710-2 [8086:0007]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 130
	Region 0: Memory at 20000000 (64-bit, prefetchable) [disabled] 
	Region 3: Memory at 21100000 (64-bit, prefetchable) [disabled] 
	[virtual] Expansion ROM at 21000000 [disabled] 
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [70] MSI-X: Enable- Count=129 Masked-
		Vector table: BAR=3 offset=00000000
		PBA: BAR=3 offset=00001000
	Capabilities: [a0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s <2us, L1 <16us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [e0] Vital Product Data
		Product Name: XL710 40GbE Controller
		Read-only fields:
			[PN] Part number: 
			[EC] Engineering changes: 
			[FG] Unknown: 
			[LC] Unknown: 
			[MN] Manufacture ID: 
			[PG] Unknown: 
			[SN] Serial number: 
			[V0] Vendor specific: 
			[RV] Reserved: checksum good, 0 byte(s) reserved
		Read/write fields:
			[V1] Vendor specific: 
		End
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr+ BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [140 v1] Device Serial Number 38-a5-3a-ff-ff-ca-05-68
	Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 1
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy-
		IOVSta:	Migration-
		Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 272, stride: 1, Device ID: 154c
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 0000000000000000 (64-bit, prefetchable)
		Region 3: Memory at 0000000000000000 (64-bit, prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Capabilities: [1a0 v1] Transaction Processing Hints
		Device specific mode supported
		No steering table available
	Capabilities: [1b0 v1] Access Control Services
		ACSCap:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [1d0 v1] #19

01:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ [8086:1572] (rev 01)
	Subsystem: Intel Corporation Ethernet Converged Network Adapter X710 [8086:0000]
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 130
	Region 0: Memory at 20800000 (64-bit, prefetchable) [disabled] 
	Region 3: Memory at 21108000 (64-bit, prefetchable) [disabled] 
	[virtual] Expansion ROM at 21080000 [disabled] 
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [70] MSI-X: Enable- Count=129 Masked-
		Vector table: BAR=3 offset=00000000
		PBA: BAR=3 offset=00001000
	Capabilities: [a0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s <2us, L1 <16us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [e0] Vital Product Data
		Product Name: XL710 40GbE Controller
		Read-only fields:
			[PN] Part number: 
			[EC] Engineering changes: 
			[FG] Unknown: 
			[LC] Unknown: 
			[MN] Manufacture ID: 
			[PG] Unknown: 
			[SN] Serial number: 
			[V0] Vendor specific: 
			[RV] Reserved: checksum good, 0 byte(s) reserved
		Read/write fields:
			[V1] Vendor specific: 
		End
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr+ BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [140 v1] Device Serial Number 38-a5-3a-ff-ff-ca-05-68
	Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy-
		IOVSta:	Migration-
		Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 01
		VF offset: 335, stride: 1, Device ID: 154c
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 0000000000000000 (64-bit, prefetchable)
		Region 3: Memory at 0000000000000000 (64-bit, prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Capabilities: [1a0 v1] Transaction Processing Hints
		Device specific mode supported
		No steering table available
	Capabilities: [1b0 v1] Access Control Services
		ACSCap:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
3

Are you sure that it is 0x142 that would hang the system?
There is one issue with some of the Intel cards (particularly with I210 controller based cards) where enabling bus-master-enable bit (which is bit-2) in COMMAND register would hang the system. But here, by writing 0x142, we are enabling bit-1 (counting from zero) which corresponds to enabling MEM space of end point … and there are no known issues with that. Wondering if it is something to do with the new firmware you have received? Also, can you please try setting 0x0144 through setpci and let us know if that also hangs the system??

4

Yes i am sure about 0x0142 causing an issue. I just tried 0x0142 and 0x0144 individually from a cold boot and they both cause a system error.

Following is the dmesg output after running each command:

#setpci -d 8086:1572 04.W=0x0144
[  305.089918] INFO: rcu_preempt self-detected stall on CPU {
[  305.089990] INFO: rcu_preempt detected stalls on CPUs/tasks:
[  305.090013]  {
[  305.090024]  0
[  305.090036]  1
[  305.090046]  3
[  305.090055] }
[  305.090076] (detected by 2, t=2106 jiffies, g=795, c=794, q=29)
[  305.090088] Task dump for CPU 0:
[  305.090103] swapper/0       R
[  305.090122]   running task
[  305.090146]     0     0      0 0x00000000
[  305.090158] Call trace:
[  305.090202] [<ffffffc000086ae0>] __switch_to+0x3c/0x48
[  305.090220] Task dump for CPU 1:
[  305.090241] swapper/1       R
[  305.090256]   running task
[  305.090277]     0     0      1 0x00000002
[  305.090287] Call trace:
[  305.090315] [<ffffffc000086ae0>] __switch_to+0x3c/0x48
[  305.090332] Task dump for CPU 3:
[  305.090350] setpci          R
[  305.090364]   running task
[  305.090386]     0  1629   1613 0x00000000
[  305.090397] Call trace:
[  305.090423] [<ffffffc000086ae0>] __switch_to+0x3c/0x48
[  305.090446] [<ffffffc0bffd3650>] 0xffffffc0bffd3650
[  305.090485]  3
[  305.090514] }  (t=2106 jiffies g=795 c=794 q=29)
[  305.090570] CPU: 3 PID: 1629 Comm: setpci Not tainted 3.10.96+ #1
[  305.090597] Call trace:
[  305.090642] [<ffffffc000089d70>] dump_backtrace+0x0/0xf0
[  305.090681] [<ffffffc00008a068>] show_stack+0x14/0x1c
[  305.090723] [<ffffffc0003258ec>] dump_stack+0x20/0x28
[  305.090766] [<ffffffc00012947c>] print_cpu_stall+0xd8/0x140
[  305.090804] [<ffffffc000129964>] check_cpu_stall+0x64/0x9c
[  305.090843] [<ffffffc0001299c8>] __rcu_pending+0x2c/0x124
[  305.090879] [<ffffffc000129b10>] rcu_pending+0x50/0x84
[  305.090921] [<ffffffc00012c350>] rcu_check_callbacks+0xd8/0x164
[  305.090962] [<ffffffc0000b8b5c>] update_process_times+0x44/0x70
[  305.091003] [<ffffffc000100f10>] tick_sched_handle+0x54/0x60
[  305.091041] [<ffffffc000100f70>] tick_sched_timer+0x54/0x7c
[  305.091077] [<ffffffc0000d3f00>] __run_hrtimer+0x168/0x27c
[  305.091113] [<ffffffc0000d48d4>] hrtimer_interrupt+0x15c/0x278
[  305.091157] [<ffffffc0007abb70>] tegra210_timer_isr+0x3c/0x4c
[  305.091198] [<ffffffc000120ec8>] handle_irq_event_percpu+0xd0/0x21c
[  305.091236] [<ffffffc00012105c>] handle_irq_event+0x48/0x74
[  305.091278] [<ffffffc000124218>] handle_fasteoi_irq+0xc4/0x10c
[  305.091315] [<ffffffc000120394>] generic_handle_irq+0x28/0x40
[  305.091353] [<ffffffc0000859d0>] handle_IRQ+0x90/0xc4
[  305.091387] [<ffffffc0000813d0>] gic_handle_irq+0x74/0x184
[  305.091420] Exception stack(0xffffffc0bffd3bb0 to 0xffffffc0bffd3cd0)
[  305.091460] 3ba0:                                     00000000 00000000 00000004 00000000
[  305.091506] 3bc0: bffd3d10 ffffffc0 00aff990 ffffffc0 60000145 00000000 00000001 00000000
[  305.091547] 3be0: 00000004 00000000 00000002 00000000 00000000 00000000 00000308 00000000
[  305.091589] 3c00: ffff0144 00000000 00000004 00000000 00000144 00000000 0037fa3c ffffffc0
[  305.091631] 3c20: 00fe6e40 ffffffc0 00324d48 ffffffc0 00000006 00000000 ed0f5fb0 0000007f
[  305.091673] 3c40: ed0f5fb0 0000007f 00000010 00000000 00403ac8 00000000 00000000 00000000
[  305.091714] 3c60: 00403ad0 00000000 00000044 00000000 00000001 00000000 b43f0bc0 0000007f
[  305.091752] 3c80: 00000a03 00000000 00000000 00000000 00000004 00000000 00000144 00000000
[  305.091789] 3ca0: 00000002 00000000 00000004 00000000 00000002 00000000 00000004 00000000
[  305.091822] 3cc0: 00000002 00000000 bee26000 ffffffc0
[  305.091860] [<ffffffc000084e04>] el1_irq+0x84/0xf0
[  305.091903] [<ffffffc000365860>] pci_user_write_config_word+0x6c/0x90
[  305.091945] [<ffffffc00037060c>] pci_write_config+0x160/0x1a8
[  305.091986] [<ffffffc00021b888>] flush_write+0x6c/0xa4
[  305.092022] [<ffffffc00021bbf4>] write+0xb0/0xec
[  305.092061] [<ffffffc0001aac18>] vfs_write+0xc8/0x1b0
[  305.092096] [<ffffffc0001ab90c>] SyS_pwrite64+0xd8/0x188
[  305.092884] Bad mode in Error handler detected, code 0xbf000002
[  305.092897] CPU: 3 PID: 500 Comm: in:imklog Not tainted 3.10.96+ #1
[  305.092905] task: ffffffc0f4a94800 ti: ffffffc0f01e4000 task.ti: ffffffc0f01e4000
[  305.092913] PC is at 0x7f781589cc
[  305.092919] LR is at 0x7f781589b8
[  305.092925] pc : [<0000007f781589cc>] lr : [<0000007f781589b8>] pstate: 80000000
[  305.092931] sp : ffffffc0f01e7ff0
[  305.092937] x29: 0000007f775f2030 x28: 0000007f775f2890
[  305.092950] x27: 0000000000001f9f x26: 0000007f77f4b2b8
[  305.092962] x25: 0000007f77f5c5c0 x24: 0000000000001fa0
[  305.092973] x23: 0000007f775f2890 x22: 0000000000000000
[  305.092985] x21: 00000000004b125d x20: 00000000004b1240
[  305.092996] x19: 0000007f775f292d x18: 0000000000008090
[  305.093007] x17: 0000007f781589a0 x16: 0000000000000000
[  305.093018] x15: 0000000000008088 x14: 0000000000008140
[  305.093029] x13: 0000000000008130 x12: 0000000000008138
[  305.093040] x11: 0000000000008128 x10: 0000000000008120
[  305.093051] x9 : 0000007f775f1ff8 x8 : 000000000000003f
[  305.093062] x7 : 0000000000000000 x6 : 0000007f776138a0
[  305.093073] x5 : 0000007f776131b0 x4 : 0000000000000002
[  305.093084] x3 : 0000007f776132b8 x2 : 0000000000001fa0
[  305.093095] x1 : 0000007f775f2890 x0 : 0000000000000edd

[  305.093120] Bad mode in Synchronous Abort handler detected, code 0x8600000f
[  305.093128] CPU: 3 PID: 500 Comm: in:imklog Not tainted 3.10.96+ #1
[  305.093135] task: ffffffc0f4a94800 ti: ffffffc0f01e4000 task.ti: ffffffc0f01e4000
[  305.093142] PC is at 0x7f781589b8
[  305.093147] LR is at 0x7f781589b8
[  305.093153] pc : [<0000007f781589b8>] lr : [<0000007f781589b8>] pstate: 800003c5
[  305.093159] sp : ffffffc0f01e7ed0
[  305.093165] x29: 0000007f775f2030 x28: 0000007f775f2890
[  305.093176] x27: 0000000000001f9f x26: 0000007f77f4b2b8
[  305.093187] x25: 0000007f77f5c5c0 x24: 0000000000001fa0
[  305.093198] x23: 0000000080000000 x22: 0000007f781589cc
[  305.093210] x21: ffffffc0f01e7ff0 x20: 00000000004b1240
[  305.093221] x19: 0000007f775f292d x18: 0000000000008090
[  305.093232] x17: 0000007f781589a0 x16: 000000000000000a
[  305.093243] x15: 0000000000000020 x14: 000000000000005d
[  305.093258] x13: 0000000000000035 x12: 0000000000000030
[  305.093270] x11: 0000000000000031 x10: 0000000000000033
[  305.093281] x9 : 0000000000000039 x8 : 0000000000000006
[  305.093292] x7 : ffffffc0000860d4 x6 : ffffffc0f01e7e90
[  305.093303] x5 : 0000000000000007 x4 : 0000000000000012
[  305.093313] x3 : 0000000000000001 x2 : 0000000000000001
[  305.093324] x1 : ffffffc0f01e7dc0 x0 : 0000000000000000
[  305.093336]
               SP: 0xffffffc0f01e7e50:
[  305.093343] 7e50  004b1240 00000000 f01e7ff0 ffffffc0 781589cc 0000007f 80000000 00000000
[  305.093376] 7e70  00001fa0 00000000 77f5c5c0 0000007f 77f4b2b8 0000007f 00001f9f 00000000
[  305.093408] 7e90  775f2890 0000007f 775f2030 0000007f 781589b8 0000007f f01e7ed0 ffffffc0
[  305.093439] 7eb0  781589b8 0000007f 800003c5 00000000 775f292d 0000007f 004b1240 00000000
[  305.093471] 7ed0  00000edd 00000000 775f2890 0000007f 00001fa0 00000000 776132b8 0000007f
[  305.093502] 7ef0  00000002 00000000 776131b0 0000007f 776138a0 0000007f 00000000 00000000
[  305.093533] 7f10  0000003f 00000000 775f1ff8 0000007f 00008120 00000000 00008128 00000000
[  305.093565] 7f30  00008138 00000000 00008130 00000000 00008140 00000000 00008088 00000000
[  305.093597]
               X1: 0xffffffc0f01e7d40:
[  305.093604] 7d40  f01e7d90 ffffffc0 000bae9c ffffffc0 f01e7e50 ffffffc0 f4a94800 ffffffc0
[  305.093635] 7d60  00000004 00000000 00000000 00000000 f01e7e50 ffffffc0 00001fa0 00000000
[  305.093667] 7d80  f01e7dc0 ffffffc0 00aff940 ffffffc0 000003c0 00000000 00000000 00000000
[  305.093698] 7da0  00000004 00000000 000003c0 00000000 00000000 00000000 f01e7dc0 ffffffc0
[  305.093729] 7dc0  00000001 00000000 00000001 00000000 00000012 00000000 00000007 00000000
[  305.093760] 7de0  f01e7e90 ffffffc0 000860d4 ffffffc0 00000006 00000000 00000039 00000000
[  305.093791] 7e00  00000033 00000000 00000031 00000000 00000030 00000000 00000035 00000000
[  305.093823] 7e20  0000005d 00000000 00000020 00000000 0000000a 00000000 781589a0 0000007f
[  305.093856]
               X6: 0xffffffc0f01e7e10:
[  305.093862] 7e10  00000030 00000000 00000035 00000000 0000005d 00000000 00000020 00000000
[  305.093893] 7e30  0000000a 00000000 781589a0 0000007f 00008090 00000000 775f292d 0000007f
[  305.093924] 7e50  004b1240 00000000 f01e7ff0 ffffffc0 781589cc 0000007f 80000000 00000000
[  305.093956] 7e70  00001fa0 00000000 77f5c5c0 0000007f 77f4b2b8 0000007f 00001f9f 00000000
[  305.093987] 7e90  775f2890 0000007f 775f2030 0000007f 781589b8 0000007f f01e7ed0 ffffffc0
[  305.094020] 7eb0  781589b8 0000007f 800003c5 00000000 775f292d 0000007f 004b1240 00000000
[  305.094051] 7ed0  00000edd 00000000 775f2890 0000007f 00001fa0 00000000 776132b8 0000007f
[  305.094088] 7ef0  00000002 00000000 776131b0 0000007f 776138a0 0000007f 00000000 00000000
[  305.094120]
               X7: 0xffffffc000086054:
[  305.094126] 6054  f9400000 36080040 9429e0ce a8c17bfd d65f03c0 a9be7bfd 910003fd a90153f3
[  305.094157] 6074  52800020 94015b3e 52800060 910003e2 9272c441 940a764a 34000260 910003e0
[  305.094189] 6094  9272c414 f9400a93 91150273 aa1303e0 9400005b d53b4221 d50342df d0008740
[  305.094221] 60b4  91008000 b9804e83 b00099c2 9123e042 f8637842 f8226813 d51b4221 940ad21e
[  305.094254] 60d4  b9021260 52800020 94015b59 910003e1 9272c420 f9400000 36080040 9429e0a9
[  305.094294] 60f4  a94153f3 a8c27bfd d65f03c0 a9be7bfd 910003fd f9000bf3 aa0003f3 52800020
[  305.094327] 6114  94015b17 aa1303e0 9400003d 52800060 910003e2 9272c441 940a7621 34000220
[  305.094361] 6134  910003e0 9272c401 f9400833 d53b4222 d50342df d0008740 91008000 b9804c23
[  305.094400]
               X21: 0xffffffc0f01e7f70:
[  305.094407] 7f70  004b1240 00000000 004b125d 00000000 00000000 00000000 775f2890 0000007f
[  305.094441] 7f90  00001fa0 00000000 77f5c5c0 0000007f 77f4b2b8 0000007f 00001f9f 00000000
[  305.094474] 7fb0  775f2890 0000007f 775f2030 0000007f 781589b8 0000007f f01e7ff0 ffffffc0
[  305.094506] 7fd0  781589cc 0000007f 80000000 00000000 00000004 00000000 0000003f 00000000
[  305.094538] 7ff0  00000403 00000000 0015e4b8 00000000 00000000 00000000 00000000 00000080
[  305.094573] 8010  fdd14780 ffffffc0 000bce90 ffffffc0 00000000 00000000 00000000 00000000
[  305.094606] 8030  00000000 00000000 00000000 00000000 00000000 00000000 00000002 00000001
[  305.094639] 8050  00000100 00000000 57ac6e9d 00000000 97ffe680 f90002c0 b4000115 b94077a0

[  305.094683] Internal error: Oops - bad mode: 0 [#1] PREEMPT SMP
[  305.094692] Enter nvdumper_crash_setup_regs
[  305.094706] nvdumper: all registers are saved.
[  305.094708] nvdumper: all registers are saved.
[  305.094710] nvdumper: all registers are saved.
[  305.094728] nvdumper: all registers are saved.
[  305.094734] Modules linked in:
[  305.094746] CPU: 3 PID: 500 Comm: in:imklog Not tainted 3.10.96+ #1
[  305.094753] task: ffffffc0f4a94800 ti: ffffffc0f01e4000 task.ti: ffffffc0f01e4000
[  305.094764] PC is at 0x7f781589b8
[  305.094769] LR is at 0x7f781589b8
[  305.094775] pc : [<0000007f781589b8>] lr : [<0000007f781589b8>] pstate: 800003c5
[  305.094781] sp : ffffffc0f01e7ed0
[  305.094786] x29: 0000007f775f2030 x28: 0000007f775f2890
[  305.094800] x27: 0000000000001f9f x26: 0000007f77f4b2b8
[  305.094811] x25: 0000007f77f5c5c0 x24: 0000000000001fa0
[  305.094824] x23: 0000000080000000 x22: 0000007f781589cc
[  305.094835] x21: ffffffc0f01e7ff0 x20: 00000000004b1240
[  305.094846] x19: 0000007f775f292d x18: 0000000000008090
[  305.094857] x17: 0000007f781589a0 x16: 000000000000000a
[  305.094868] x15: 0000000000000020 x14: 000000000000005d
[  305.094880] x13: 0000000000000035 x12: 0000000000000030
[  305.094891] x11: 0000000000000031 x10: 0000000000000033
[  305.094902] x9 : 0000000000000039 x8 : 0000000000000006
[  305.094914] x7 : ffffffc0000860d4 x6 : ffffffc0f01e7e90
[  305.094924] x5 : 0000000000000007 x4 : 0000000000000012
[  305.094935] x3 : 0000000000000001 x2 : 0000000000000001
[  305.094948] x1 : ffffffc0f01e7dc0 x0 : 0000000000000000
[  305.094959]
               SP: 0xffffffc0f01e7e50:
[  305.094966] 7e50  004b1240 00000000 f01e7ff0 ffffffc0 781589cc 0000007f 80000000 00000000
[  305.094999] 7e70  00001fa0 00000000 77f5c5c0 0000007f 77f4b2b8 0000007f 00001f9f 00000000
[  305.095031] 7e90  775f2890 0000007f 775f2030 0000007f 781589b8 0000007f f01e7ed0 ffffffc0
[  305.095063] 7eb0  781589b8 0000007f 800003c5 00000000 775f292d 0000007f 004b1240 00000000
[  305.095095] 7ed0  00000edd 00000000 775f2890 0000007f 00001fa0 00000000 776132b8 0000007f
[  305.095127] 7ef0  00000002 00000000 776131b0 0000007f 776138a0 0000007f 00000000 00000000
[  305.095158] 7f10  0000003f 00000000 775f1ff8 0000007f 00008120 00000000 00008128 00000000
[  305.095191] 7f30  00008138 00000000 00008130 00000000 00008140 00000000 00008088 00000000
[  305.095223]
               X1: 0xffffffc0f01e7d40:
[  305.095230] 7d40  781589b8 0000007f 000bae9c ffffffc0 f01e7e50 ffffffc0 f4a94800 ffffffc0
[  305.095261] 7d60  00000004 00000000 00000000 00000000 f01e7e50 ffffffc0 00001fa0 00000000
[  305.095293] 7d80  f01e7dc0 ffffffc0 00aff940 ffffffc0 000003c0 00000000 00000000 00000000
[  305.095326] 7da0  00000004 00000000 000003c0 00000000 00000000 00000000 f01e7dc0 ffffffc0
[  305.095356] 7dc0  00000001 00000000 00000001 00000000 00000012 00000000 00000007 00000000
[  305.095387] 7de0  f01e7e90 ffffffc0 000860d4 ffffffc0 00000006 00000000 00000039 00000000
[  305.095419] 7e00  00000033 00000000 00000031 00000000 00000030 00000000 00000035 00000000
[  305.095450] 7e20  0000005d 00000000 00000020 00000000 0000000a 00000000 781589a0 0000007f
[  305.095483]
               X6: 0xffffffc0f01e7e10:
[  305.095490] 7e10  00000030 00000000 00000035 00000000 0000005d 00000000 00000020 00000000
[  305.095521] 7e30  0000000a 00000000 781589a0 0000007f 00008090 00000000 775f292d 0000007f
[  305.095553] 7e50  004b1240 00000000 f01e7ff0 ffffffc0 781589cc 0000007f 80000000 00000000
[  305.095585] 7e70  00001fa0 00000000 77f5c5c0 0000007f 77f4b2b8 0000007f 00001f9f 00000000
[  305.095617] 7e90  775f2890 0000007f 775f2030 0000007f 781589b8 0000007f f01e7ed0 ffffffc0
[  305.095649] 7eb0  781589b8 0000007f 800003c5 00000000 775f292d 0000007f 004b1240 00000000
[  305.095681] 7ed0  00000edd 00000000 775f2890 0000007f 00001fa0 00000000 776132b8 0000007f
[  305.095713] 7ef0  00000002 00000000 776131b0 0000007f 776138a0 0000007f 00000000 00000000
[  305.095745]
               X7: 0xffffffc000086054:
[  305.095752] 6054  f9400000 36080040 9429e0ce a8c17bfd d65f03c0 a9be7bfd 910003fd a90153f3
[  305.095783] 6074  52800020 94015b3e 52800060 910003e2 9272c441 940a764a 34000260 910003e0
[  305.095815] 6094  9272c414 f9400a93 91150273 aa1303e0 9400005b d53b4221 d50342df d0008740
[  305.095846] 60b4  91008000 b9804e83 b00099c2 9123e042 f8637842 f8226813 d51b4221 940ad21e
[  305.095876] 60d4  b9021260 52800020 94015b59 910003e1 9272c420 f9400000 36080040 9429e0a9
[  305.095907] 60f4  a94153f3 a8c27bfd d65f03c0 a9be7bfd 910003fd f9000bf3 aa0003f3 52800020
[  305.095939] 6114  94015b17 aa1303e0 9400003d 52800060 910003e2 9272c441 940a7621 34000220
[  305.095972] 6134  910003e0 9272c401 f9400833 d53b4222 d50342df d0008740 91008000 b9804c23
[  305.096007]
               X21: 0xffffffc0f01e7f70:
[  305.096014] 7f70  004b1240 00000000 004b125d 00000000 00000000 00000000 775f2890 0000007f
[  305.096046] 7f90  00001fa0 00000000 77f5c5c0 0000007f 77f4b2b8 0000007f 00001f9f 00000000
[  305.096077] 7fb0  775f2890 0000007f 775f2030 0000007f 781589b8 0000007f f01e7ff0 ffffffc0
[  305.096108] 7fd0  781589cc 0000007f 80000000 00000000 00000004 00000000 0000003f 00000000
[  305.096139] 7ff0  00000403 00000000 0015e4b8 00000000 00000000 00000000 00000000 00000080
[  305.096172] 8010  fdd14780 ffffffc0 000bce90 ffffffc0 00000000 00000000 00000000 00000000
[  305.096204] 8030  00000000 00000000 00000000 00000000 00000000 00000000 00000002 00000001
[  305.096236] 8050  00000100 00000000 57ac6e9d 00000000 97ffe680 f90002c0 b4000115 b94077a0

[  305.096275] Process in:imklog (pid: 500, stack limit = 0xffffffc0f01e4058)
[  305.096284] Stack: (0xffffffc0f01e7ed0 to 0xffffffc0f01e8000)
[  305.096294] 7ec0:                                     00000edd 00000000 775f2890 0000007f
[  305.096302] 7ee0: 00001fa0 00000000 776132b8 0000007f 00000002 00000000 776131b0 0000007f
[  305.096311] 7f00: 776138a0 0000007f 00000000 00000000 0000003f 00000000 775f1ff8 0000007f
[  305.096321] 7f20: 00008120 00000000 00008128 00000000 00008138 00000000 00008130 00000000
[  305.096329] 7f40: 00008140 00000000 00008088 00000000 00000000 00000000 781589a0 0000007f
[  305.096337] 7f60: 00008090 00000000 775f292d 0000007f 004b1240 00000000 004b125d 00000000
[  305.096345] 7f80: 00000000 00000000 775f2890 0000007f 00001fa0 00000000 77f5c5c0 0000007f
[  305.096353] 7fa0: 77f4b2b8 0000007f 00001f9f 00000000 775f2890 0000007f 775f2030 0000007f
[  305.096361] 7fc0: 781589b8 0000007f f01e7ff0 ffffffc0 781589cc 0000007f 80000000 00000000
[  305.096369] 7fe0: 00000004 00000000 0000003f 00000000 00000403 00000000 0015e4b8 00000000
[  305.096376] Call trace:
[  305.096386] Code: 34fffe50 a9bc03fe a9010be1 97fffe45 (aa0003f0)
[  305.096654] ---[ end trace 7f2efc76296d9c38 ]---

#setpci -d 8086:1572 04.W=0x0142
[  118.103778] Bad mode in Error handler detected, code 0xbf000002
[  118.103929] CPU: 1 PID: 1531 Comm: gmain Not tainted 3.10.96+ #1
[  118.104037] task: ffffffc0de6e3180 ti: ffffffc0c07ac000 task.ti: ffffffc0c07ac000
[  118.104130] PC is at 0x7f979ef0e4
[  118.104207] LR is at 0x7f979ef0c4
[  118.104286] pc : [<0000007f979ef0e4>] lr : [<0000007f979ef0c4>] pstate: 80000000
[  118.104353] sp : ffffffc0c07afff0
[  118.104420] x29: 0000007f967625e0 x28: 0000007f96762ef0
[  118.104569] x27: 0000000000000001 x26: 0000000000000001
[  118.104707] x25: 0000000000000001 x24: 0000007f97d93000
[  118.104839] x23: 0000007f97cea2a0 x22: 0000000000000f9b
[  118.104970] x21: 0000007f96762610 x20: 0000007f900008e0
[  118.105098] x19: 0000000000000002 x18: 000000000044a590
[  118.105225] x17: 0000007f979ef018 x16: 0000007f97d94220
[  118.105351] x15: 003b9aca00000000 x14: 001e458bd3000000
[  118.105477] x13: ffffffffa7cba445 x12: 0000000000000018
[  118.105604] x11: 0000000032dc1d3b x10: 000000000000005e
[  118.105728] x9 : 0000000000003d92 x8 : 0000000000000049
[  118.105854] x7 : 0000000000000000 x6 : 0000007f967635e0
[  118.105977] x5 : 0000000000000000 x4 : 0000000000000000
[  118.106101] x3 : 0000000000000000 x2 : 0000007f96762610
[  118.106227] x1 : 0000000000000002 x0 : 0000000000000000

[  118.106496] Bad mode in Synchronous Abort handler detected, code 0x8600000f
[  118.106594] CPU: 1 PID: 1531 Comm: gmain Not tainted 3.10.96+ #1
[  118.106677] task: ffffffc0de6e3180 ti: ffffffc0c07ac000 task.ti: ffffffc0c07ac000
[  118.106760] PC is at 0x7f979ef0c4
[  118.106826] LR is at 0x7f979ef0c4
[  118.106901] pc : [<0000007f979ef0c4>] lr : [<0000007f979ef0c4>] pstate: 800003c5
[  118.106964] sp : ffffffc0c07afed0
[  118.107027] x29: 0000007f967625e0 x28: 0000007f96762ef0
[  118.107162] x27: 0000000000000001 x26: 0000000000000001
[  118.107289] x25: 0000000000000001 x24: 0000007f97d93000
[  118.107417] x23: 0000000080000000 x22: 0000007f979ef0e4
[  118.107546] x21: ffffffc0c07afff0 x20: 0000007f900008e0
[  118.107670] x19: 0000000000000002 x18: 000000000044a590
[  118.107795] x17: 0000007f979ef018 x16: 000000000000000a
[  118.107920] x15: 0000000000000020 x14: 000000000000005d
[  118.108043] x13: 0000000000000034 x12: 0000000000000034
[  118.108168] x11: 0000000000000033 x10: 0000000000000036
[  118.108293] x9 : 0000000000000030 x8 : 0000000000000006
[  118.108416] x7 : ffffffc0000860d4 x6 : ffffffc0c07afe90
[  118.108542] x5 : ffffffc001177896 x4 : 0000000000000010
[  118.108668] x3 : 0000000000000001 x2 : 0000000000000001
[  118.108790] x1 : ffffffc0c07afda0 x0 : 0000000000000000
[  118.108922]
               SP: 0xffffffc0c07afe50:
[  118.109002] fe50  900008e0 0000007f c07afff0 ffffffc0 979ef0e4 0000007f 80000000 00000000
[  118.109363] fe70  97d93000 0000007f 00000001 00000000 00000001 00000000 00000001 00000000
[  118.109714] fe90  96762ef0 0000007f 967625e0 0000007f 979ef0c4 0000007f c07afed0 ffffffc0
[  118.110065] feb0  979ef0c4 0000007f 800003c5 00000000 00000002 00000000 900008e0 0000007f
[  118.110413] fed0  00000000 00000000 00000002 00000000 96762610 0000007f 00000000 00000000
[  118.110765] fef0  00000000 00000000 00000000 00000000 967635e0 0000007f 00000000 00000000
[  118.111118] ff10  00000049 00000000 00003d92 00000000 0000005e 00000000 32dc1d3b 00000000
[  118.111466] ff30  00000018 00000000 a7cba445 ffffffff d3000000 001e458b 00000000 003b9aca
[  118.111836]
               X1: 0xffffffc0c07afd20:
[  118.111911] fd20  00000002 00000000 900008e0 0000007f de6e3180 ffffffc0 96762ef0 0000007f
[  118.112261] fd40  c07afd90 ffffffc0 000bae9c ffffffc0 c07afe50 ffffffc0 de6e3180 ffffffc0
[  118.112612] fd60  00000004 00000000 00000000 00000000 c07afe50 ffffffc0 97d93000 0000007f
[  118.112957] fd80  c07afdc0 ffffffc0 00aff940 ffffffc0 000003c0 00000000 00000000 00000000
[  118.113304] fda0  c07afdc0 ffffffc0 00aff968 ffffffc0 00000000 00000000 c07afda0 ffffffc0
[  118.113654] fdc0  00000001 00000000 00000001 00000000 00000010 00000000 01177896 ffffffc0
[  118.114000] fde0  c07afe90 ffffffc0 000860d4 ffffffc0 00000006 00000000 00000030 00000000
[  118.114349] fe00  00000036 00000000 00000033 00000000 00000034 00000000 00000034 00000000
[  118.114720]
               X5: 0xffffffc001177816:
[  118.114793] 7814  00009d40 04040001 ff30dd68 00009d38 04040001 ff30e1f4 00009d30 04040001
[  118.115151] 7834  ff30e8a8 00009d28 04040001 ff30e924 00009d20 04040001 ff30e9dc 00009d18
[  118.115503] 7854  04040001 ff30ea74 00009d10 04040001 ff30eb04 00009d08 04040001 ff30ebb0
[  118.115855] 7874  00009d00 04040001 ff30f94c 00009cf8 04040001 ff30fad0 00009cf0 04040001
[  118.116207] 7894  ff30fbbc 00009ce8 04040001 ff312390 00009ce0 04040001 ff312534 00009cd8
[  118.116559] 78b4  04040001 ff3126d8 00009cd0 04040001 ff3129bc 00009cc8 04040001 ff312a2c
[  118.116920] 78d4  00009cc0 04040001 ff312aa0 00009cb8 04040001 ff312b10 00009cb0 04040001
[  118.116972] 78f4  ff312b84 00009ca8 04040001 ff312bf4 00009ca0 04040001 ff313774 00009c98
[  118.117006] 7914  04040001 ff31396c 00009c90 04040001 ff313b54 00009c88 04040001 ff313c58
[  118.117040]
               X6: 0xffffffc0c07afe10:
[  118.117047] fe10  00000034 00000000 00000034 00000000 0000005d 00000000 00000020 00000000
[  118.117080] fe30  0000000a 00000000 979ef018 0000007f 0044a590 00000000 00000002 00000000
[  118.117113] fe50  900008e0 0000007f c07afff0 ffffffc0 979ef0e4 0000007f 80000000 00000000
[  118.117155] fe70  97d93000 0000007f 00000001 00000000 00000001 00000000 00000001 00000000
[  118.117191] fe90  96762ef0 0000007f 967625e0 0000007f 979ef0c4 0000007f c07afed0 ffffffc0
[  118.117224] feb0  979ef0c4 0000007f 800003c5 00000000 00000002 00000000 900008e0 0000007f
[  118.117257] fed0  00000000 00000000 00000002 00000000 96762610 0000007f 00000000 00000000
[  118.117291] fef0  00000000 00000000 00000000 00000000 967635e0 0000007f 00000000 00000000
[  118.117326]
               X7: 0xffffffc000086054:
[  118.117334] 6054  f9400000 36080040 9429e0ce a8c17bfd d65f03c0 a9be7bfd 910003fd a90153f3
[  118.117367] 6074  52800020 94015b3e 52800060 910003e2 9272c441 940a764a 34000260 910003e0
[  118.117400] 6094  9272c414 f9400a93 91150273 aa1303e0 9400005b d53b4221 d50342df d0008740
[  118.117433] 60b4  91008000 b9804e83 b00099c2 9123e042 f8637842 f8226813 d51b4221 940ad21e
[  118.117466] 60d4  b9021260 52800020 94015b59 910003e1 9272c420 f9400000 36080040 9429e0a9
[  118.117500] 60f4  a94153f3 a8c27bfd d65f03c0 a9be7bfd 910003fd f9000bf3 aa0003f3 52800020
[  118.117533] 6114  94015b17 aa1303e0 9400003d 52800060 910003e2 9272c441 940a7621 34000220
[  118.117566] 6134  910003e0 9272c401 f9400833 d53b4222 d50342df d0008740 91008000 b9804c23
[  118.117605]
               X21: 0xffffffc0c07aff70:
[  118.117611] ff70  900008e0 0000007f 96762610 0000007f 00000f9b 00000000 97cea2a0 0000007f
[  118.117646] ff90  97d93000 0000007f 00000001 00000000 00000001 00000000 00000001 00000000
[  118.117678] ffb0  96762ef0 0000007f 967625e0 0000007f 979ef0c4 0000007f c07afff0 ffffffc0
[  118.117711] ffd0  979ef0e4 0000007f 80000000 00000000 900008e0 0000007f 00000049 00000000
[  118.117744] fff0  00000043 00007b73 00017c00 6b00007c 00000000 00000000 00000000 00000080
[  118.117778] 0010  cb46a6c0 ffffffc0 000bce90 ffffffc0 00000000 00000000 00000000 00000000
[  118.117811] 0030  00000000 00000000 00000000 00000000 00000000 00000000 00000002 00000002
[  118.117844] 0050  00000100 00000000 57ac6e9d 00000000 5f5f7469 7970632e 6e6f6874 2e35332d

[  118.117889] Internal error: Oops - bad mode: 0 [#1] PREEMPT SMP
[  118.117899] Enter nvdumper_crash_setup_regs
[  118.117914] nvdumper: all registers are saved.
[  118.117916] nvdumper: all registers are saved.
[  118.117918] nvdumper: all registers are saved.
[  118.117937] nvdumper: all registers are saved.
[  118.117943] Modules linked in:
[  118.117955] CPU: 1 PID: 1531 Comm: gmain Not tainted 3.10.96+ #1
[  118.117963] task: ffffffc0de6e3180 ti: ffffffc0c07ac000 task.ti: ffffffc0c07ac000
[  118.117972] PC is at 0x7f979ef0c4
[  118.117977] LR is at 0x7f979ef0c4
[  118.117983] pc : [<0000007f979ef0c4>] lr : [<0000007f979ef0c4>] pstate: 800003c5
[  118.117989] sp : ffffffc0c07afed0
[  118.117995] x29: 0000007f967625e0 x28: 0000007f96762ef0
[  118.118007] x27: 0000000000000001 x26: 0000000000000001
[  118.118018] x25: 0000000000000001 x24: 0000007f97d93000
[  118.118030] x23: 0000000080000000 x22: 0000007f979ef0e4
[  118.118042] x21: ffffffc0c07afff0 x20: 0000007f900008e0
[  118.118053] x19: 0000000000000002 x18: 000000000044a590
[  118.118064] x17: 0000007f979ef018 x16: 000000000000000a
[  118.118075] x15: 0000000000000020 x14: 000000000000005d
[  118.118086] x13: 0000000000000034 x12: 0000000000000034
[  118.118097] x11: 0000000000000033 x10: 0000000000000036
[  118.118108] x9 : 0000000000000030 x8 : 0000000000000006
[  118.118119] x7 : ffffffc0000860d4 x6 : ffffffc0c07afe90
[  118.118130] x5 : ffffffc001177896 x4 : 0000000000000010
[  118.118141] x3 : 0000000000000001 x2 : 0000000000000001
[  118.118152] x1 : ffffffc0c07afda0 x0 : 0000000000000000
[  118.118163]
               SP: 0xffffffc0c07afe50:
[  118.118169] fe50  900008e0 0000007f c07afff0 ffffffc0 979ef0e4 0000007f 80000000 00000000
[  118.118201] fe70  97d93000 0000007f 00000001 00000000 00000001 00000000 00000001 00000000
[  118.118232] fe90  96762ef0 0000007f 967625e0 0000007f 979ef0c4 0000007f c07afed0 ffffffc0
[  118.118263] feb0  979ef0c4 0000007f 800003c5 00000000 00000002 00000000 900008e0 0000007f
[  118.118294] fed0  00000000 00000000 00000002 00000000 96762610 0000007f 00000000 00000000
[  118.118326] fef0  00000000 00000000 00000000 00000000 967635e0 0000007f 00000000 00000000
[  118.118356] ff10  00000049 00000000 00003d92 00000000 0000005e 00000000 32dc1d3b 00000000
[  118.118388] ff30  00000018 00000000 a7cba445 ffffffff d3000000 001e458b 00000000 003b9aca
[  118.118421]
               X1: 0xffffffc0c07afd20:
[  118.118427] fd20  00000002 00000000 900008e0 0000007f 00000004 00000000 00030001 0000007f
[  118.118458] fd40  979ef0c4 0000007f 000bae9c ffffffc0 c07afe50 ffffffc0 de6e3180 ffffffc0
[  118.118489] fd60  00000004 00000000 00000000 00000000 c07afe50 ffffffc0 97d93000 0000007f
[  118.118520] fd80  c07afdc0 ffffffc0 00aff940 ffffffc0 000003c0 00000000 00000000 00000000
[  118.118552] fda0  c07afdc0 ffffffc0 00aff968 ffffffc0 00000000 00000000 c07afda0 ffffffc0
[  118.118583] fdc0  00000001 00000000 00000001 00000000 00000010 00000000 01177896 ffffffc0
[  118.118614] fde0  c07afe90 ffffffc0 000860d4 ffffffc0 00000006 00000000 00000030 00000000
[  118.118645] fe00  00000036 00000000 00000033 00000000 00000034 00000000 00000034 00000000
[  118.118678]
               X5: 0xffffffc001177816:
[  118.118684] 7814  00009d40 04040001 ff30dd68 00009d38 04040001 ff30e1f4 00009d30 04040001
[  118.118716] 7834  ff30e8a8 00009d28 04040001 ff30e924 00009d20 04040001 ff30e9dc 00009d18
[  118.118747] 7854  04040001 ff30ea74 00009d10 04040001 ff30eb04 00009d08 04040001 ff30ebb0
[  118.118778] 7874  00009d00 04040001 ff30f94c 00009cf8 04040001 ff30fad0 00009cf0 04040001
[  118.118809] 7894  ff30fbbc 00009ce8 04040001 ff312390 00009ce0 04040001 ff312534 00009cd8
[  118.118841] 78b4  04040001 ff3126d8 00009cd0 04040001 ff3129bc 00009cc8 04040001 ff312a2c
[  118.118872] 78d4  00009cc0 04040001 ff312aa0 00009cb8 04040001 ff312b10 00009cb0 04040001
[  118.118903] 78f4  ff312b84 00009ca8 04040001 ff312bf4 00009ca0 04040001 ff313774 00009c98
[  118.118934] 7914  04040001 ff31396c 00009c90 04040001 ff313b54 00009c88 04040001 ff313c58
[  118.118966]
               X6: 0xffffffc0c07afe10:
[  118.118973] fe10  00000034 00000000 00000034 00000000 0000005d 00000000 00000020 00000000
[  118.119004] fe30  0000000a 00000000 979ef018 0000007f 0044a590 00000000 00000002 00000000
[  118.119035] fe50  900008e0 0000007f c07afff0 ffffffc0 979ef0e4 0000007f 80000000 00000000
[  118.119066] fe70  97d93000 0000007f 00000001 00000000 00000001 00000000 00000001 00000000
[  118.119097] fe90  96762ef0 0000007f 967625e0 0000007f 979ef0c4 0000007f c07afed0 ffffffc0
[  118.119128] feb0  979ef0c4 0000007f 800003c5 00000000 00000002 00000000 900008e0 0000007f
[  118.119159] fed0  00000000 00000000 00000002 00000000 96762610 0000007f 00000000 00000000
[  118.119190] fef0  00000000 00000000 00000000 00000000 967635e0 0000007f 00000000 00000000
[  118.119222]
               X7: 0xffffffc000086054:
[  118.119228] 6054  f9400000 36080040 9429e0ce a8c17bfd d65f03c0 a9be7bfd 910003fd a90153f3
[  118.119260] 6074  52800020 94015b3e 52800060 910003e2 9272c441 940a764a 34000260 910003e0
[  118.119291] 6094  9272c414 f9400a93 91150273 aa1303e0 9400005b d53b4221 d50342df d0008740
[  118.119322] 60b4  91008000 b9804e83 b00099c2 9123e042 f8637842 f8226813 d51b4221 940ad21e
[  118.119354] 60d4  b9021260 52800020 94015b59 910003e1 9272c420 f9400000 36080040 9429e0a9
[  118.119385] 60f4  a94153f3 a8c27bfd d65f03c0 a9be7bfd 910003fd f9000bf3 aa0003f3 52800020
[  118.119416] 6114  94015b17 aa1303e0 9400003d 52800060 910003e2 9272c441 940a7621 34000220
[  118.119447] 6134  910003e0 9272c401 f9400833 d53b4222 d50342df d0008740 91008000 b9804c23
[  118.119483]
               X21: 0xffffffc0c07aff70:
[  118.119489] ff70  900008e0 0000007f 96762610 0000007f 00000f9b 00000000 97cea2a0 0000007f
[  118.119520] ff90  97d93000 0000007f 00000001 00000000 00000001 00000000 00000001 00000000
[  118.119551] ffb0  96762ef0 0000007f 967625e0 0000007f 979ef0c4 0000007f c07afff0 ffffffc0
[  118.119582] ffd0  979ef0e4 0000007f 80000000 00000000 900008e0 0000007f 00000049 00000000
[  118.119613] fff0  00000043 00007b73 00017c00 6b00007c 00000000 00000000 00000000 00000080
[  118.119644] 0010  cb46a6c0 ffffffc0 000bce90 ffffffc0 00000000 00000000 00000000 00000000
[  118.119675] 0030  00000000 00000000 00000000 00000000 00000000 00000000 00000002 00000002
[  118.119706] 0050  00000100 00000000 57ac6e9d 00000000 5f5f7469 7970632e 6e6f6874 2e35332d

[  118.119747] Process gmain (pid: 1531, stack limit = 0xffffffc0c07ac058)
[  118.119755] Stack: (0xffffffc0c07afed0 to 0xffffffc0c07b0000)
[  118.119762] fec0:                                     00000000 00000000 00000002 00000000
[  118.119771] fee0: 96762610 0000007f 00000000 00000000 00000000 00000000 00000000 00000000
[  118.119780] ff00: 967635e0 0000007f 00000000 00000000 00000049 00000000 00003d92 00000000
[  118.119789] ff20: 0000005e 00000000 32dc1d3b 00000000 00000018 00000000 a7cba445 ffffffff
[  118.119797] ff40: d3000000 001e458b 00000000 003b9aca 97d94220 0000007f 979ef018 0000007f
[  118.119805] ff60: 0044a590 00000000 00000002 00000000 900008e0 0000007f 96762610 0000007f
[  118.119814] ff80: 00000f9b 00000000 97cea2a0 0000007f 97d93000 0000007f 00000001 00000000
[  118.119822] ffa0: 00000001 00000000 00000001 00000000 96762ef0 0000007f 967625e0 0000007f
[  118.119830] ffc0: 979ef0c4 0000007f c07afff0 ffffffc0 979ef0e4 0000007f 80000000 00000000
[  118.119838] ffe0: 900008e0 0000007f 00000049 00000000 00000043 00007b73 00017c00 6b00007c
[  118.119845] Call trace:
[  118.119853] Code: 34fffde2 aa0103f3 aa0003f4 9400517c (2a0003e5)
[  118.120006] ---[ end trace c47dd65dc1d587f7 ]---
5

Now that it is hanging with BME also, it is most likely due to Intel card sending a VDM (Vendor Defined Message) upstream to root port and TX1 not being able to handle VDM, which is causing this hang.
To confirm this, either we can connect a PCIe logical analyzer and then capture VDM going upstream or can ask Intel if they can provide this info.

6

From the datasheet of Intel X710 it does look like the card sends VDM. I am checking with Intel to see there is anything they can do to turn it off. I am not sure what the purpose of the VDM is so not sure if turning it off is an option.

Is there anything that can be done on the TX1 side to add support for VDM? How much work do you think it would be?

7

Is there anything that can be done on the TX1 side to add support for VDM? How much work do you think it would be?
Sorry, but there is no way to program TX1 PCIe to handle VDM