DMA Controller on the IDT switch unable to access Tegra tx2 system memory address

kalpana_sethi · April 15, 2019, 11:42am

We have a set up with tegra Tx2 running Jetpack 3.3 connected on one of the pcie ports of the switch configured as upstream with DMA,

We disabled smmu from the Pcie controller in dts file

The DMA of the IDT switch is enumerated as function 2 of the switch and the config space is mapped to BAR0. The DMA requires descriptor address, source address and destination address .

As to start we wanted to transfer from memory within the host with dma on the IDT switch.
We have allocated memory from pci_alloc_coherent for descriptor, source and destination ( Ex desc 0xfe000000, 0xfec00000, 0xfed00000 resp and provided the physical addresses to the DMA mapped on BAR0

00:01.0 PCI bridge: NVIDIA Corporation Device 10e5 (rev a1) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 409
Bus: primary=00, secondary=01, subordinate=02, sec-latency=0
Memory behind bridge: 51000000-53ffffff
Prefetchable memory behind bridge: 0000000058000000-0000000059ffffff
Capabilities:
Kernel driver in use: pcieport

01:00.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES32NT8AG2 (rev 02) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
Capabilities:
Kernel driver in use: pcieport

01:00.1 Bridge: Integrated Device Technology, Inc. [IDT] PES32NT8AG2 (rev 02)
Flags: bus master, fast devsel, latency 0, IRQ 439
Memory at 51000000 (64-bit, non-prefetchable)
Memory at 58000000 (64-bit, prefetchable)
Memory at 52000000 (64-bit, non-prefetchable)
Capabilities:
Kernel driver in use: idt-g2-ntb

01:00.2 System peripheral: Integrated Device Technology, Inc. [IDT] PES32NT8AG2 (rev 02)
Flags: fast devsel, IRQ 409
Memory at 51001000 (64-bit, non-prefetchable) [disabled]
Capabilities:

The issue we are facing is the DMA controller is unable to see the Host memory as we see “unsupported request”.

We wanted to know do we need to expose the whole system memory to the end device using the device tree dma-range for address translation?
what are the possible setting we need to check ?

Regards,
kalpana

ahuska · April 20, 2019, 12:14am

Hello,

I see you marked your question as answered, may I ask what the answers were to your questions?

Andy

kalpana_sethi · April 22, 2019, 5:24am

Hi Andy,

Sorry i didn’t realize i marked as answered.I corrected it. I still didn’t get answer to my question.

vidyas · April 24, 2019, 5:31am

Whatever you are trying to do should work with SMMU enabled as well (and of course it can work with SMMU disabled). When you say, you are writing descriptor, source and destination addresses to BAR-0 (which is where I assume DMA controller registers are present) of Function-2, how are you writing them exactly? do you have a device driver for Function-2 which takes care of it? If yes, can you please share the driver privately for us to take a look at it? (You can send it as a message in this forum to me). If not, are you sure that you are enabling bus mastering for Function-2?

kalpana_sethi · April 24, 2019, 2:11pm

Yes bus mastering is enabled in type 0 configuration register of DMA controller PCI cmd register value 0x100007

how are you writing them exactly?
Ans=>

created pool

/*
- Create the DMA coherent memory desc pool
  */
  idt_dev->desc_pool = pci_pool_create(“idt_desc_pool”, idt_dev->pdev,
  sizeof(struct idt_g2_dma_desc),
  IDT_G2_DMA_DESC_ALIGN,
  IDT_G2_DMA_DESC_BOUNDARY);
  if (!idt_dev->desc_pool) {
  DMA_LOG_ERR(“%s: Failed to create DMA coherent memory pool \n”,
  FUNCTION);
  rval = -ENOMEM;
  goto lerror;
  }

 hwdesc = pci_pool_alloc(idt_dev->desc_pool, GFP_KERNEL, &phys);

Allocating hwdesc buffer address to hold 8 Dword descriptor info. got physical address 0xFC202000 (descriptor address )

To start with i did immediate data transfer ( IDT switch DMA supports this ) We were trying to copy 8 bytes using DMA to destination address physical address dst_buf = kzalloc(MEM_SIZE, GFP_ATOMIC);
DEST Address 0xdb800000
The descriptor is as follows nvidia@tegra-ubuntu:~/source$ sudo busybox devmem 0xfc202000
0x44000010
nvidia@tegra-ubuntu:~/source$ sudo busybox devmem 0xfc202004
0x00000008
nvidia@tegra-ubuntu:~/source$ sudo busybox devmem 0xfc202008
0x0C0C0C0C
nvidia@tegra-ubuntu:~/source$ sudo busybox devmem 0xfc20200c
0x0C0C0C0C
nvidia@tegra-ubuntu:~/source$ sudo busybox devmem 0xfc202010
0xDB800000

sudo busybox devmem 0x51001528 0xFC202000 ( this address set to the DMA controller )

4)When i set the run bit i see Abort and Halt with error as " unsupported Request "

Manikanta · April 27, 2019, 6:20am

Hi,

Are you trying to transfer data from host to IDT switch or IDT switch to host. Based on the description of this post you are trying to transfer data from host to IDT swicth, but in the recent comment you said that memory allocated by Tegra is used as destination address. Since the DMA is in IDT switch, to transfer data from host to IDT switch, Tegra address will be source address not destination. Could you check it and confirm?
I would like to know DMA HW desc format definition, is it possible to share this information from IDT switch datasheet? If there is any link to download IDT switch DMA datasheet, please share it.
I believe you are observing “unsupported request” AER in Jetson-Tx2 dmesg logs, could you share it?
Can you try DMA write instead of read and share your observations?
Please share “sudo lspci -vvv” output
Please share below command output,
“sudo ls /sys/kernel/debug/12000000.iommu/masters”
Please replace kzalloc() with dma_alloc_coherent(). If SMMU is enabled, it’ll give IOVA address. If SMMU is disabled IOVA will be same as PHY address. It is safe function to allocate DMA’able buffer.

Thanks,
Manikanta

Manikanta · April 27, 2019, 6:44am

Hi,

Also I don’t see mem flag in lspci output, could you please check that memory space access is enabled using pci_enable_device()/pci_enable_device_mem() function?

Thanks,
Manikanta

kalpana_sethi · April 29, 2019, 3:34am

Answer for your above 5 question:

i tried all 3 possible things.

a) src, dst ,desc in Host memory ( As IDT switch recommends it to be possible)
b) src and desc in host memory and dst in NT shared partition
c) desc in host memory and src , dst from bar ( NT shared partition)

All there are not working .

First of all if i set dummy descriptor and stride descriptor which is recommended by IDT itself is failing .It does not do a transfer.It just set descriptor. First setting descriptor with host memory need to be solved.

refer this link for all support for IDT DMA

https://www.idt.com/document/apn/714-dma-pcie-switches

I see this error in DMA controller of IDT ( DMA status register )
Tried all possible combination. See answer 1
00:01.0 PCI bridge: NVIDIA Corporation Device 10e5 (rev a1) (prog-if 00 [Normal decode])
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 409
Bus: primary=00, secondary=01, subordinate=02, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: 51000000-53ffffff
Prefetchable memory behind bridge: 0000000058000000-0000000059ffffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Subsystem: NVIDIA Corporation Device 0000
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/2 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [60] HyperTransport: MSI Mapping Enable- Fixed-
Mapping Address Base: 00000000fee00000
Capabilities: [80] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Off, PwrInd On, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
Changed: MRL- PresDet+ LinkState+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Kernel driver in use: pcieport

01:00.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES32NT8AG2 (rev 02) (prog-if 00 [Normal decode])
Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0
Bus: primary=01, secondary=02, subordinate=02, sec-latency=0
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Express (v2) Upstream Port, MSI 00
DevCap: MaxPayload 2048 bytes, PhantFunc 0
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #8, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [c0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [200 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [320 v1] Access Control Services
ACSCap: SrcValid- TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd- EgressCtrl- DirectTrans+
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Kernel driver in use: pcieport

01:00.1 Bridge: Integrated Device Technology, Inc. [IDT] PES32NT8AG2 (rev 02)
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0
Interrupt: pin D routed to IRQ 440
Region 0: Memory at 51000000 (64-bit, non-prefetchable)
Region 2: Memory at 58000000 (64-bit, prefetchable)
Region 4: Memory at 52000000 (64-bit, non-prefetchable)
Capabilities: [40] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #8, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [c0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 0000000266258000 Data: 0001
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [330 v1] #12
Kernel driver in use: idt-g2-ntb

01:00.2 System peripheral: Integrated Device Technology, Inc. [IDT] PES32NT8AG2 (rev 02)
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0
Interrupt: pin C routed to IRQ 439
Region 0: Memory at 51001000 (64-bit, non-prefetchable)
Capabilities: [40] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #8, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <4us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [c0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 0000000266258000 Data: 0000
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Kernel driver in use: idt-g2-dma

13e10000.host1x 13e10000.host1x:ctx4 15100000.tsecb 15500000.tsec 15830000.se 2993000.adsp 3180000.i2c 3210000.spi 3530000.xhci c250000.i2c
13e10000.host1x:ctx0 13e10000.host1x:ctx5 15340000.vic 15600000.isp 15840000.se 3100000.serial 3190000.i2c 3240000.spi 3550000.xudc c260000.spi
13e10000.host1x:ctx1 13e10000.host1x:ctx6 15380000.nvjpg 15700000.vi 17000000.gp10b 3110000.serial 31b0000.i2c 3400000.sdhci b000000.rtcpu c280000.serial
13e10000.host1x:ctx2 13e10000.host1x:ctx7 15480000.nvdec 15810000.se 2490000.ether_qos 3130000.serial 31c0000.i2c 3460000.sdhci c1a0000.aon d000000.bpmp
13e10000.host1x:ctx3 150c0000.nvcsi 154c0000.nvenc 15820000.se 2600000.dma 3160000.i2c 31e0000.i2c 3507000.ahci-sata c240000.i2c smmu_test
DMA Memory - idt_dev->dma_local_mem: vaddr ffffffc07c102000 bus_addr 00000000fc102000 size 0x00001000
[ 67.025889] DMA Memory -Single vaddr 00000000fc102000

I tried to use this physical address for DMA desc and tried immediate transfer .Still same issue

SMMU is disabled . So i used physical address

Later question:

Yes for both function 1 and function 2 of the IDT switch pci_enable_device() is done.

Manikanta · April 29, 2019, 4:38am

Hi,

Mem access is not enabled for upstream port of the switch. Enable it using below command and give it a try,
“sudo setpci -s 0000.01:00.0 0x4.w” #First read the value
“sudo setpci -s 0000.01:00.0 0x4.w=< | 0x7>” #Enable bus master, IO & mem access

01:00.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES32NT8AG2 (rev 02) (prog-if 00 [Normal decode])
Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-

Manikanta

kalpana_sethi · April 29, 2019, 10:38am

Hi Manikanta,

Thanks for your input. After enabling the em access for the function 0 of the IDT switch , DMA trasfer is working .

Thanks a lot.

Manikanta · April 29, 2019, 11:04am

Hi,

Glad to hear that DMA is working. Please check why Mem access not enabled for switch upstream port by default.
Probably idt-g2-ntb driver need to call pci_enable_device()/pci_enable_device_mem() for parent pci_dev.

-Manikanta

kalpana_sethi · April 29, 2019, 11:18am

Sure, I am looking into it only. We have registered driver only for function 1 and function 2.

Might be we need to add small drive to enable function 0.

Manikanta · April 29, 2019, 11:24am

01:00.0 is an upstream port, pcieport driver from PCIe subsystem is binded to it. portdrv_core.c driver enables mem access using pci_enable_device(). pcieport driver should be sufficient for upstream port.

-Manikanta

kalpana.jeevraj · May 24, 2019, 12:38pm

portdrv_core.c, i see that pci_enable_device() is called. But looks like the Pci_set_master is not getting called. what is the static void tegra_pcie_fixup_bridge(struct pci_dev *dev) been added?

static void pci_enable_bridge(struct pci_dev *dev)
{
	struct pci_dev *bridge;
	int retval;

	bridge = pci_upstream_bridge(dev);
	if (bridge)
		pci_enable_bridge(bridge);

	if (pci_is_enabled(dev)) {
		if (!dev->is_busmaster)
			pci_set_master(dev);
		return;
	}

	retval = pci_enable_device(dev);
	if (retval)
		dev_err(&dev->dev, "Error enabling bridge (%d), continuing\n",
			retval);
	pci_set_master(dev);
}

Manikanta · May 26, 2019, 4:52am

Hi,

It is old fixup code, probably added when PCI subsystem is not enabling bus master for host bridges. However it is not required now because subsystem is taking care of it.

Manikanta

Phoenixlee · February 3, 2021, 3:10pm

There is one special scenario: 2 partitions are created in the switch, each partition serves a traditional (Virtual) PCIE switch.
Additionally, both upstream ports of the 2 partitions are also binded to NT/DMA function(s). His intention is to let TWO TX2 connected to the 2 upstream ports of the 2 partition communicate to each other via DMA and NT funcs. But, i guess, no valid EP(s) are attatched to the downstream ports, for at least one partition. Thus, in that specific partition, Tegra RC will not enable mem access, since it cannot see valid endpoints through the standard upstream port, resulting in its decision to disable this pcie-sub-tree (no mem, no io, etc … all minus) .

Phoenixlee · February 3, 2021, 3:34pm

Hello, i am working around 89HPES24NT6AG2.
and:

i have managed to get two TX2 talk to each other via their respective NT func. i.,e.,
one TX2 writes (use copy_to_user to issue DW32 writes towards the kernel virtual address of BAR2) to BAR2 (outbound window), and another TX2 is able to get the expected data in its inbound buffer
dma function driver is also implemented. but there is something strange about DMA
2.1: when i tried to use dma to move data in a memory-to-memory fashion (both src and dst are in the memory of the same TX2), i can only see part of data copied correctly for the initial tests. when i repeated the same test, for most of time, i can’t see any data copied correctly from src to dst (src: 0, 1, 2, 3, … and dst: 0, 0, 0, 0, …)
2.2: when i tried to use dma to move data in a memory-to-memory fashion (src in the memory of TX2 and dst in BAR2, which is also the outbound window), i have not observed that data can be copied over to its NT peer correctly, not even partially, for even once.

Putting a and b together, i suspect there is cache coherence issue. However, the inbound buffers of NT peer are allocated with dma_alloc_coherent hence no cache issue should exist !

Have u ever encountered problems like this ?

thank you.