xavier pci_alloc_consistent & dma_alloc_coherent issues

WindInTheWillows · January 11, 2020, 3:54pm

Now I’m working on xavier and fpga platform, fpga connect to xavier PCIe slot, with sg dma to transform data to xavier. The driver allocates a buffer for SG DMA descriptors which it then passes to the device. I need to pass physical address of first descriptor to sgdma register, but I’ve tried pci_alloc_consistent & dma_alloc_coherent, the bus_addr returned by pci_alloc_consistent & dma_alloc_coherent seemed couldn’t work. I’ve checked in the forum, found maybe this was about smmu. If I disable smmu, will bus_addr return by pci_alloc_consistent & dma_alloc_coherent are equal to physical addr? But with smmu enabled, can I obtain physical addr of buffer which come from pci_alloc_consistent & dma_alloc_coherent?
Any sugestion, please!

WayneWWW · January 14, 2020, 9:47am

Hi,

We need more details about the failure. Can you share kernel log with failure return value?

In case if there is some issue is due to smmu. You can try below change. Comment out highlighted code from dt file to disable smmu.

If EP supports 64 bit dma, there should not be any issue with smmu disabled.

Please remove below one.

File: kernel-dts/tegra194-soc/tegra194-soc-pcie.dtsi
Node:

pcie@141a0000 {
                iommus = <&smmu TEGRA_SID_PCIE5>;# remove 
                dma-coherent; # remove 
#if LINUX_VERSION >= 414
                iommu-map = <0x0 &smmu TEGRA_SID_PCIE5 0x1000>;# remove 
                iommu-map-mask = <0x0>; # remove 
#endif
}

WindInTheWillows · January 15, 2020, 9:55am

Thanks WayneWWW
I’m working on xavier, a FPGA board plug in xavier PCIe slot. FPGA transfer data to xavier through PCIe in SG DMA mode. Now I’m writing linux driver for FPGA board(PCIe device). The driver will do:

kmalloc 4096 bytes memory for descriptor:
descriptor_vir = kmalloc(4096, GFP_KERNEL);
kmalloc 0xFFFF0 memory to receive FPGA data:
unsigned int *receive_vir = kmalloc(0xFFFF0, GFP_KERNEL);
convert receive_vir to physical address:
phys_addr_t receive_phy = virt_to_phys(receive_vir);
Put receive_phy into descriptor_vir, then obtain descriptor’s physical address:
phys_addr_t descriptor_phy = virt_to_phys(descriptor_vir);
write descriptor_phy to FPGA register , then start SG DMA transfer.
If every step work fine, xavier will receive data like: 00 01 02 03 04 05 06…,the data will store in receive_vir.

with smmu enabled, the procedure returned nothing. I’m sure the SG DMA didn’t work. I searched in the forum, found some similar topics are releated to smmu, so I decided to disbale smmu:

I’ve removed all
iommus = <&smmu TEGRA_SID_PCIE0>;
dma-coherent;

        iommus = <&smmu TEGRA_SID_PCIE1>;
        dma-coherent;
        ...
        iommus = <&smmu TEGRA_SID_PCIE5>;
        dma-coherent;

in tegra194-soc-pcie.dtsi, and flash the modified dtb to xavier, in /sys/kernel/debug/12000000.iommu/masters, there is no pcie items:
root@nvidia:/sys/kernel/debug/12000000.iommu/masters# ls
13e10000.host1x 13e10000.host1x:ctx7 15380000.nvjpg 15880000.nvdla0 2600000.dma 31b0000.i2c aconnect@2a41000:adsp_audio sound
13e10000.host1x:ctx0 14800000.isp 15480000.nvdec 158c0000.nvdla1 2993000.adsp 31c0000.i2c bc00000.rtcpu
13e10000.host1x:ctx1 15100000.tsecb 154c0000.nvenc 15a80000.nvenc1 3100000.serial 31e0000.i2c c1a0000.aon
13e10000.host1x:ctx2 15140000.nvdec1 15500000.tsec 15c10000.vi 3110000.serial 3400000.sdhci c240000.i2c
13e10000.host1x:ctx3 15200000.nvdisplay 15810000.se 16000000.pva0 3140000.serial 3460000.sdhci c250000.i2c
13e10000.host1x:ctx4 15210000.nvdisplay 15820000.se 16800000.pva1 3160000.i2c 3510000.hda c260000.spi
13e10000.host1x:ctx5 15220000.nvdisplay 15830000.se 2450000.ufshci 3180000.i2c 3550000.xudc d000000.bpmp
13e10000.host1x:ctx6 15340000.vic 15840000.se 2490000.ether_qos 3190000.i2c 3610000.xhci smmu_test

But, SG DMA still can’t work. And kmalloc and virt_to_phys returned as below:

[ 84.005296] descriptor_vir ffffffc3993d0000
[ 84.005299] descriptor_phy 4193d0000
[ 84.005325]
[ 84.005329]
[ 84.005340]
[ 84.005346] receive_vir ffffffc395f00000
[ 84.005626] receive_phy 415f00000
Are those physical addresses normal?

Now I’m modifying dts as your advice, only remove
pcie@141a0000 {
iommus = <&smmu TEGRA_SID_PCIE5>;# remove
dma-coherent; # remove
#if LINUX_VERSION >= 414
iommu-map = <0x0 &smmu TEGRA_SID_PCIE5 0x1000>;# remove
iommu-map-mask = <0x0>; # remove
#endif
}

omp · February 3, 2020, 5:17am

can I obtain physical addr of buffer which come from pci_alloc_consistent & dma_alloc_coherent?
Yes you can get physical address by using virt_to_phys call on virtual address return by dma_alloc_coherent. if smmu is disabled virt_to_phys(virtual_address) should be same as dma_addr_t value return from dma_alloc_coherent.

Are those physical addresses normal?
Yes address is withing DRAM addressable range. So it looks fine.

Can you share you code of DMA transfer that you are referring in your previous comment.
also can please share output of “lspci -vv”

Thanks,
Om

WindInTheWillows · February 3, 2020, 4:21pm

Hi omp
As you mentioned:
1.code of DMA transfer:
I tried kmalloc & dma_alloc_coherent, below is dma_alloc_coherent:
descriptor_vir = dma_alloc_coherent(&gDev->dev, 4096, &descriptor_phy, GFP_KERNEL);
printk(“descriptor_vir %lx, descriptor_phy %lx\n”, descriptor_vir, descriptor_phy);
memset(descriptor_vir, 0, 4096);

phys_addr_t dptr_phy = virt_to_phys(descriptor_vir);
printk(“dptr_phy %lx\n”, dptr_phy);

pcie_vir = dma_alloc_coherent(&gDev->dev, 1048560, &pcie_phy, GFP_KERNEL);
printk(“pcie_vir %lx, pcie_phy %lx\n”, pcie_vir, pcie_phy);
memset(pcie_vir, 0, 1048560);

phys_addr_t p_phy = virt_to_phys(pcie_vir);
printk(“p_phy %lx\n”, p_phy);
…
*descriptor_vir = 0x800FFFF0;
printk(“control %x\n”, *descriptor_vir);

*(descriptor_vir + 0x04) = 0;
printk(“FPGA addr %x\n”, *(descriptor_vir + 0x04));

*(descriptor_vir + 0x08) = (PCI_DMA_L(p_phy));
printk(“PCIe addr Lower %x\n”, *(descriptor_vir + 0x08));

*(descriptor_vir + 0x0c) = (PCI_DMA_H(p_phy));
printk(“PCIe addr Upper %x\n”, *(descriptor_vir + 0x0c));

/***********************************************************************/
“if smmu is disabled virt_to_phys(virtual_address) should be same as dma_addr_t value return from dma_alloc_coherent.”
Now I have disabled smmu, but if I execute those code, the two adresses below are not same:
dptr_phy != descriptor_phy, it’s really confused me!

“lspci -vv”
0005:01:00.0 Unassigned class [ff80]: Xilinx Corporation Device 7024
Subsystem: Xilinx Corporation Device 0007
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 255
Region 0: [virtual] Memory at 1f40000000 (32-bit, non-prefetchable)
Region 1: [virtual] Memory at 1f40001000 (32-bit, non-prefetchable)
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Device Serial Number 00-00-00-01-01-00-0a-35

Any help, thanks a lot!

omp · February 4, 2020, 4:10am

Looks like mismatch of MaxPayload value with RP is causing this issue. Xavier PCIe RC is capable of 256 Bytes as MaxPayload. Please confirm in your setup for RP device “lspci -s 0005:00:00.0 -vv | grep MaxPayload”

“lspci -vv”
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
…
MaxPayload 128 bytes, MaxReadReq 512 bytes

Can you try setting MaxPayload from EP driver

pcie_set_mps(pdev, 256);

Thanks,
Om

WindInTheWillows · February 8, 2020, 3:51pm

Hi omp
I tried

pcie_set_mps(pdev, 256);

still couldn’t work.

I’ve tried dma_alloc_coherent as below:
/********************************************************/
descriptor_vir = dma_alloc_coherent(&gDev->dev, 4096, &descriptor_phy, GFP_KERNEL);
printk(“descriptor_vir %lx, descriptor_phy %lx\n”, descriptor_vir, descriptor_phy);
memset(descriptor_vir, 0, 4096);

phys_addr_t dptr_phy = virt_to_phys(descriptor_vir);
printk(“dptr_phy %lx\n”, dptr_phy);

unsigned int *dptr_phy_ = dma_map_single(&gDev->dev, descriptor_vir, 4096, DMA_BIDIRECTIONAL);
printk(“dptr_phy_ %lx\n”, dptr_phy_);
/********************************************************/

when I insmod the driver, it printed :
descriptor_vir ffffff8008020000, descriptor_phy 91020000
dptr_phy 80020000
dptr_phy_ 88020000

The smmu is disabled. I’m totally confused, any help please

Manikanta · March 16, 2020, 5:24am

Hi,

virt_to_phys() should not be used on dma memory, please refer to the comment in file arch/arm64/include/asm/memory.h.

Please try pcie_phy in place of p_phy as shown below.

*(descriptor_vir + 0x08) = (PCI_DMA_L(pcie_phy));
printk(“PCIe addr Lower %x\n”, *(descriptor_vir + 0x08));

*(descriptor_vir + 0x0c) = (PCI_DMA_H(pcie_phy));
printk(“PCIe addr Upper %x\n”, *(descriptor_vir + 0x0c));

Also, you have to use descriptor_phy instead of dptr_phy when informing FPGA device about descriptor location.

Thanks,
Manikanta

Topic		Replies	Views
PCIE DMA Problem between TX2 & FPGA Jetson TX2	19	7502	October 18, 2021
NX shared memory issue - any insights? Jetson Xavier NX	6	1566	October 18, 2021
PCIe DMA on Tegra (Xavier NX) Jetson AGX Xavier kernel	24	1883	July 13, 2022
Slow remote DMA write and read Jetson TX2	26	2735	June 29, 2019
What is the current status of PCIe DMA? Jetson TX1	21	7971	October 18, 2021
Altera FPGA DMA to TX2 via PCIe problem Jetson TX2	18	3488	October 18, 2021
pcie smmu issue Jetson AGX Xavier	7	5440	October 18, 2021
How to Carry out DMA transfer when sending data using PCIe from NVIDIA Root Port to a custom end point Jetson AGX Xavier pcie	16	3120	August 3, 2023
How to disable SMMU on TX2 platform?? Jetson TX2	10	2335	October 18, 2021
DMA Controller on the IDT switch unable to access Tegra tx2 system memory address Jetson TX2	17	1093	October 18, 2021

xavier pci_alloc_consistent & dma_alloc_coherent issues

Related topics