Pcie vnet

yougang.hu · March 14, 2023, 8:02am

hi nv
I have a question about nvdia/drivers/pci/endpoint/functions/pci-epf-tegra-vnet.c
function:
static netdev_tx_t tvnet_ep_start_xmit(struct sk_buff *skb,
struct net_device *ndev)
…

	/* Trigger DMA write from src_iova to dst_iova */
	desc_widx = desc_cnt->wr_cnt % DMA_DESC_COUNT;
	ep_dma_virt[desc_widx].size = len;
	ep_dma_virt[desc_widx].sar_low = lower_32_bits(src_iova);
	ep_dma_virt[desc_widx].sar_high = upper_32_bits(src_iova);
	ep_dma_virt[desc_widx].dar_low = lower_32_bits(dst_iova);
	ep_dma_virt[desc_widx].dar_high = upper_32_bits(dst_iova);
//Dose this mean writing address to the DMA Address Registers?

Jasonm · March 15, 2023, 3:11am

Hi yougang,
Correct!
You can check the tvnet_ep_pci_epf_bind() for the the BAR0 mapping and related initial.

File:
nvidia/drivers/pci/endpoint/functions/pci-epf-tegra-vnet.c

Function:
static int tvnet_ep_pci_epf_bind(struct pci_epf *epf)

 /* BAR0 EP memory allocation */
 amap = &tvnet->bar0_amap[EP_MEM];
 amap->iova = tvnet->bar0_amap[SIMPLE_IRQ].iova +
         tvnet->bar0_amap[SIMPLE_IRQ].size;
 size = sizeof(struct ep_own_cnt) + (RING_COUNT *
         (sizeof(struct ctrl_msg) + 2 * sizeof(struct data_msg)));
 amap->size = PAGE_ALIGN(size);
 ret = tvnet_ep_alloc_multi_page_bar0_mem(epf, EP_MEM);


 /* BAR0 host memory allocation */
 amap = &tvnet->bar0_amap[HOST_MEM];
 amap->iova = tvnet->bar0_amap[EP_MEM].iova +
                                 tvnet->bar0_amap[EP_MEM].size;


 /* Set link list pointer to create a dma desc ring */
 memset(amap->virt, 0, amap->size);
 dma_desc = (struct tvnet_dma_desc *)amap->virt;
 dma_desc[DMA_DESC_COUNT].sar_low = (amap->iova & 0xffffffff);
 dma_desc[DMA_DESC_COUNT].sar_high = ((amap->iova >> 32) & 0xffffffff);
 dma_desc[DMA_DESC_COUNT].ctrl_reg.ctrl_e.llp = 1;

yougang.hu · March 15, 2023, 6:47am

hi nv
thank you for reply
I read this part of the code, I understand that only allocate memory
How to map to DMA registers?
Function:
static netdev_tx_t tvnet_ep_start_xmit
#if ENABLE_DMA
/* Trigger DMA write from src_iova to dst_iova */
desc_widx = desc_cnt->wr_cnt % DMA_DESC_COUNT;
ep_dma_virt[desc_widx].size = len;
ep_dma_virt[desc_widx].sar_low = lower_32_bits(src_iova);
ep_dma_virt[desc_widx].sar_high = upper_32_bits(src_iova);
ep_dma_virt[desc_widx].dar_low = lower_32_bits(dst_iova);
ep_dma_virt[desc_widx].dar_high = upper_32_bits(dst_iova);

	printk("%s   sar_low    0x%x  0x%x dar_low  0x%x  0x%x\n",__func__,
		dma_channel_rd(tvnet->dma_base, DMA_WR_DATA_CH,DMA_SAR_LOW_OFF_WRCH),
		ep_dma_virt[desc_widx].sar_low,
	dma_channel_rd(tvnet->dma_base, DMA_WR_DATA_CH,DMA_DAR_LOW_OFF_WRCH),
	ep_dma_virt[desc_widx].dar_low);
dma_common_wr8(tvnet->dma_base, DMA_WR_DATA_CH, DMA_WRITE_DOORBELL_OFF);

// I print DMA Address Regsters value, ep_dma_virt[desc_widx].sar_low not equal dma_channel_rd(tvnet->dma_base, DMA_WR_DATA_CH,DMA_SAR_LOW_OFF_WRCH)

Jasonm · March 15, 2023, 7:23am

OK, Seems the question is where to set the DMA registers?

dts file:
nvidia/soc/t23x/kernel-dts/tegra234-soc/tegra234-soc-pcie.dtsi
node and reg:
pcie_c5_ep: pcie_ep@141a0000 {
compatible = “nvidia,tegra234-pcie-ep”, “snps,dw-pcie”;
power-domains = <&bpmp TEGRA234_POWER_DOMAIN_PCIEX8A>;
reg = <0x00 0x141a0000 0x0 0x00020000 /* appl registers (128K) /
0x00 0x3a040000 0x0 0x00040000 / iATU_DMA reg space (256K) /
0x00 0x3a080000 0x0 0x00040000 / DBI space (256K) /
0x27 0x40000000 0x4 0x00000000>; / Address Space (16G) */
reg-names = “appl”, “atu_dma”, “dbi”, “addr_space”;
driver code parse and set DMA resource:
res = platform_get_resource_byname(pdev, IORESOURCE_MEM, “atu_dma”);
if (!res) {
dev_err(fdev, “missing atu_dma resource in DT\n”);
ret = PTR_ERR(res);
goto fail;
}
```
 tvnet->dma_base = devm_ioremap(fdev, res->start + DMA_OFFSET,
                                resource_size(res) - DMA_OFFSET);
```

trigger the DMA:
static void tvnet_ep_setup_dma(struct pci_epf_tvnet *tvnet)
{
dma_addr_t iova = tvnet->bar0_amap[HOST_DMA].iova;
struct dma_desc_cnt *desc_cnt = &tvnet->desc_cnt;
u32 val;

desc_cnt->rd_cnt = desc_cnt->wr_cnt = 0;

/* Enable linked list mode and set CCS for write channel-0 */
val = dma_channel_rd(tvnet->dma_base, DMA_WR_DATA_CH,
                     DMA_CH_CONTROL1_OFF_WRCH);
val |= DMA_CH_CONTROL1_OFF_WRCH_LLE;
val |= DMA_CH_CONTROL1_OFF_WRCH_CCS;
dma_channel_wr(tvnet->dma_base, DMA_WR_DATA_CH, val,
               DMA_CH_CONTROL1_OFF_WRCH);

/* Unmask write channel-0 done irq to enable LIE */
val = dma_common_rd(tvnet->dma_base, DMA_WRITE_INT_MASK_OFF);
val &= ~0x1;
dma_common_wr(tvnet->dma_base, val, DMA_WRITE_INT_MASK_OFF);

/* Enable write channel-0 local abort irq */
val = dma_common_rd(tvnet->dma_base, DMA_WRITE_LINKED_LIST_ERR_EN_OFF);
val |= (0x1 << 16);
dma_common_wr(tvnet->dma_base, val, DMA_WRITE_LINKED_LIST_ERR_EN_OFF);

/* Program DMA write linked list base address to DMA LLP register */
dma_channel_wr(tvnet->dma_base, DMA_WR_DATA_CH,
               lower_32_bits(tvnet->ep_dma_iova),
               DMA_LLP_LOW_OFF_WRCH);
dma_channel_wr(tvnet->dma_base, DMA_WR_DATA_CH,
               upper_32_bits(tvnet->ep_dma_iova),
               DMA_LLP_HIGH_OFF_WRCH);

/* Enable DMA write engine */
dma_common_wr(tvnet->dma_base, DMA_WRITE_ENGINE_EN_OFF_ENABLE,
                DMA_WRITE_ENGINE_EN_OFF);

/* Enable linked list mode and set CCS for read channel-0 */
val = dma_channel_rd(tvnet->dma_base, DMA_RD_DATA_CH,
                     DMA_CH_CONTROL1_OFF_RDCH);
val |= DMA_CH_CONTROL1_OFF_RDCH_LLE;
val |= DMA_CH_CONTROL1_OFF_RDCH_CCS;
dma_channel_wr(tvnet->dma_base, DMA_RD_DATA_CH, val,
               DMA_CH_CONTROL1_OFF_RDCH);

/* Mask read channel-0 done irq to enable RIE */
val = dma_common_rd(tvnet->dma_base, DMA_READ_INT_MASK_OFF);
val |= 0x1;
dma_common_wr(tvnet->dma_base, val, DMA_READ_INT_MASK_OFF);

val = dma_common_rd(tvnet->dma_base, DMA_READ_LINKED_LIST_ERR_EN_OFF);
/* Enable read channel-0 remote abort irq */
val |= 0x1;
dma_common_wr(tvnet->dma_base, val, DMA_READ_LINKED_LIST_ERR_EN_OFF);

/* Program DMA read linked list base address to DMA LLP register */
dma_channel_wr(tvnet->dma_base, DMA_RD_DATA_CH,
               lower_32_bits(iova), DMA_LLP_LOW_OFF_RDCH);
dma_channel_wr(tvnet->dma_base, DMA_RD_DATA_CH,
               upper_32_bits(iova), DMA_LLP_HIGH_OFF_RDCH);

/* Enable DMA read engine */
dma_common_wr(tvnet->dma_base, DMA_READ_ENGINE_EN_OFF_ENABLE,
                DMA_READ_ENGINE_EN_OFF);

}

yougang.hu · March 15, 2023, 7:34am

hi nv
thank you for reply
I don’t understand why sar&dst address is not equal to the value read from DMA Registers

Jasonm · March 15, 2023, 7:44am

Hello, yougang,
Would you mind share more details about your question?
“I don’t understand why sar&dst address is not equal to the value read from DMA Registers”

What’s your test steps (command or test code much better)
The output value and the “sar&dst address” you have set

BR.,
Jason

yougang.hu · March 15, 2023, 7:51am

[ 94.342000] tvnet_ep_start_xmit sar_low 0x0 0xfebdf002 dar_low 0x0 0xffff0040
[ 94.413543] tvnet_ep_start_xmit sar_low 0xfebdf038 0xfebde402 dar_low 0xffff0076 0xfffe0040
[ 94.664594] tvnet_ep_start_xmit sar_low 0xfebde48c 0xfebde402 dar_low 0xfffe00ca 0xfffd0040
This is the print log
ep_dma_virt[desc_widx].sar_low value not equal dma_channel_rd(tvnet->dma_base, DMA_WR_DATA_CH,DMA_SAR_LOW_OFF_WRCH) value
ep_dma_virt[desc_widx].dar_low value not equal dma_channel_rd(tvnet->dma_base, DMA_WR_DATA_CH,DMA_DAR_LOW_OFF_WRCH) value

yougang.hu · March 15, 2023, 8:07am

Hi nv
I test follow below link
https://docs.nvidia.com/jetson/archives/r35.2.1/DeveloperGuide/text/SD/Communications/PcieEndpointMode.html?highlight=vnet#bringing-up-an-ethernet-interface-over-pcie
I want know how to use PCI DMA from vnet code

Jasonm · March 15, 2023, 8:09am

Can you show your code changes?
The print debug code you have added.

yougang.hu · March 15, 2023, 8:23am

pci-epf-tegra-vnet.c (49.7 KB)
Can I describe this problem in Chinese？tks

yougang.hu · March 16, 2023, 2:28am

hi nv
I disabled linked list mode,and use DMA SAR&DAR Registers to set address
DMA work is not fine
pci-epf-tegra-vnet.c (50.2 KB)

yougang.hu · March 16, 2023, 3:30am

hi nv
Function:
static netdev_tx_t tvnet_ep_start_xmit(struct sk_buff *skb,
struct net_device *ndev)

	ep_dma_virt[desc_widx].dar_low = lower_32_bits(dst_iova);
	ep_dma_virt[desc_widx].dar_high = upper_32_bits(dst_iova);
//dst_iova value geted from RP,Does this mean that DMA can directly transfer data to another orin?

Jasonm · March 16, 2023, 3:49am

Regarding: "dst_iova value geted from RP,Does this mean that DMA can directly transfer data to another orin? "
Yes, more specifically, int PCIe vnet scenario , uses the EP DMA

yougang.hu · March 16, 2023, 5:40am

hi nv
thank you for reply
I modify pci-epf-nv-test.c, use DMA send data to RC
dst address also taken from RC,but the system will crash in EP
why? can you give me some suggestions?

Jasonm · March 16, 2023, 8:06am

Does this issue occur in native driver?
Can you make sure your code follows our native code?

yougang.hu · March 17, 2023, 2:23am

hi nv
thank you for reply
native code is ok
how to privately send file to you?
can you help me check the code?

Topic		Replies	Views
Immediate dma_write() and dma_read() cause system crash Jetson Xavier NX pcie	8	721	August 22, 2023
The bandwidth of of virtual ethernet over PCIe between two xaviers is low Jetson AGX Xavier	92	9183	October 18, 2021
Jetson Orin PCIE-NTB function with DMA failed Jetson AGX Orin pcie	19	1879	May 7, 2024
PCIE will crash if DMA size%256 less than 4 Jetson TK1	8	1763	October 18, 2021
PCIe DMA doesn't work for L4T 24.1 Jetson TX1	36	6806	August 11, 2016
PCIE test probelem Jetson Orin NX pcie	10	692	January 22, 2025
PCIE DMA Problem between TX2 & FPGA Jetson TX2	19	7495	October 18, 2021
System crash Jetson AGX Orin pcie	17	912	April 12, 2023
When start DMA ,system restart, jetson nx Jetson Xavier NX kernel	4	1036	October 18, 2021
What is the current status of PCIe DMA? Jetson TX1	21	7970	October 18, 2021

Pcie vnet

Related topics