How does the mlx5 driver use the IOVA generated by IOMMU as DMA address?

Infrastructure & Networking Adapters and Cables InfiniBand/VPI Adapter Cards
, , ,
1

I have enabled IOMMU on the physical machine(iommu=pt intel_iommu=on). I expect the RDMA NIC to use the IOVA allocated by the IOMMU module for DMA after enabling IOMMU. However, in reality, the RDMA NIC does not use the IOVA for DMA：

image
image1573×705 75.9 KB

I found through reading the kernel source code that ib_dma_map_sgtable_attrs() is called in ib_umem_get to obtain the DMA address for each scatterlist (sg) entry. During the process of obtaining the DMA address, it first checks whether dev->ops is empty, and then decides whether to assign the DMA address to IOVA. Currently, it was debugged that dev->ops is empty at this point, and the DMA address is not assigned to IOVA.

Does the mlx5 driver perform any other actions (such as assigning dev->ops) when obtaining the struct device *dev to be able to use IOMMU?

2

Hi Kuao,

Thank you for posting your query on NVIDIA Forum.

Based on internal check, this requires a deeper dig into the driver code and unfortunately, we cannot perform debug over the forum.

As this is programming related, a support ticket with valid support contract in place will be required for further debug. Support ticket can be opened by emailing Networking-support@nvidia.com

For details on contracts, please feel free to contact our contracts team at Networking-contracts@nvidia.com

Thanks,
Namrata.