I have enabled IOMMU on the physical machine. I expect the RDMA NIC to use the IOVA allocated by the IOMMU module for DMA after enabling IOMMU. However, in reality, the RDMA NIC does not use the IOVA for DMA：
I found through reading the kernel source code that ib_dma_map_sgtable_attrs() is called in ib_umem_get to obtain the DMA address for each scatterlist (sg) entry. During the process of obtaining the DMA address, it first checks whether dev->ops is empty, and then decides whether to assign the DMA address to IOVA. Currently, it was debugged that dev->ops is empty at this point, and the DMA address is not assigned to IOVA.
Does the mlx5 driver perform any other actions (such as assigning dev->ops) when obtaining the struct device *dev to be able to use IOMMU?
I have added iommu=pt intel_iommu=on and I have ensured that the physical machine has enabled IOMMU. However, it seems that the RDMA driver is still not using the IOVA provided by IOMMU for DMA (just like I described above). I don’t know how to make the RDMA driver use IOVA for DMA successfully…If anyone knows, please tell me.
for using mlx5 on host there is no need to enable/use IOMMU
iommu-pt is telling iommu to create page table for pass thorough devices/when used inside VM
on host and iommu=pt, device->ops willl be NULL
on host and if you remove iommu=pt, device->ops not be NULL
