DPDK mlx5 without root permissions

Hi,

I am trying to use DPDK on a Connectx-5 using the mlx5 driver without root permissions.

I followed the documentation on how to use DPDK without root permissions, but the guide information only concerns the VFIO driver.

Following this guide only, I get the error “Failed to create TIS using DevX”, for example with testpmd:


$ ./dpdk-testpmd -a 8b:00.0 -l 0-7 -- --forward-mode=txonly

EAL: Detected CPU lcores: 80

EAL: Detected NUMA nodes: 2

EAL: Detected static linkage of DPDK

EAL: Multi-process socket /run/user/1003/dpdk/rte/mp_socket

EAL: Selected IOVA mode 'VA'

EAL: No free 1048576 kB hugepages reported on node 0

EAL: No free 1048576 kB hugepages reported on node 1

EAL: No available 1048576 kB hugepages reported

EAL: VFIO support initialized

EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:8b:00.0 (socket 1)

mlx5_common: Failed to create TIS using DevX

mlx5_net: Failed to TIS 0 for bonding device mlx5_3.

mlx5_net: TIS allocation failure

mlx5_net: probe of PCI device 0000:8b:00.0 aborted after encountering an error: Cannot allocate memory

mlx5_common: Failed to load driver mlx5_eth

EAL: Requested device 0000:8b:00.0 cannot be used

EAL: Bus (pci) probe failed.

By launching the same command as root, no error occurs.

I already tried tuning the permissions of the mst devices, without success:


$ ll /dev/mst

crw-rw---- 1 root rdma 507, 0 janv. 19 11:59 mt4119_pciconf0

crw-rw---- 1 root rdma 507, 1 janv. 19 11:59 mt4119_pciconf1

crw-rw---- 1 root rdma 507, 2 janv. 19 11:59 mt4119_pciconf2

crw-rw---- 1 root rdma 507, 3 janv. 19 11:59 mt4119_pciconf3

(I am in group rdma)

I suppose that some other permissions have to be tuned with the mlx5 driver, but I can’t figure which ones.

Q1: Is it actually possible to use the mlx5 driver without root permissions ?

Q2: If yes, what is the procedure to use mlx5 driver backend for DPDK without root permissions ?

Thanks for any help,

Julien

Hi Julien,

You can try running the test with debug level, by adding ‘–log-level=eal,8’, and look for additional useful prints.

I.E:

./dpdk-testpmd --log-level=eal,8 -a 81:00.0 -l 0-7 – --forward-mode=txonly

In addition, for testing purposes you can try disabling DV flow and use the Verbs flow engine instead, by adding devargs of dv_flow_en=0. Do you see different results?

I.E:

./dpdk-testpmd --log-level=eal,8 -a 81:00.0,dv_flow_en=0 -l 0-7 – --forward-mode=txonly

Best Regards,

Chen

Hi Chen,

Thanks for your answer.

I attached to this message the logs with maximum verbosity for the EAL, both with and without root permissions. The error message is essentially the same (lines 219-223 of the non-root log).

I noticed a few differences between the two logs, especially regarding the NUMA nodes, but I am not sure they have anything to do with my problem.

By disabling DV flow, I obtained the exact same result.

By gathering information about your dv_flow_en parameter, I came across the CONFIG_RTE_LIBRTE_MLX5_DEBUG compilation option, that I did not know about. I’ll try and see if this brings me more information…

Please let me know if you have any other idea,

Best regards,

Julien

Hi again,

I just figured that it is possible to increase verbosity of mlx5-related messages. Please find attached the logs produced this way, as a standard user. I also tried disabling DevX (UCTX_EN=0), which gives a different error, but still not much information.

I’ll let you know of any progress on my side.

Best regards,

Julien

Just as an update, I managed to run testpmd without root permissions by tuning file capabilities, which is a viable option for me.

For now, I simply use sudo setcap all=eip dpdk-testpmd, but I’ll try and narrow down the exact capabilities required.

So it looks like the only file capability needed is cap_net_raw, which makes sense.

Everything seem to run fine once I run sudo setcap cap_net_raw=eip dpdk-testpmd before launching testpmd.

EDIT: To avoid tweaking file permissions on hugepages, I now set the cap_dac_override capability at the same time, with sudo setcap cap_net_raw,cap_dac_override=eip dpdk-testpmd