DPDK MLX5 PMD RX queue deferred start

[julien.plante2]

Hi,

I’m using DPDK with the MLX5 PMD to receive UDP packets. In order to receive UDP packets only, I use DPDK’s Flow API, which requires starting the port before configuration (rte_eth_dev_start).

The thing is that I don’t actually want to start receiving packets as soon as the port is started, but only do this because it is advised before using the Flow API.

In order to start receiving packets at the right time for my application, I want to use rte_eth_rxconf::rx_deferred_start together with rte_eth_dev_rx_queue_start. On DPDK 22.07, this does not work as expected:

...
struct rte_eth_rxconf rxconf = dev_info.default_rxconf;
rxconf.rx_deferred_start = 1;
ret = rte_eth_rx_queue_setup(port, 0, nb_rxd, rte_socket_id(), &rxconf, mpool);
...

RX queue 0 still starts on rte_eth_dev_start, and packets are received even if do not call rte_eth_dev_rx_queue_start. No commit seems to change this behaviour up to the latest DPDK version at the time of writing (23.03).
After a quick look in the code, rx_deferred_start seems to be mentionned only by the MLX5 PMD for configuration getter, not setters.

Is it intended ? (limitations in mlx5 ?)
Could it be fixed in a future version ?

Thanks,
Julien

[namrata1]

Hi Julien,

Thank you for posting your query on NVIDIA Forum.

Based on internal check, unfortunately, yes, this is a limitation and currently this is not on the roadmap.

We regret the inconvenience caused due to the same.

Thanks,
Namrata.

[julien.plante2]

Hi Namrata,

Thank you for the information. Could you at least document this as a limitation of the MLX5 PMD ? I may be wrong but I found no mention of this.

Can you share any details about the reason of this limitation ? I’m not familiar with MNLX_OFED programming, but does the limitation stems from the driver, or only that it has not been wrapped in DPDK ? I could provide an implementation in the latter situation.

Best regards,
Julien

[viacheslavo]

Hi, Julien
Deferred start is imposed to provide late queue setup, after device start.
mlx5 PMD does not support deferred start - all queues must be configured before device start.

I would recommend to consider the following scenarios:
Scenario A - using queue stop/start

• configure queues
• rte_eth_dev_start
• rte_eth_rx_queue_stop
• no traffic will be received on the stopped queues
• configure flows
• rte_eth_rx_queue_start

Scenario B - using isolated mode

• configure queues
• rte_flow_isolate
• rte_eth_dev_start
• configure flows
• traffic will appear once the specific queue/RSS is specified in Flow being applied

Scenario C - do not call rx_burst till flows configured

• start device
• configure flows
• drain Rx queues until timeout/max number of packets
• clear drop statistics if needed
• continue receiving as intended

[julien.plante2]

Hi Viecheslavo,

Thanks for all of these details, this is very informative. I didn’t know that deferred start provided late queue setup, I understood it as “do not start the queue on device start <=> stop it right at device start”

I was actually using your scenario C as a workaround currently, but scenario A looks promising ! For some reason, I thought that if deferred start didn’t work, rte_eth_rx_queue_stop would not work either.

Thanks again for the complete answer,
Julien