Doca_gpu_packet_processing Failed to start mmap DOCA Driver call failure

I am attempting to run the doca_gpu_packet_processing sample application and getting an error. My configuration is described by option 1 GPUNetIO Programming Guide :: NVIDIA DOCA SDK Documentation

I have a ConnectX-6 Dx NIC and RTX A4500 GPU. I am getting an error that says: Failed to start mmap DOCA Driver call failure. Any ideas?

Here is the output I get:
sudo -E ./doca_gpu_packet_processing -n 51:00.0 -g 52:00.0 -q 2
[17:59:11:497742][DOCA][INF][GPU_PACKET_PROCESSING:215]: ===========================================================
[17:59:11:497764][DOCA][INF][GPU_PACKET_PROCESSING:216]: DOCA version: 2.2.0080
[17:59:11:497767][DOCA][INF][GPU_PACKET_PROCESSING:217]: ===========================================================
[17:59:11:497781][DOCA][INF][GPU_PACKET_PROCESSING:238]: Options enabled:
GPU 52:00.0
NIC 51:00.0
GPU Rx queues 2
GPU HTTP server enabled No
EAL: Detected CPU lcores: 20
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode ‘PA’
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
EAL: Probe PCI driver: mlx5_pci (15b3:101d) device: 0000:51:00.0 (socket 0)
EAL: Probe PCI driver: gpu_cuda (10de:2232) device: 0000:52:00.0 (socket 0)
[17:59:11:723164][DOCA][WRN][engine_model:72]: adapting queue depth to 128.
[17:59:11:723182][DOCA][INF][engine_model:124]: engine model defined with mode=vnf
[17:59:11:723186][DOCA][INF][engine_model:126]: engine model defined with nr_pipe_queues=2
[17:59:11:723189][DOCA][INF][engine_model:127]: engine model defined with pipe_queue_depth=128
[17:59:11:723191][DOCA][INF][engine_model:130]: engine model defined in isolated mode
[17:59:11:727413][DOCA][INF][engine_field_mapping:329]: Engine field mapping initialized
[17:59:11:727422][DOCA][INF][engine_shared_resources:94]: Engine shared resources initialized successfully
[17:59:11:727427][DOCA][INF][dpdk_engine:288]: ACL number of collisions is zero, set it to default 3
[17:59:11:727437][DOCA][INF][dpdk_port:545]: dpdk port module init
[17:59:11:727443][DOCA][INF][dpdk_table:72]: Initializing dpdk table successfully
[17:59:11:727447][DOCA][INF][dpdk_flow:68]: Initializing dpdk flow successfully
[17:59:11:727450][DOCA][INF][dpdk_resource_manager:184]: Dpdk resource manager register completed
[17:59:11:727635][DOCA][INF][dpdk_pipe_items:186]: Initialized dpdk pipe items module
[17:59:11:727638][DOCA][INF][dpdk_pipe:142]: Dpdk pipe initialized successfully
[17:59:11:727642][DOCA][INF][dpdk_layer:446]: Dpdk layer register completed
[17:59:11:727694][DOCA][INF][doca_flow_match:563]: Doca flow match UDS initialized
[17:59:11:727755][DOCA][INF][doca_flow_actions:746]: Doca flow actions UDS initialized
[17:59:11:727758][DOCA][INF][doca_flow_layer:65]: Doca flow layer initialized
[17:59:11:727762][DOCA][INF][doca_flow:614]: Doca flow initialized successfully
[17:59:11:727779][DOCA][INF][utils_hash_table:123]: hash table a_tmplt_t port 0 created
[17:59:11:727785][DOCA][INF][utils_hash_table:123]: hash table p_tmplt_t port 0 created
[17:59:11:727814][DOCA][INF][utils_hash_table:123]: hash table dpdk_tbl_mgr port 0 created
[17:59:11:727826][DOCA][INF][utils_hash_table:123]: hash table grp_fwd port 0 created
[17:59:12:119509][DOCA][INF][dpdk_port:721]: Dpdk port 0 initialized successfully with 3 queues
[17:59:12:119525][DOCA][WRN][engine_port:246]: detected representor used in VNF mode (driver port id 0)
[17:59:12:146542][DOCA][INF][doca_flow:1454]: doca flow port with id=0, type=0 started
[17:59:12:146550][DOCA][INF][GPU_PACKET_PROCESSING_UDP:40]: Creating UDP Eth Rxq 0
[17:59:12:147644][DOCA][ERR][GPU_PACKET_PROCESSING_UDP:106]: Failed to start mmap DOCA Driver call failure
[17:59:12:147656][DOCA][INF][GPU_PACKET_PROCESSING_UDP:228]: Destroying UDP queue 0
[17:59:12:147659][DOCA][ERR][GPU_PACKET_PROCESSING_UDP:246]: Failed doca_ctx_stop: Invalid input
[17:59:12:147665][DOCA][ERR][GPU_PACKET_PROCESSING:264]: Function create_udp_queues returned Bad State

What is the BAR1 size of your GPU? I’ve seen this error in the past with small BAR1 sizes like 256MiB.

To check:

$ nvidia-smi -q | grep -i bar -A 3
BAR1 Memory Usage
Total : 131072 MiB
Used : 1 MiB
Free : 131071 MiB

Here is the output:

nvidia-smi -q | grep -i bar -A 3
BAR1 Memory Usage
Total : 256 MiB
Used : 6 MiB
Free : 250 MiB

Does this mean I con not use DOCA with this GPU?

No, it means you need to reconfigure the BAR1 size of the GPU in your system.
Typically this is done in the BIOS settings. Maybe in your BIOS you have an option named ReBAR or Resizeable BAR, look for it and be sure it’s enabled (if not enable the option).
Few useful references:

I have an update on this. I went in the BIOS and enable the resizable BAR option. Now I get the following:

nvidia-smi -q | grep -i bar -A 3
BAR1 Memory Usage
Total : 32768 MiB
Used : 6 MiB
Free : 32762 MiB

I ran the doca_gpu_packet_processing app again and it worked! Thanks for pointing me in the right direction. Maybe you should add that the resizable BAR BIOS setting to the documentation?

Great! You’re right, I will add this info to the documentation

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.