Hi, I’m trying to run the gpu_packet_processing example described in the DOCA GPUNetIO Install Page .
My eventual target is the system described in another forum post: DOCA gpu_packet_processing runtime error, but for now, I’m trying to run the example on a normal x86_64 tower PC with and NVIDIA RTX A4500 GPU and a ConnectX-6 Dx NIC.
My PCIe bus looks like this:
+-[0000:50]-+-00.0 Intel Corporation Device 09a2
| +-00.1 Intel Corporation Device 09a4
| +-00.2 Intel Corporation Device 09a3
| +-00.4 Intel Corporation Device 0998
| +-02.0-[51]--+-00.0 Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
| | \-00.1 Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
| \-04.0-[52]--+-00.0 NVIDIA Corporation GA102GL [RTX A4500]
| \-00.1 NVIDIA Corporation GA102 High Definition Audio Controller
Per the guide, I have the latest versions of CUDA (12.5) and the “open” flavor of the NVIDIA drivers (version 555).
I’ve also set up the ConnectX adapter in Ethernet mode, disabled ACS in BIOS, enabled resizeable BAR1, and set up hugepages.
I’ve run the app with some debug logging enabled and was hoping someone would be able to point me in the right direction. I’m mostly getting errors like:
devx adapter 0x55ffaca23c70: Failed to register user memory. Got errno=UNKNOWN-errno14 (14)
Failed to create umem with dmabuf_fd with exception:
DOCA exception [DOCA_ERROR_DRIVER] with message priv_doca_umem constructor failed
Here’s the full output:
# ./doca_gpu_packet_processing -n 51:00.0 -g 52:00.0 -q 1 -l 60 --sdk-log-level 50
[22:34:55:548040][1211][DOCA][INF][gpu_packet_processing.c:284][main] ===========================================================
[22:34:55:548095][1211][DOCA][INF][gpu_packet_processing.c:285][main] DOCA version: 2.7.0085
[22:34:55:548098][1211][DOCA][INF][gpu_packet_processing.c:286][main] ===========================================================
[22:34:55:548123][1211][DOCA][INF][gpu_packet_processing.c:307][main] Options enabled:
GPU 52:00.0
NIC 51:00.0
GPU Rx queues 1
GPU HTTP server enabled No
[22:34:55:830769][1211][DOCA][INF][doca_dev.cpp:578][doca_devinfo_create_list] Devinfo list 0x5570f5528098: Added device=0x5570f55271d0 to devinfo list
[22:34:55:830790][1211][DOCA][INF][doca_dev.cpp:578][doca_devinfo_create_list] Devinfo list 0x5570f5528098: Added device=0x5570f5528310 to devinfo list
[22:34:55:830794][1211][DOCA][INF][doca_dev.cpp:587][doca_devinfo_create_list] Devinfo list 0x5570f5528098 was created
[22:34:55:837860][1211][DOCA][INF][doca_dev.cpp:1003][doca_dev_open] Local device 0x5570f55271d0 was opened
[22:34:55:837879][1211][DOCA][INF][doca_dev.cpp:146][dev_put] Device 0x5570f5528310 was destroyed
[22:34:55:837891][1211][DOCA][INF][doca_dev.cpp:668][doca_devinfo_destroy_list] Devinfo list 0x5570f5528098 was destroyed
EAL: Detected CPU lcores: 20
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
EAL: Probe PCI driver: mlx5_pci (15b3:101d) device: 0000:51:00.0 (socket 0)
EAL: Probe PCI driver: gpu_cuda (10de:2232) device: 0000:52:00.0 (socket 0)
[22:34:56:036111][1211][DOCA][INF][doca_sub_dev.cpp:45][priv_doca_sub_dev_gpu_ops_set] sub_dev: gpu_ops was set to 0x7fcc771dfd20
[22:34:56:663305][1211][DOCA][WRN][engine_model.c:90][adapt_queue_depth] adapting queue depth to 128.
[22:34:56:663332][1211][DOCA][INF][engine_model.c:151][engine_model_init] engine model defined with mode=vnf
[22:34:56:663342][1211][DOCA][INF][engine_model.c:152][engine_model_init] engine model defined with nr_pipe_queues=1
[22:34:56:663352][1211][DOCA][INF][engine_model.c:153][engine_model_init] engine model defined with pipe_queue_depth=128
[22:34:56:663360][1211][DOCA][INF][engine_model.c:155][engine_model_init] engine model defined in isolated mode
[22:34:56:663369][1211][DOCA][INF][engine_model.c:156][engine_model_init] engine model defined RSS with nr_queues=0
[22:34:56:663377][1211][DOCA][INF][engine_model.c:157][engine_model_init] engine model defined with nr_counters=524228
[22:34:56:663386][1211][DOCA][INF][engine_model.c:158][engine_model_init] engine model defined with nr_meters=0
[22:34:56:663395][1211][DOCA][INF][engine_model.c:159][engine_model_init] engine model defined with nr_acl_collisions=3
[22:34:56:663982][1211][DOCA][INF][engine_field_mapping.c:109][engine_field_mapping_init] Engine field mapping initialized
[22:34:56:663992][1211][DOCA][INF][engine_shared_resources.c:155][engine_shared_resources_init] Engine shared resources initialized successfully
[22:34:56:664321][1211][DOCA][INF][dpdk_port.c:853][dpdk_port_module_init] dpdk port module init
[22:34:56:664331][1211][DOCA][INF][dpdk_table.c:1353][dpdk_table_module_init] Initializing dpdk table successfully
[22:34:56:664339][1211][DOCA][INF][dpdk_flow.c:59][dpdk_flow_module_init] Initializing dpdk flow successfully
[22:34:56:664349][1211][DOCA][INF][dpdk_resource_manager.c:210][dpdk_resource_manager_module_init] Dpdk resource manager register completed
[22:34:56:710311][1211][DOCA][INF][dpdk_pipe_items.c:210][dpdk_pipe_items_module_init] Initialized dpdk pipe items module
[22:34:56:710338][1211][DOCA][INF][dpdk_pipe_geneve_opt.c:125][dpdk_pipe_geneve_opt_module_init] Initialized dpdk pipe GENEVE options module
[22:34:56:710427][1211][DOCA][INF][dpdk_pipe.c:203][dpdk_pipe_module_init] Dpdk pipe initialized successfully
[22:34:56:710456][1211][DOCA][INF][dpdk_layer.c:150][dpdk_layer_register] Dpdk layer register completed
[22:34:56:710636][1211][DOCA][INF][doca_flow_match.c:694][doca_flow_match_init] Doca flow match UDS initialized
[22:34:56:710885][1211][DOCA][INF][doca_flow_actions.c:1209][doca_flow_actions_init] Doca flow actions UDS initialized
[22:34:56:710903][1211][DOCA][INF][doca_flow_monitor.c:202][doca_flow_monitor_init] Doca flow monitor UDS initialized
[22:34:56:710907][1211][DOCA][INF][doca_flow_layer.c:94][doca_flow_layer_init] Doca flow layer initialized
[22:34:56:710911][1211][DOCA][INF][doca_flow.c:617][doca_flow_init] Doca flow initialized successfully
[22:34:56:710976][1211][DOCA][INF][utils_hash_table.c:119][utils_hash_table_create] hash table a_tmplt_t port 0 created
[22:34:56:710986][1211][DOCA][INF][utils_hash_table.c:119][utils_hash_table_create] hash table p_tmplt_t port 0 created
[22:34:56:711051][1211][DOCA][INF][utils_hash_table.c:119][utils_hash_table_create] hash table dpdk_tbl_mgr port 0 created
[22:34:56:711530][1211][DOCA][INF][dpdk_meter_profiles.c:202][dpdk_meter_profiles_create] Created meter profiles on port 0 with 2 caches, 128 profiles
[22:34:57:522200][1211][DOCA][INF][dpdk_port.c:1033][dpdk_port_create] Dpdk port 0 initialized successfully with 2 queues
[22:34:57:563033][1211][DOCA][INF][doca_flow.c:1536][doca_flow_port_start] doca flow port with id=0 started
[22:34:57:563099][1211][DOCA][INF][doca_pe.cpp:46][doca_pe_create] Progress engine 0x5570f566cb80 was created
[22:34:57:563113][1211][DOCA][INF][udp_queues.c:45][create_udp_queues] Creating UDP Eth Rxq 0
[22:34:57:563140][1211][DOCA][INF][doca_eth_rxq.c:1955][doca_eth_rxq_set_type] ETH_RXQ 0x5570f566cc30: queue_type was set to DOCA_ETH_RXQ_TYPE_CYCLIC
[22:34:57:563160][1211][DOCA][INF][doca_mmap.cpp:558][doca_mmap_create] Mmap 0x5570f566d0c0 was created, access_mask=0x1
[22:34:57:563796][1211][DOCA][INF][udp_queues.c:121][create_udp_queues] Mapping receive queue buffer (0x0x7fc3ac000000 size 536870912B dmabuf fd 134) with dmabuf mode
[22:34:57:563818][1211][DOCA][INF][doca_mmap.cpp:1846][doca_mmap_set_dmabuf_memrange] Mmap 0x5570f566d0c0: Set dmabuf_memrange.
[22:34:57:563831][1211][DOCA][INF][doca_mmap.cpp:1950][doca_mmap_set_permissions] Mmap 0x5570f566d0c0: Set permissions with access_mask=0x41
[22:34:57:568642][1211][DOCA][INF][doca_mmap.cpp:741][doca_mmap_start] Mmap 0x5570f566d0c0: mmap was started
[22:34:57:568656][1211][DOCA][INF][doca_eth_rxq.c:2642][doca_eth_rxq_set_pkt_buf] ETH_RXQ 0x5570f566cc30: mmap was set to 0x5570f566d0c0
[22:34:57:568662][1211][DOCA][INF][doca_eth_rxq.c:2643][doca_eth_rxq_set_pkt_buf] ETH_RXQ 0x5570f566cc30: offset was set to 0
[22:34:57:568666][1211][DOCA][INF][doca_eth_rxq.c:2644][doca_eth_rxq_set_pkt_buf] ETH_RXQ 0x5570f566cc30: size was set to 536870912
[22:34:57:568673][1211][DOCA][INF][doca_ctx.cpp:203][doca_ctx_start] CTX 0x5570f566cc30 does not require PE
[22:34:57:571139][1211][DOCA][INF][doca_buf_array.cpp:218][doca_buf_arr] buf_arr 0x5570f559c3e0 was created
[22:34:57:571157][1211][DOCA][INF][doca_buf_array.cpp:219][doca_buf_arr] num_elem=65536
[22:34:57:571162][1211][DOCA][INF][doca_buf_array.cpp:286][set_target_gpu] buf_arr 0x5570f559c3e0: target_gpu was set to 0x5570f55a1490
[22:34:57:571167][1211][DOCA][INF][doca_buf_array.cpp:236][set_params] buf_arr 0x5570f559c3e0: elem_size was set to 8192
[22:34:57:571171][1211][DOCA][INF][doca_buf_array.cpp:237][set_params] buf_arr 0x5570f559c3e0: start_offset was set to 0
[22:34:57:572543][1211][DOCA][INF][doca_buf_array.cpp:414][start] buf_arr 0x5570f559c3e0: buf_arr was started
[22:34:57:573155][1211][DOCA][INF][doca_uar.cpp:207][bridge_init] UAR 0x5570f566f6c0 created: page=0x7fcc684b2000, reg_addr=0x7fcc684b2800, base_addr=0x7fcc684b2000, id=133, alloc_type=BLUEFLAME
[22:34:57:574980][1211][DOCA][ERR][linux_devx_adapter.cpp:322][umem_reg] devx adapter 0x5570f55281c0: Failed to register user memory. Got errno=UNKNOWN-errno14 (14)
[22:34:57:575072][1211][DOCA][ERR][doca_umem.cpp:256][bridge_init_dmabuf] Failed to register dmabuf umem with exception:
[22:34:57:575089][1211][DOCA][ERR][doca_umem.cpp:256][bridge_init_dmabuf] DOCA exception [DOCA_ERROR_DRIVER] with message Failed to register user memory
[22:34:57:575108][1211][DOCA][ERR][doca_umem.cpp:131][priv_doca_umem_create_dmabuf] Failed to create umem with dmabuf_fd with exception:
[22:34:57:575119][1211][DOCA][ERR][doca_umem.cpp:131][priv_doca_umem_create_dmabuf] DOCA exception [DOCA_ERROR_DRIVER] with message priv_doca_umem constructor failed
[22:34:57:577291][1211][DOCA][INF][eth_rxq_common.c:451][eth_rxq_common_create_cq] ETH_RXQ 0x5570f566cc30: Created CQ 0x49a
[22:34:57:577439][1211][DOCA][ERR][linux_devx_adapter.cpp:322][umem_reg] devx adapter 0x5570f55281c0: Failed to register user memory. Got errno=UNKNOWN-errno14 (14)
[22:34:57:577466][1211][DOCA][ERR][doca_umem.cpp:256][bridge_init_dmabuf] Failed to register dmabuf umem with exception:
[22:34:57:577482][1211][DOCA][ERR][doca_umem.cpp:256][bridge_init_dmabuf] DOCA exception [DOCA_ERROR_DRIVER] with message Failed to register user memory
[22:34:57:577495][1211][DOCA][ERR][doca_umem.cpp:131][priv_doca_umem_create_dmabuf] Failed to create umem with dmabuf_fd with exception:
[22:34:57:577505][1211][DOCA][ERR][doca_umem.cpp:131][priv_doca_umem_create_dmabuf] DOCA exception [DOCA_ERROR_DRIVER] with message priv_doca_umem constructor failed
[22:34:57:579415][1211][DOCA][INF][eth_rxq_common.c:718][eth_rxq_common_create_rq] ETH_RXQ 0x5570f566cc30: Created RQ 0xc0004b
[22:34:57:579559][1211][DOCA][ERR][linux_devx_adapter.cpp:322][umem_reg] devx adapter 0x5570f55281c0: Failed to register user memory. Got errno=UNKNOWN-errno14 (14)
[22:34:57:579585][1211][DOCA][ERR][doca_umem.cpp:256][bridge_init_dmabuf] Failed to register dmabuf umem with exception:
[22:34:57:579597][1211][DOCA][ERR][doca_umem.cpp:256][bridge_init_dmabuf] DOCA exception [DOCA_ERROR_DRIVER] with message Failed to register user memory
[22:34:57:579608][1211][DOCA][ERR][doca_umem.cpp:131][priv_doca_umem_create_dmabuf] Failed to create umem with dmabuf_fd with exception:
[22:34:57:579617][1211][DOCA][ERR][doca_umem.cpp:131][priv_doca_umem_create_dmabuf] DOCA exception [DOCA_ERROR_DRIVER] with message priv_doca_umem constructor failed
[22:34:57:580479][1211][DOCA][INF][eth_rxq_common.c:451][eth_rxq_common_create_cq] ETH_RXQ 0x5570f566cc30: Created CQ 0x49b
[22:34:57:580546][1211][DOCA][ERR][linux_devx_adapter.cpp:322][umem_reg] devx adapter 0x5570f55281c0: Failed to register user memory. Got errno=UNKNOWN-errno14 (14)
[22:34:57:580559][1211][DOCA][ERR][doca_umem.cpp:256][bridge_init_dmabuf] Failed to register dmabuf umem with exception:
[22:34:57:580569][1211][DOCA][ERR][doca_umem.cpp:256][bridge_init_dmabuf] DOCA exception [DOCA_ERROR_DRIVER] with message Failed to register user memory
[22:34:57:580578][1211][DOCA][ERR][doca_umem.cpp:131][priv_doca_umem_create_dmabuf] Failed to create umem with dmabuf_fd with exception:
[22:34:57:580589][1211][DOCA][ERR][doca_umem.cpp:131][priv_doca_umem_create_dmabuf] DOCA exception [DOCA_ERROR_DRIVER] with message priv_doca_umem constructor failed
[22:34:57:583586][1211][DOCA][INF][doca_qp.cpp:986][priv_doca_dev_qp_create] Device 0x5570f55271d0: qp=0x5570f566a780 was created
[22:34:57:585903][1211][DOCA][INF][doca_qp.cpp:948][set_state] QP 0x5570f566a780: State change INIT -> CONNECTED
[22:34:57:585935][1211][DOCA][INF][eth_rxq_common.c:299][eth_rxq_common_create_flush_qp] ETH_RXQ 0x5570f566cc30: Created flush QP 0xab
[22:34:57:590187][1211][DOCA][ERR][linux_devx_adapter.cpp:322][umem_reg] devx adapter 0x5570f55281c0: Failed to register user memory. Got errno=UNKNOWN-errno14 (14)
[22:34:57:590227][1211][DOCA][ERR][doca_umem.cpp:256][bridge_init_dmabuf] Failed to register dmabuf umem with exception:
[22:34:57:590243][1211][DOCA][ERR][doca_umem.cpp:256][bridge_init_dmabuf] DOCA exception [DOCA_ERROR_DRIVER] with message Failed to register user memory
[22:34:57:590263][1211][DOCA][ERR][doca_dev.cpp:2773][priv_doca_dev_mapped_memory_region_create_dmabuf] Failed to create mapped memory region with dmabuf: failed to allocate memory for mr with exception:
[22:34:57:590276][1211][DOCA][ERR][doca_dev.cpp:2773][priv_doca_dev_mapped_memory_region_create_dmabuf] DOCA exception [DOCA_ERROR_DRIVER] with message priv_doca_umem constructor failed
[22:34:57:590283][1211][DOCA][ERR][cb_ops.cpp:395][cb_doca_gpu_export_eth_rxq] Failed allocating Flush QP mem region
[22:34:57:590318][1211][DOCA][ERR][doca_eth_rxq.c:1665][eth_rxq_start_gpu_ctx] ETH_RXQ 0x5570f566cc30: Failed to start eth_rxq: unable to export eth_rxq to GPU. err=DOCA_ERROR_DRIVER
[22:34:57:592435][1211][DOCA][INF][eth_rxq_common.c:618][eth_rxq_common_destroy_rq] ETH_RXQ 0x5570f566cc30: Destroyed RQ 0xc0004b
[22:34:57:593352][1211][DOCA][INF][eth_rxq_common.c:524][eth_rxq_common_destroy_cq] ETH_RXQ 0x5570f566cc30: Destroyed CQ 0x49a
[22:34:57:593367][1211][DOCA][INF][doca_qp.cpp:999][priv_doca_dev_qp_destroy] Destroying qp=0x5570f566a780
[22:34:57:594825][1211][DOCA][INF][eth_rxq_common.c:174][eth_rxq_common_destroy_flush_qp] ETH_RXQ 0x5570f566cc30: Destroyed flush QP 0xab
[22:34:57:595397][1211][DOCA][INF][eth_rxq_common.c:524][eth_rxq_common_destroy_cq] ETH_RXQ 0x5570f566cc30: Destroyed CQ 0x49b
[22:34:57:595466][1211][DOCA][INF][doca_buf_array.cpp:427][stop] buf_arr 0x5570f559c3e0: buf_arr was stopped
[22:34:57:595472][1211][DOCA][ERR][doca_ctx.cpp:227][doca_ctx_start] Failed to start context 0x7ffff3f05ce8 with status DOCA_ERROR_DRIVER
[22:34:57:595477][1211][DOCA][ERR][udp_queues.c:180][create_udp_queues] Failed doca_ctx_start: DOCA Driver call failure
[22:34:57:595482][1211][DOCA][INF][udp_queues.c:267][destroy_udp_queues] Destroying UDP queue 0
[22:34:57:595487][1211][DOCA][INF][doca_pe.cpp:114][priv_doca_pe_ctx_destroy] Destroying progress engine ctx=0x5570f566cc30
[22:34:57:595493][1211][DOCA][INF][doca_mmap.cpp:575][doca_mmap_destroy] Mmap 0x5570f566d0c0: Destroying mmap
[22:34:57:595644][1211][DOCA][INF][doca_mmap.cpp:762][doca_mmap_stop] Mmap 0x5570f566d0c0: mmap was stopped
[22:34:57:595708][1211][DOCA][ERR][gpu_packet_processing.c:350][main] Function create_udp_queues returned Bad State
Please let me know if anything sticks out to your or there are other details I could provide. Thank you!