Hi,
I was wondering if you still had this issue?
I am running into the same issue using the gpunetio_simple_receive example. I get the following similar output:
[14:18:04:277350][36339][DOCA][INF][gpunetio_simple_receive_main.c:155][main] Starting the sample
[14:18:04:575992][36339][DOCA][INF][gpunetio_simple_receive_main.c:185][main] Sample configuration:
GPU 86:00.0
NIC af:00.0
[14:18:04:581563][36339][DOCA][WRN][engine_model.c:90][adapt_queue_depth] adapting queue depth to 128.
EAL: Detected CPU lcores: 20
EAL: Detected NUMA nodes: 2
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created
EAL: Probe PCI driver: mlx5_pci (15b3:101d) device: 0000:af:00.0 (socket 1)
EAL: Probe PCI driver: gpu_cuda (10de:20b7) device: 0000:86:00.0 (socket 1)
[14:18:06:245065][36339][DOCA][INF][gpunetio_simple_receive_sample.c:457][create_rxq] Creating Sample Eth Rxq
[14:18:06:245490][36339][DOCA][INF][gpunetio_simple_receive_sample.c:520][create_rxq] Mapping receive queue buffer (0x0x7f3d44000000 size 33554432B dmabuf fd 134) with dmabuf mode
[14:18:06:245539][36339][DOCA][ERR][linux_devx_adapter.cpp:322][umem_reg] devx adapter 0x557bd454dd60: Failed to register user memory. Got errno=UNKNOWN-errno14 (14)
[14:18:06:245621][36339][DOCA][ERR][doca_umem.cpp:256][bridge_init_dmabuf] Failed to register dmabuf umem with exception:
[14:18:06:245639][36339][DOCA][ERR][doca_umem.cpp:256][bridge_init_dmabuf] DOCA exception [DOCA_ERROR_DRIVER] with message Failed to register user memory
[14:18:06:245662][36339][DOCA][ERR][doca_mmap.cpp:276][priv_doca_mmap_dev_to_mkey_init_mkey] Failed to initialize mkey: failed to create memory region with exception:
[14:18:06:245675][36339][DOCA][ERR][doca_mmap.cpp:276][priv_doca_mmap_dev_to_mkey_init_mkey] DOCA exception [DOCA_ERROR_DRIVER] with message priv_doca_umem constructor failed
[14:18:06:245686][36339][DOCA][ERR][doca_mmap.cpp:431][priv_doca_mmap_init_dev_to_mkey] Mmap 0x557bd4747380: Failed to initialize device=0x557bd454df80. err=DOCA_ERROR_DRIVER
[14:18:06:245697][36339][DOCA][ERR][doca_mmap.cpp:473][priv_doca_mmap_init_dev_to_mkeys] Mmap 0x557bd4747380: Failed to initialize memory range. Failed to register MR for device with id: 1. err=DOCA_ERROR_DRIVER
[14:18:06:245707][36339][DOCA][ERR][gpunetio_simple_receive_sample.c:544][create_rxq] Failed to start mmap DOCA Driver call failure
[14:18:06:245718][36339][DOCA][INF][gpunetio_simple_receive_sample.c:381][destroy_rxq] Destroying Rxq
[14:18:07:053283][36339][DOCA][ERR][gpunetio_simple_receive_sample.c:647][gpunetio_simple_receive] Function create_rxq returned Bad State
[14:18:07:053303][36339][DOCA][INF][gpunetio_simple_receive_sample.c:381][destroy_rxq] Destroying Rxq
[14:18:07:053317][36339][DOCA][ERR][gpunetio_simple_receive_sample.c:401][destroy_rxq] Failed to free gpu memory: Operation not permitted
[14:18:07:053328][36339][DOCA][ERR][gpunetio_simple_receive_sample.c:686][gpunetio_simple_receive] Function destroy_rxq returned Bad State
[14:18:07:053338][36339][DOCA][ERR][gpunetio_simple_receive_main.c:191][main] gpunetio_simple_receive() encountered an error: Bad State
[14:18:07:053349][36339][DOCA][INF][gpunetio_simple_receive_main.c:203][main] Sample finished with errors
My NIC is a ConnectX-6 DX and I use an A30 GPU. I have no issues with my BAR size.
In https://docs.nvidia.com/doca/sdk/doca+gpunetio/index.html#src-2617060674_safe-id-aWQtLkRPQ0FHUFVOZXRJT3YyLjcuMC1HUFVNZW1vcnlNYXBwaW5nKG52aWRpYS1wZWVybWVtdnMuZG1hYnVmKQ, it is written that:
Later, when calling the doca_mmap_start, the DOCA library tries to map the GPU memory buffer using the dmabuf file descriptor. If it fails (something incorrectly set on the Linux system), it fallbacks trying to map the GPU buffer with the legacy mode (nvidia-peermem ). If it fails, an informative error is returned.
In the case of my program it uses dmabuf and then returns this as ‘informative’ error.
What steps could I take to resolve this?