Request: Increase MEMIC allocation on ConnectX-6

Infrastructure & Networking Adapters and Cables InfiniBand/VPI Adapter Cards
1

Dear NVIDIA Support,

I am a researcher working with on-NIC device memory (MEMIC) on the ConnectX-6, and I am seeking guidance on whether the MEMIC allocation can be increased beyond the current firmware-enforced limit on our card.

Device Information

Field Value
Device Type ConnectX-6
Part Number MCX653106A-HDA_Ax
Description ConnectX-6 VPI, HDR IB (200Gb/s) and 200GbE, dual-port QSFP56, PCIe4.0 x16
PSID MT_0000000225
PCI Address 0000:ca:00.0 (IB) / 0000:ca:00.1 (ETH)
Firmware 20.43.8002 (latest available, updated from 20.43.3608)
Base GUID e8eb:d303:003b:ecb7

The Problem

The firmware caps MEMIC at 256 KB total, organized as 128 KB per physical function:

MEMIC_SIZE_LIMIT = _256KB(1)   ← firmware max = 1, rejects _512KB(2) or _1024KB(3)
MEMIC_BAR_SIZE   = 0           ← firmware max = 0, rejects 1

Measured via ibv_alloc_dm:

Function max_dm_size Allocatable
mlx5_0 (IB) 128 KB 128 KB
mlx5_1 (ETH) 128 KB 128 KB
Total — 256 KB

MEMIC is per-function (not shared): we verified that mlx5_1 still gets 128 KB while mlx5_0 holds its full pool via a concurrent allocation test.

What We Have Tried

  1. mlxconfig set MEMIC_SIZE_LIMIT=_512KB → rejected: “value is larger than maximum allowed 1”
  2. mlxconfig set MEMIC_BAR_SIZE=1 → rejected: “value is larger than maximum allowed 0”
  3. Firmware update 20.43.3608 → 20.43.8002 → no change in limits
  4. SR-IOV: SRIOV_EN=1, NUM_OF_VFS=4 accepted in firmware, but PCIe SR-IOV capability is not exposed by this SKU (no sriov_totalvfs in sysfs, and lspci shows no SR-IOV extended capability on either function)
  5. Raw TLV / set_raw override → firmware validates at boot and rejects out-of-range values

This is consistent with another user’s experience reported on the NVIDIA Developer Forum: How much device memory can I allocate in ConnectX-6?

Our Goal

We are conducting research on using MEMIC as a low-latency on-NIC cache for distributed machine learning workloads. 256 KB (128 KB/function) is sufficient for a proof-of-concept, but larger MEMIC — ideally 1 MB or more — would significantly broaden the applicability of our approach.

We understand that the ConnectX-6 ASIC has approximately 4 MB of on-chip SRAM, and that most of it is reserved for steering, QP/CQ/WQE contexts. We are asking whether a larger portion of that SRAM can be re-allocated to the MEMIC pool on this specific SKU.

Our Questions

  1. Is there a firmware build — official or custom — that raises MEMIC_SIZE_LIMIT above _256KB (index 1) for PSID MT_0000000225?

  2. Is there a way to enable MEMIC_BAR_SIZE=1 on this card, effectively doubling the addressable MEMIC window (from 256 KB × 2^0 = 256 KB to 256 KB × 2^1 = 512 KB)?

  3. Does SR-IOV on this SKU require a different firmware image, or is PCIe SR-IOV simply not supported on the MCX653106A-HDA_Ax?

  4. If the above are not possible on this SKU, can you recommend a ConnectX-6 (or ConnectX-7) SKU that exposes 1 MB or more of user-allocatable MEMIC?

Context

This is an academic research project. We are happy to sign any required NDAs and would greatly appreciate any guidance — even if the answer is simply “this SKU cannot allocate more MEMIC.” Understanding the hardware’s ceiling is valuable for our paper.

Thank you for your time.

Best regards,
Steven

2

Hi Steven,

As far as I know, the MEMIC allocation can’t be increased.

Thanks,

Suo