How much device memory can I allocate in ConnectX-6?

[cerotyki]

Hi.
I’m trying to use device memory in ConnectX-6.
When I query MEMIC_SIZE_LIMIT, it shows 256KB.
Is it the maximum device memory size I can allocate?
Also, what does MEMIC_BAR_SIZE mean?

I want to allocate more than 256KB on ConnectX-6 memory for partial DMA.
It is known to have 4MiB for its SRAM, but I do not know why MEMIC_SIZE_LIMIT cannot be set larger than 256KB.
Can you tell me how to allocate large device memory in ConnectX-6?

Thank you.

[ssimcoejr]

Hi cerotki,

Thank you for posting your inquiry to the NVIDIA Developer Forums.

Description for these firmware configuration elements can be found via mlxconfig (included with Mellanox Firmware Tools - MFT).

For example, using the latest MFT (available at this link), and assuming your device is mlx5_0:

mlxconfig -d mlx5_0 show_confs

The MEMIC CONF section displays the following details:

                MEMIC_BAR_SIZE=<NUM>                    The amount of BAR size assigned for MEMIC. The size in bytes is memic_size_limit*2^log_memic_bar_size.
                                                        When activating MEMIC Atomic access by MEMIC_ATOMIC, the allocated BAR size for MEMIC is doubled.
                MEMIC_SIZE_LIMIT=<DISABLED|_1024KB|_256KB|_512KB>The maximum amount of internal device memory that can be consumed by the MEMIC application.
                                                        0x0: DISABLED
                                                        0x1: _256KB
                                                        0x2: _512KB
                                                        0x3: _1024KB

If you have any additional questions about this, and have valid support entitlement, please open a support ticket with Enterprise Support.

Thanks,
NVIDIA Enterprise Experience

[cerotyki]

Thank you for your response.
I have one more simple question.
What do BAR and MEMIC mean?

[cerotyki]

Also, when I tried to set MEMIC_BAR_SIZE as larger than 0, or MEMIC_SIZE_LIMIT as larger than 1 (256KB), mlxconfig returned “Applying … Failed!”.

Parameter MEMIC_BAR_SIZE’ value is larger than maximum allowed 0
Parameter MEMIC_SIZE_LIMIT’ value is larger than maximum allowed 1

Is there any way to set the maximum allowed values?