How much device memory can I allocate in ConnectX-6?

I’m trying to use device memory in ConnectX-6.
When I query MEMIC_SIZE_LIMIT, it shows 256KB.
Is it the maximum device memory size I can allocate?
Also, what does MEMIC_BAR_SIZE mean?

I want to allocate more than 256KB on ConnectX-6 memory for partial DMA.
It is known to have 4MiB for its SRAM, but I do not know why MEMIC_SIZE_LIMIT cannot be set larger than 256KB.
Can you tell me how to allocate large device memory in ConnectX-6?

Thank you.

Hi cerotki,

Thank you for posting your inquiry to the NVIDIA Developer Forums.

Description for these firmware configuration elements can be found via mlxconfig (included with Mellanox Firmware Tools - MFT).

For example, using the latest MFT (available at this link), and assuming your device is mlx5_0:

mlxconfig -d mlx5_0 show_confs

The MEMIC CONF section displays the following details:

                MEMIC_BAR_SIZE=<NUM>                    The amount of BAR size assigned for MEMIC. The size in bytes is memic_size_limit*2^log_memic_bar_size.
                                                        When activating MEMIC Atomic access by MEMIC_ATOMIC, the allocated BAR size for MEMIC is doubled.
                MEMIC_SIZE_LIMIT=<DISABLED|_1024KB|_256KB|_512KB>The maximum amount of internal device memory that can be consumed by the MEMIC application.
                                                        0x0: DISABLED
                                                        0x1: _256KB
                                                        0x2: _512KB
                                                        0x3: _1024KB

If you have any additional questions about this, and have valid support entitlement, please open a support ticket with Enterprise Support.

NVIDIA Enterprise Experience

Thank you for your response.
I have one more simple question.
What do BAR and MEMIC mean?

Also, when I tried to set MEMIC_BAR_SIZE as larger than 0, or MEMIC_SIZE_LIMIT as larger than 1 (256KB), mlxconfig returned “Applying … Failed!”.

Parameter MEMIC_BAR_SIZE’ value is larger than maximum allowed 0
Parameter MEMIC_SIZE_LIMIT’ value is larger than maximum allowed 1

Is there any way to set the maximum allowed values?

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.