Recently I use device memory on Connext5 ?
When I query max_dm_size using ibv_query_device_ex, the number is 128KB,
but when I use mlxconfig -d mlx5_0 q to query device firmware configuration
the MEMIC_SIZE_LIMIT = 256KB.
Why the two number are not same? Does that implies there is some MEMIC type memory is not used as dm which can be alloc by ibv_alloc_dm?
Besides what is the meaning MEMIC_BAR_SIZE? It says MEMIC_BAR_SIZE= The amount of BAR size assigned for MEMIC. The size in bytes is memic_size_limit*2^log_memic_bar_size.
Is log_memic_bar_size same as MEMIC_BAR_SIZE ?
Thank you for posting your query on our community.
max_dm_size reflects the maximum size of Device Memory that the driver and firmware currently expose as available for allocation via ibv_alloc_dm(). This is the portion of memory that your application can directly allocate and use.
MEMIC_SIZE_LIMIT represents the total size of the memory available to the NIC’s internal MEMIC engine for copy operations.
The MEMIC_BAR_SIZE refers to the actual size of the PCI BAR region that is reserved for MEMIC operations. This size is calculated using the formula:
MEMIC_BAR_SIZE = MEMIC_SIZE_LIMIT × 2^log_memic_bar_size
So, for example, with MEMIC_SIZE_LIMIT = 256KB and log_memic_bar_size = 2, the resulting MEMIC_BAR_SIZE would be 1MB.
To clarify, log_memic_bar_size is not the same as MEMIC_BAR_SIZE. It is a firmware parameter that determines how large the BAR region will be by scaling MEMIC_SIZE_LIMIT.
Hope this helps. If you have any additional questions about this, and have valid support entitlement, please open a support ticket with Enterprise Support.
That makes sense for me for the question “why MEMIC_SIZ_LIMIT and max_dm_size are not same?”
However, when I use mlxconfig -d mlx5_0 q , it shows MEMIC_BAR_SIZE = 0. It is unreasonable that we have MEMIC_SIZE_LIMIT = 256KB but MEMIC_BAR_SIZE = 0 for MEMIC_BAR_SIZE refers to the actual size of the PCI BAR region that is reserved for MEMIC operations. It does not match the formula:
MEMIC_BAR_SIZE = MEMIC_SIZE_LIMIT × 2^log_memic_bar_size