When a ConnectX-8 NIC gets probed by the driver the HCA bits get cached. When the firmware of the card enables those capabilities after the driver probe happened, a rebind is required to get the latest set of HCA capabilities.
For instance, reseating a cable (or doing a PMAOS reset) can enable the bits that show whether enhanced reorder is supported. Conversely, we’ve seen a card come up with “set_reg_c” showing REG_C_6 to be available, resulting in DPDK enabling NAT64 support. However, when it actually tried to program this at a later stage it resulted in an error (25.11 LTS, which does not do lazy allocate NAT64 actions).
It’s risky to have this possibility of drift in production systems, is there a signal that can be polled or listen to invalidate this HCA cache? Or is there some event that we can watch that might cause the HCA capabilities to change?
It would be nice to also have an option through sysfs to refresh the cached HCA bits without requiring a full rebind.