Setting a Connect-X card to ethernet mode before OS installed

Infrastructure & Networking Software And Drivers Mellanox OFED
1

We’ve bought quite a number of MT27800 Connect-X 5 cards. Some are delivered in Infiniband mode. In our data center, we install our servers using PXE over ethernet, which the cards support! But changing their config requires a full Linux install.

For some servers we were able to install Linux over an alternative 1G interface, and then install the NVIDIA tools from there, and bring the 100G cards up. But some other servers didn’t have another interface.

After a lot of frustration, we ended up sending the cards back and asking them to be reconfigured to ethernet by the vendor! Even with a working BMC, virtual media and a selection of live CDs, I couldn’t work out how to get the right tools installed.

Is there a trick I missed here, or is there a live CD (or BIOS option) that could do just this job?

2

Hi matthew187,

Welcome, and thank you for posting your inquiry to the NVIDIA Developer Forums.

2 options exist - RDE and using a LiveCD/image with MFT installed (NVIDIA Firmware Tools (MFT))

LiveCD option:

  • Boot into your LiveCD image which has MFT installed.
  • Find your device, either with MST character device or PCIe address (mst start && mst status -v, or lspci -d 15b3: respectively).
  • You can use either the MST device name (IE: /dev/mst/mt4115_pciconf0), or the PCIe address for the next step.
  • Invoke: mlxconfig -d (device) s LINK_TYPE_P1=2
  • Restart the system, or perform a function-level reset of the adapter with mlxfwreset -d (device) r --yes
  • Validate the configuration took effect with:
    mlxconfig -d mlx5_0 -e q | grep 'Configurations\|LINK_TYPE'
    Configurations: Default Current Next Boot
    * LINK_TYPE_P1 IB(1) ETH(2) ETH(2)

Redfish/RDE

ConnectX adapters (from ConnectX-4 onwards) support interfacing with adapters via Redfish/RDE through the server BMC. The implementation is vendor-specific, and some vendors may not have support for this capability - you would need to engage with your server vendor to understand capability and use - but the general steps would be as follows (assuming your server BMC is at 10.7.118.14 and your credentials are ‘root/password’:

Step 1:

# curl -skL -u root:password -X PATCH -H "Content-Type: application/json" -d "{\"NetDevFuncType\": \"InfiniBand\"}" https://10.7.118.14/redfish/v1/Chassis/1/NetworkAdapters/DE082000/NetworkDeviceFunctions/0

{"NetDevFuncType":"InfiniBand"}

Step 2: (Note: Although this step may return an error, it is required)

# curl -skL -u root:password -X PATCH -H "Content-Type: application/json" -d "{\"NetDevFuncType\": \"InfiniBand\"}" https://10.7.118.14/redfish/v1/Chassis/1/NetworkAdapters/DE082000/NetworkDeviceFunctions/0/Settings

{"error":{"code":"iLO.0.10.ExtendedInfo","message":"See @Message.ExtendedInfo for more information.","@Message.ExtendedInfo":[{"MessageId":"Base.1.18.Success"}]}}

Step 3:

# curl -skL -u root:password https://10.7.118.14/redfish/v1/Chassis/1/NetworkAdapters/DE082000/NetworkDeviceFunctions/0/Settings

{"@odata.id":"/redfish/v1/Chassis/1/NetworkAdapters/DE082000/NetworkDeviceFunctions/0/Settings","@odata.type":"#NetworkDeviceFunction.v1_8_0.NetworkDeviceFunction","@odata.etag":"W\/\"A06F124C\"","@Redfish.SettingsApplyTime":{"ApplyTime":"OnReset"},"BootMode":"PXE","DeviceEnabled":true,"Ethernet":{"MACAddress":"58:a2:e1:05:f6:dc","MTUSize":1522,"PermanentMACAddress":"58:a2:e1:05:f6:dc","MTUSizeMaximum":10000},"Id":"0","Links":{"PhysicalNetworkPortAssignment":{"@odata.id":"/redfish/v1/Chassis/1/NetworkAdapters/DE082000/Ports/0"}},"MaxVirtualFunctions":127,"Name":"NetworkDeviceFunction","NetDevFuncCapabilities":["Ethernet","InfiniBand"],**"NetDevFuncType":"InfiniBand"** ,"Status":{"Health":"OK","HealthRollup":"OK","State":"Enabled"},"VirtualFunctionsEnabled":true,"AssignablePhysicalNetworkPorts":[{"@odata.id":"/redfish/v1/Chassis/1/NetworkAdapters/DE082000/Ports/0"}],"InfiniBand":{"MTUSize":null,"NodeGUID":"58a2:e103:0005:f6dc","PermanentNodeGUID":"58a2:e103:0005:f6dc","PermanentPortGUID":"58a2:e103:0005:f6dc","PermanentSystemGUID":"58a2:e103:0005:f6dc","PortGUID":"58a2:e103:0005:f6dc","SystemGUID":"58a2:e103:0005:f6dc"},"Metrics":{"@odata.id":"/redfish/v1/Chassis/1/NetworkAdapters/DE082000/NetworkDeviceFunctions/0/Metrics"},"SAVIEnabled":false}

Once the above steps are completed, you can confirm the change with the mlxconfig command:

# mlxconfig -d mlx5_0 -e q | grep 'Configurations\|LINK_TYPE'
Configurations: Default Current Next Boot
LINK_TYPE_P1 IB(1) IB(1) IB(1)

If you experience difficulties, and have a valid NVIDIA Enterprise Support entitlement, we highly recommend opening a support case so we can assist you further.

Thanks, and best regards,
NVIDIA Enterprise Experience

3

Thanks for the fast reply - my question was: where can I find such a CD? 👀

That’s really interesting that the BMC can offer remote access but it sounds a bit fraught.

4

Hi matthew187,

Unfortunately, such an ISO doesn’t exist - typically, users will build their own.

We have an open-source flavor of MFT available in distro repositories (EG: Ubuntu, RedHat) - mstflint - which might be easier to pull down and automate than running the MFT installer.

Syntax is the same, except replace mlxconfig commands with mstconfig, and mlxfwreset with mstfwreset: GitHub - Mellanox/mstflint: Mstflint - an open source version of MFT (Mellanox Firmware Tools)

There are various tools that can be leveraged to modify LiveCD images.
One such example is cubic (link) , which requires a system with a GUI.

An example workflow for creating a Ubuntu LiveCD which has mstflint preinstalled using cubic might be:

Install cubic, and launch:

sudo apt-add-repository universe
sudo apt-add-repository ppa:cubic-wizard/release
sudo apt update
sudo apt install --no-install-recommends cubic
sudo cubic

Create project

  1. Create New Project:
  • Click “New Project”
  • Select your Ubuntu ISO file (e.g., ubuntu-22.04-desktop-amd64.iso)
  • Choose a project directory (e.g., ~/cubic-projects/ubuntu-mstflint)
  • Click “Next”
  1. Select Components:
  • Keep all components selected (this is usually the default)
  • Click “Next”

Customize the ISO

When cubic opens the terminal window, you’ll be in a chroot environment. From here:

# Update package lists
apt update

# Install mstflint
apt install -y mstflint

# Verify installation
mstflint --version

# Clean up package cache to reduce ISO size
apt clean
apt autoremove -y

# Exit the chroot environment
exit

Generate the Custom ISO

In Cubic GUI:

  • Click “Generate” button
  • Wait for the process to complete (this can take a few minutes)
  • The custom ISO will be saved in your project directory
  • Test the new image on a VM.

An alternate approach, using just the command line and squashfs-tools, can be scripted as well:

#!/bin/bash
# ubuntu-mstflint-custom.sh
set -e

# Configuration
ISO_FILE="ubuntu-22.04.3-desktop-amd64.iso"
WORK_DIR="$HOME/ubuntu-custom"
OUTPUT_ISO="ubuntu-custom-mstflint.iso"

echo "Creating custom Ubuntu ISO with mstflint..."

# Install dependencies
sudo apt update
sudo apt install -y squashfs-tools xorriso isolinux syslinux-common genisoimage

# Create working directory
mkdir -p "$WORK_DIR"
cd "$WORK_DIR"

# Mount and extract ISO
echo "Extracting ISO..."
sudo mkdir -p /mnt/iso
sudo mount -o loop "$ISO_FILE" /mnt/iso
cp -r /mnt/iso/* .
cp -r /mnt/iso/.disk .
sudo umount /mnt/iso

# Extract filesystem
echo "Extracting filesystem..."
sudo unsquashfs casper/filesystem.squashfs

# Mount for chroot
echo "Installing mstflint..."
sudo mount --bind /dev squashfs-root/dev
sudo mount --bind /run squashfs-root/run
sudo mount --bind /proc squashfs-root/proc
sudo mount --bind /sys squashfs-root/sys

# Install mstflint
sudo chroot squashfs-root bash -c "
apt update
apt install -y mstflint
apt clean
apt autoremove -y
"

# Unmount
sudo umount squashfs-root/dev
sudo umount squashfs-root/run
sudo umount squashfs-root/proc
sudo umount squashfs-root/sys

# Recreate filesystem
echo "Recreating filesystem..."
sudo rm casper/filesystem.squashfs
sudo mksquashfs squashfs-root casper/filesystem.squashfs -comp xz -e boot

# Update manifest
sudo chmod +w casper/filesystem.manifest
sudo chroot squashfs-root dpkg-query -W --showformat='${Package} ${Version}\n' > casper/filesystem.manifest
sudo chmod -w casper/filesystem.manifest

# Update size
sudo printf $(sudo du -sx --block-size=1 squashfs-root | cut -f1) > casper/filesystem.size

# Generate ISO
echo "Generating custom ISO..."
sudo genisoimage -o "$OUTPUT_ISO" \
    -b isolinux/isolinux.bin \
    -c isolinux/boot.cat \
    -no-emul-boot \
    -boot-load-size 4 \
    -boot-info-table \
    -joliet-long \
    -r \
    -V "Ubuntu Custom MSTFLINT" \
    -cache-inodes \
    -e boot/grub/efi.img \
    -no-emul-boot \
    -eltorito-alt-boot \
    -e boot/grub/efi.img \
    -no-emul-boot \
    .

# Make bootable
sudo isohybrid "$OUTPUT_ISO"

echo "Custom ISO created: $OUTPUT_ISO"
echo "Size: $(du -h "$OUTPUT_ISO" | cut -f1)"

A bit of lifting, yes, but once it’s done it’s done.

Hope this helps,
NVIDIA Enterprise Experience