K8S Nvidia Network Operator

Infrastructure & Networking DOCA Enterprise Networking
, ,
1

Hello Team.

We have cluster k8s with 4 baremetal nodes with 8GPU B200 and IB between of them. We did k8s cluster rollout via kubespray and after with gpu network operator installed doca ofed driver.

By default our nodes doesnt have any installation of opensmd.service and our nicClusterPolicy is like this:

apiVersion: mellanox.com/v1alpha1
kind: NicClusterPolicy
metadata:
  name: nic-cluster-policy
spec:
  ofedDriver:
    forcePrecompiled: false
    image: doca-driver
    imagePullSecrets: []
    repository: nvcr.io/nvidia/mellanox
    terminationGracePeriodSeconds: 300
    version: doca3.3.0-26.01-1.0.0.0-0
  rdmaSharedDevicePlugin:
    config: |
      {
        "configList": [
          {
            "resourceName": "rdma_shared_device_a",
            "rdmaHcaMax": 63,
            "selectors": {
              "vendors": ["15b3"],
              "deviceIDs": ["1021"],
              "drivers": ["mlx5_core"],
              "ifNames": [],
              "linkTypes": []
            }
          }
        ]
      }
    image: k8s-rdma-shared-dev-plugin
    imagePullSecrets: []
    repository: nvcr.io/nvidia/mellanox
    version: network-operator-v26.1.1

The main question do we have any issues without installled/configured opensmd.service? because We first did manual installation of OFED driver and after devicided to use operator. Nodes where it were installed manually now have service opensmd and it could be run.

Operator in this config didnt installled opensmd and Im woried could be a issue in the furure.

2

Hi,

Yes, your cluster will have issues without opensmd. An InfiniBand Subnet Manager is mandatory for the IB fabric to initialize. Without it, no LIDs get assigned, routing tables won’t be programmed, and RDMA traffic won’t flow.

The NVIDIA Network Operator does not deploy or manage opensmd, that’s intentional, as the SM is a fabric-level concern outside the operator’s scope.

  • Option 1 - Host-based OpenSM (software SM)
    Run opensmd as a systemd service on at least one node (two for redundancy). It coexists fine with the DOCA OFED driver container deployed by the operator

  • Option 2 - Switch-embedded SM
    If you’re using a managed IB switch (e.g., NVIDIA Quantum series), you can enable the embedded SM directly on the switch instead of running it on a host node. This offloads SM responsibility entirely from your compute nodes and is the preferred approach in larger fabrics.

In either case, only one active SM should be running at a time (others can run in standby mode for redundancy).

Hope that helps!

Ilan.