Cannot-change-vf-trusted-mode-in-host-machine

Infrastructure & Networking BlueField
1

Trying to configure SRIOV VF trust = on after SRIOV pool is created for bluefield NIC card, it failed with following “operation not permittted” error even if sudo is added. how to debug this problem?

[core@xx-xx-xx-xx xx_xxcall]$ sudo ip link set enP2s2f1np1 vf 1 trust on
RTNETLINK answers: Operation not permitted

[core@xx-xx-xx-xx xx_xxcall]$ ip link show
4: enP2s2f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 5c:25:73:82:cd:8d brd ff:ff:ff:ff:ff:ff
vf 0 link/ether 76:85:01:04:f4:0d brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 1 link/ether 86:e3:a4:9a:21:7a brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 2 link/ether 0a:48:11:96:63:28 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 3 link/ether 6e:37:d5:ab:46:9e brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 4 link/ether 3e:2c:89:a7:91:11 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 5 link/ether e6:10:b1:a3:46:7d brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 6 link/ether 3a:4b:f5:1c:65:12 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 7 link/ether c6:9b:95:aa:43:ac brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 8 link/ether b2:7d:8b:37:18:81 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 9 link/ether 12:07:bb:ce:d5:44 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 10 link/ether b2:23:2a:f9:1f:16 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 11 link/ether 5e:cc:6b:e6:40:a9 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 12 link/ether 3a:f7:37:9c:88:4d brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 13 link/ether e2:ba:df:3f:4f:62 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 14 link/ether 82:92:61:8d:3e:87 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 15 link/ether 76:10:43:02:c6:66 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 16 link/ether 9a:42:e1:31:72:44 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 17 link/ether 3a:5f:3d:f7:63:a8 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 18 link/ether 52:49:02:87:a9:64 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
vf 19 link/ether 2e:10:0c:6a:1a:60 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off

Hardware information:

  • GPU model: GH200
  • CPU model: Grace A02
  • NIC card: 900-9D3B6-00CV-AA0(BF3)

Cloud platform: Redhat openshift version 4.17.3

NIC card configuration:
[core@xx-xx-xx-xx xx_xxcall]$ ethtool -i enP2s2f1np1
driver: mlx5_core
version: 5.14.0-427.42.1.el9_4.aarch64
firmware-version: 32.41.1000 (MT_0000000884)
expansion-rom-version:
bus-info: 0002:01:00.1

supports-statistics: yes

supports-test: yes

supports-eeprom-access: no

supports-register-dump: no

supports-priv-flags: yes

2

In addition, the internal CPU Offload engine is disabled in BIOS configuration.

[core@xx-xx-xx-xx-xx]$ ethtool -k enP2s2f1np1
Features for enP2s2f1np1:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: on
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: on
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: on [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on
tls-hw-tx-offload: on
tls-hw-rx-offload: off
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: on
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]

3

The issue is addressed via disable the DPU functionality in BIOS.