I can’t see the ConnectX-7 interfaces in ifconfig or lspci. Is this normal?
0000:00:00.0 PCI bridge: NVIDIA Corporation Device 22ce (rev 01)
0002:00:00.0 PCI bridge: NVIDIA Corporation Device 22ce (rev 01)
0004:00:00.0 PCI bridge: NVIDIA Corporation Device 22ce (rev 01)
0004:01:00.0 Non-Volatile memory controller: Phison Electronics Corporation PS5027-E27T PCIe4 NVMe Controller (DRAM-less) (rev 01)
0007:00:00.0 PCI bridge: NVIDIA Corporation Device 22d0 (rev 01)
0007:01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. Device 8127 (rev 05)
0009:00:00.0 PCI bridge: NVIDIA Corporation Device 22d0 (rev 01)
0009:01:00.0 Network controller: MEDIATEK Corp. Device 7925
000f:00:00.0 PCI bridge: NVIDIA Corporation Device 22d1
000f:01:00.0 VGA compatible controller: NVIDIA Corporation Device 2e12 (rev a1)
running sudo dmesg | grep -E '0000:01:00|0002:01:00|AER|link down|remove' I get
`[ 0.077012] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability LTR]`
`[ 0.078267] pci 0000:01:00.0: [15b3:1021] type 00 class 0x020000 PCIe Endpoint`
`[ 0.078527] pci 0000:01:00.0: BAR 0 [mem 0x00000000-0x01ffffff 64bit pref]`
`[ 0.078559] pci 0000:01:00.0: ROM [mem 0x00000000-0x000fffff pref]`
`[ 0.079429] pci 0000:01:00.0: PME# supported from D3cold`
`[ 0.079780] pci 0000:01:00.0: VF BAR 0 [mem 0x00000000-0x000fffff 64bit pref]`
`[ 0.079785] pci 0000:01:00.0: VF BAR 0 [mem 0x00000000-0x007fffff 64bit pref]: contains BAR 0 for 8 VFs`
`[ 0.081376] pci 0000:01:00.1: [15b3:1021] type 00 class 0x020000 PCIe Endpoint`
`[ 0.081613] pci 0000:01:00.1: BAR 0 [mem 0x00000000-0x01ffffff 64bit pref]`
`[ 0.081644] pci 0000:01:00.1: ROM [mem 0x00000000-0x000fffff pref]`
`[ 0.082251] pci 0000:01:00.1: PME# supported from D3cold`
`[ 0.082588] pci 0000:01:00.1: VF BAR 0 [mem 0x00000000-0x000fffff 64bit pref]`
`[ 0.082592] pci 0000:01:00.1: VF BAR 0 [mem 0x00000000-0x007fffff 64bit pref]: contains BAR 0 for 8 VFs`
`[ 0.083589] pci 0000:01:00.0: BAR 0 [mem 0x7500000000-0x7501ffffff 64bit pref]: assigned`
`[ 0.083642] pci 0000:01:00.1: BAR 0 [mem 0x7502000000-0x7503ffffff 64bit pref]: assigned`
`[ 0.083697] pci 0000:01:00.0: ROM [mem 0x67100000-0x671fffff pref]: assigned`
`[ 0.083701] pci 0000:01:00.0: VF BAR 0 [mem 0x7504000000-0x75047fffff 64bit pref]: assigned`
`[ 0.083736] pci 0000:01:00.1: ROM [mem 0x67200000-0x672fffff pref]: assigned`
`[ 0.083740] pci 0000:01:00.1: VF BAR 0 [mem 0x7504800000-0x7504ffffff 64bit pref]: assigned`
`[ 0.083861] pci 0000:01:00.0: Max Payload Size set to 512/ 512 (was 128), Max Read Rq 512`
`[ 0.083914] pci 0000:01:00.1: Max Payload Size set to 512/ 512 (was 128), Max Read Rq 512`
`[ 0.084214] acpi PNP0A08:01: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability LTR]`
`[ 0.085346] pci 0002:01:00.0: [15b3:1021] type 00 class 0x020000 PCIe Endpoint`
`[ 0.085602] pci 0002:01:00.0: BAR 0 [mem 0x00000000-0x01ffffff 64bit pref]`
`[ 0.085634] pci 0002:01:00.0: ROM [mem 0x00000000-0x000fffff pref]`
`[ 0.086525] pci 0002:01:00.0: PME# supported from D3cold`
`[ 0.086882] pci 0002:01:00.0: VF BAR 0 [mem 0x00000000-0x000fffff 64bit pref]`
`[ 0.086886] pci 0002:01:00.0: VF BAR 0 [mem 0x00000000-0x007fffff 64bit pref]: contains BAR 0 for 8 VFs`
`[ 0.088459] pci 0002:01:00.1: [15b3:1021] type 00 class 0x020000 PCIe Endpoint`
`[ 0.088705] pci 0002:01:00.1: BAR 0 [mem 0x00000000-0x01ffffff 64bit pref]`
`[ 0.088737] pci 0002:01:00.1: ROM [mem 0x00000000-0x000fffff pref]`
`[ 0.089355] pci 0002:01:00.1: PME# supported from D3cold`
`[ 0.089700] pci 0002:01:00.1: VF BAR 0 [mem 0x00000000-0x000fffff 64bit pref]`
`[ 0.089704] pci 0002:01:00.1: VF BAR 0 [mem 0x00000000-0x007fffff 64bit pref]: contains BAR 0 for 8 VFs`
`[ 0.090715] pci 0002:01:00.0: BAR 0 [mem 0x3d00000000-0x3d01ffffff 64bit pref]: assigned`
`[ 0.090769] pci 0002:01:00.1: BAR 0 [mem 0x3d02000000-0x3d03ffffff 64bit pref]: assigned`
`[ 0.090826] pci 0002:01:00.0: ROM [mem 0x5d100000-0x5d1fffff pref]: assigned`
`[ 0.090831] pci 0002:01:00.0: VF BAR 0 [mem 0x3d04000000-0x3d047fffff 64bit pref]: assigned`
`[ 0.090862] pci 0002:01:00.1: ROM [mem 0x5d200000-0x5d2fffff pref]: assigned`
`[ 0.090866] pci 0002:01:00.1: VF BAR 0 [mem 0x3d04800000-0x3d04ffffff 64bit pref]: assigned`
`[ 0.090989] pci 0002:01:00.0: Max Payload Size set to 512/ 512 (was 128), Max Read Rq 512`
`[ 0.091043] pci 0002:01:00.1: Max Payload Size set to 512/ 512 (was 128), Max Read Rq 512`
`[ 0.091331] acpi PNP0A08:02: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability LTR]`
`[ 0.094243] acpi PNP0A08:03: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability LTR]`
`[ 0.095038] acpi PNP0A08:04: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability LTR]`
`[ 0.098285] acpi PNP0A08:05: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability LTR]`
`[ 0.099070] acpi PNP0A08:06: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability LTR]`
`[ 0.108165] acpi PNP0A08:0b: _OSC: OS now controls [PCIeHotplug PME AER PCIeCapability LTR]`
`[ 1.261265] pci 0000:01:00.0: Adding to iommu group 7`
`[ 1.261316] pci 0000:01:00.1: Adding to iommu group 8`
`[ 1.262227] pci 0002:01:00.0: Adding to iommu group 10`
`[ 1.262278] pci 0002:01:00.1: Adding to iommu group 11`
`[ 1.443578] pcieport 0000:00:00.0: AER: enabled with IRQ 330`
`[ 1.445266] pcieport 0002:00:00.0: AER: enabled with IRQ 333`
`[ 1.446816] pcieport 0004:00:00.0: AER: enabled with IRQ 336`
`[ 1.448490] pcieport 0007:00:00.0: AER: enabled with IRQ 339`
`[ 1.450099] pcieport 0009:00:00.0: AER: enabled with IRQ 342`
`[ 1.451066] pcieport 000f:00:00.0: AER: enabled with IRQ 345`
`[ 2.228780] mlx5_core 0000:01:00.0: enabling device (0000 -> 0002)`
`[ 2.228929] mlx5_core 0000:01:00.0: firmware version: 28.45.4028`
`[ 2.228950] mlx5_core 0000:01:00.0: 126.028 Gb/s available PCIe bandwidth (32.0 GT/s PCIe x4 link)`
`[ 2.599182] mlx5_core 0000:01:00.0: Rate limit: 127 rates are supported, range: 0Mbps to 195312Mbps`
`[ 2.600473] mlx5_core 0000:01:00.0: E-Switch: Total vports 10, per vport: max uc(128) max mc(2048)`
`[ 2.606438] mlx5_core 0000:01:00.0: Flow counters bulk query buffer size increased, bulk_query_len(8)`
`[ 2.613477] mlx5_core 0000:01:00.0: Port module event: module 0, Cable unplugged`
`[ 2.614616] mlx5_core 0000:01:00.0: mlx5_pcie_event:326:(pid 12): Detected insufficient power on the PCIe slot (27W).`
`[ 2.630034] mlx5_core 0000:01:00.0: mlx5e: IPSec ESP acceleration enabled`
`[ 2.785262] mlx5_core 0000:01:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 enhanced)`
`[ 2.791519] mlx5_core 0000:01:00.1: enabling device (0000 -> 0002)`
`[ 2.791663] mlx5_core 0000:01:00.1: firmware version: 28.45.4028`
`[ 2.791685] mlx5_core 0000:01:00.1: 126.028 Gb/s available PCIe bandwidth (32.0 GT/s PCIe x4 link)`
`[ 3.149502] mlx5_core 0000:01:00.1: Rate limit: 127 rates are supported, range: 0Mbps to 195312Mbps`
`[ 3.149977] mlx5_core 0000:01:00.1: E-Switch: Total vports 10, per vport: max uc(128) max mc(2048)`
`[ 3.153679] mlx5_core 0000:01:00.1: Flow counters bulk query buffer size increased, bulk_query_len(8)`
`[ 3.163118] mlx5_core 0000:01:00.1: Port module event: module 1, Cable unplugged`
`[ 3.165543] mlx5_core 0000:01:00.1: mlx5_pcie_event:326:(pid 164): Detected insufficient power on the PCIe slot (27W).`
`[ 3.175233] mlx5_core 0000:01:00.1: mlx5e: IPSec ESP acceleration enabled`
`[ 3.327916] mlx5_core 0000:01:00.1: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 enhanced)`
`[ 3.335686] mlx5_core 0002:01:00.0: enabling device (0000 -> 0002)`
`[ 3.335845] mlx5_core 0002:01:00.0: firmware version: 28.45.4028`
`[ 3.335871] mlx5_core 0002:01:00.0: 126.028 Gb/s available PCIe bandwidth (32.0 GT/s PCIe x4 link)`
`[ 3.721799] mlx5_core 0002:01:00.0: Rate limit: 127 rates are supported, range: 0Mbps to 195312Mbps`
`[ 3.723056] mlx5_core 0002:01:00.0: E-Switch: Total vports 10, per vport: max uc(128) max mc(2048)`
`[ 3.733990] mlx5_core 0002:01:00.0: Flow counters bulk query buffer size increased, bulk_query_len(8)`
`[ 3.744260] mlx5_core 0002:01:00.0: Port module event: module 0, Cable unplugged`
`[ 3.744790] mlx5_core 0002:01:00.0: mlx5_pcie_event:326:(pid 164): Detected insufficient power on the PCIe slot (27W).`
`[ 3.757909] mlx5_core 0002:01:00.0: mlx5e: IPSec ESP acceleration enabled`
`[ 3.925729] mlx5_core 0002:01:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 enhanced)`
`[ 3.932018] mlx5_core 0002:01:00.1: enabling device (0000 -> 0002)`
`[ 3.932192] mlx5_core 0002:01:00.1: firmware version: 28.45.4028`
`[ 3.932220] mlx5_core 0002:01:00.1: 126.028 Gb/s available PCIe bandwidth (32.0 GT/s PCIe x4 link)`
`[ 4.296236] mlx5_core 0002:01:00.1: Rate limit: 127 rates are supported, range: 0Mbps to 195312Mbps`
`[ 4.296727] mlx5_core 0002:01:00.1: E-Switch: Total vports 10, per vport: max uc(128) max mc(2048)`
`[ 4.301522] mlx5_core 0002:01:00.1: Flow counters bulk query buffer size increased, bulk_query_len(8)`
`[ 4.311381] mlx5_core 0002:01:00.1: Port module event: module 1, Cable unplugged`
`[ 4.312326] mlx5_core 0002:01:00.1: mlx5_pcie_event:326:(pid 410): Detected insufficient power on the PCIe slot (27W).`
`[ 4.324035] mlx5_core 0002:01:00.1: mlx5e: IPSec ESP acceleration enabled`
`[ 4.462233] mlx5_core 0002:01:00.1: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 enhanced)`
`[ 4.467266] mlx5_core 0002:01:00.1 enP2p1s0f1np1: renamed from eth3`
`[ 4.467498] mlx5_core 0000:01:00.0 enp1s0f0np0: renamed from eth0`
`[ 4.468014] mlx5_core 0002:01:00.0 enP2p1s0f0np0: renamed from eth2`
`[ 4.468244] mlx5_core 0000:01:00.1 enp1s0f1np1: renamed from eth1`
`[ 6.580497] mlx5_core 0000:01:00.0: E-Switch: Unload vfs: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)`
`[ 6.588138] mlx5_core 0000:01:00.0: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)`
`[ 7.906963] NOTICE: Automounting of tracing to debugfs is deprecated and will be removed in 2030`
`[ 8.652016] mlx5_core 0002:01:00.0 enP2p1s0f0np0: Link down`
`[ 9.043593] mlx5_core 0002:01:00.1 enP2p1s0f1np1: Link down`
`[ 9.779224] mlx5_core 0000:01:00.0: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)`
`[ 10.140858] mlx5_core 0000:01:00.0: E-Switch: cleanup`
`[ 10.478621] mlx5_core 0000:01:00.1: E-Switch: Unload vfs: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)`
`[ 10.496114] mlx5_core 0000:01:00.1: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)`
`[ 10.727929] mlx5_core 0000:01:00.1 enp1s0f1np1: Link down`
`[ 12.965214] mlx5_core 0000:01:00.1: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)`
`[ 13.704194] mlx5_core 0000:01:00.1: E-Switch: cleanup`
`[ 14.032326] mlx5_core 0002:01:00.0: E-Switch: Unload vfs: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)`
`[ 14.043122] mlx5_core 0002:01:00.0: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)`
`[ 16.793206] mlx5_core 0002:01:00.0: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)`
`[ 17.570512] mlx5_core 0002:01:00.0: E-Switch: cleanup`
`[ 17.896627] mlx5_core 0002:01:00.1: E-Switch: Unload vfs: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)`
`[ 17.915180] mlx5_core 0002:01:00.1: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)`
`[ 20.305118] mlx5_core 0002:01:00.1: E-Switch: Disable: mode(LEGACY), nvfs(0), necvfs(0), active vports(0)`
`[ 21.199705] mlx5_core 0002:01:00.1: E-Switch: cleanup`
`[ 21.650838] pcieport 0000:00:00.0: AER: Correctable error message received from 0000:00:00.0`
`[ 21.650849] pcieport 0002:00:00.0: AER: Multiple Correctable error message received from 0002:00:00.0`
running sudo dmesg | grep -Ei 'mlx5|mellanox' | head -30 I get
[ 2.228780] mlx5_core 0000:01:00.0: enabling device (0000 -> 0002)
[ 2.228929] mlx5_core 0000:01:00.0: firmware version: 28.45.4028
[ 2.228950] mlx5_core 0000:01:00.0: 126.028 Gb/s available PCIe bandwidth (32.0 GT/s PCIe x4 link)
[ 2.599182] mlx5_core 0000:01:00.0: Rate limit: 127 rates are supported, range: 0Mbps to 195312Mbps
[ 2.600473] mlx5_core 0000:01:00.0: E-Switch: Total vports 10, per vport: max uc(128) max mc(2048)
[ 2.606438] mlx5_core 0000:01:00.0: Flow counters bulk query buffer size increased, bulk_query_len(8)
[ 2.613477] mlx5_core 0000:01:00.0: Port module event: module 0, Cable unplugged
[ 2.614616] mlx5_core 0000:01:00.0: mlx5_pcie_event:326:(pid 12): Detected insufficient power on the PCIe slot (27W).
[ 2.630034] mlx5_core 0000:01:00.0: mlx5e: IPSec ESP acceleration enabled
[ 2.785262] mlx5_core 0000:01:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 enhanced)
[ 2.791519] mlx5_core 0000:01:00.1: enabling device (0000 -> 0002)
[ 2.791663] mlx5_core 0000:01:00.1: firmware version: 28.45.4028
[ 2.791685] mlx5_core 0000:01:00.1: 126.028 Gb/s available PCIe bandwidth (32.0 GT/s PCIe x4 link)
[ 3.149502] mlx5_core 0000:01:00.1: Rate limit: 127 rates are supported, range: 0Mbps to 195312Mbps
[ 3.149977] mlx5_core 0000:01:00.1: E-Switch: Total vports 10, per vport: max uc(128) max mc(2048)
[ 3.153679] mlx5_core 0000:01:00.1: Flow counters bulk query buffer size increased, bulk_query_len(8)
[ 3.163118] mlx5_core 0000:01:00.1: Port module event: module 1, Cable unplugged
[ 3.165543] mlx5_core 0000:01:00.1: mlx5_pcie_event:326:(pid 164): Detected insufficient power on the PCIe slot (27W).
[ 3.175233] mlx5_core 0000:01:00.1: mlx5e: IPSec ESP acceleration enabled
[ 3.327916] mlx5_core 0000:01:00.1: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 enhanced)
[ 3.335686] mlx5_core 0002:01:00.0: enabling device (0000 -> 0002)
[ 3.335845] mlx5_core 0002:01:00.0: firmware version: 28.45.4028
[ 3.335871] mlx5_core 0002:01:00.0: 126.028 Gb/s available PCIe bandwidth (32.0 GT/s PCIe x4 link)
[ 3.721799] mlx5_core 0002:01:00.0: Rate limit: 127 rates are supported, range: 0Mbps to 195312Mbps
[ 3.723056] mlx5_core 0002:01:00.0: E-Switch: Total vports 10, per vport: max uc(128) max mc(2048)
[ 3.733990] mlx5_core 0002:01:00.0: Flow counters bulk query buffer size increased, bulk_query_len(8)
[ 3.744260] mlx5_core 0002:01:00.0: Port module event: module 0, Cable unplugged
[ 3.744790] mlx5_core 0002:01:00.0: mlx5_pcie_event:326:(pid 164): Detected insufficient power on the PCIe slot (27W).
[ 3.757909] mlx5_core 0002:01:00.0: mlx5e: IPSec ESP acceleration enabled
[ 3.925729] mlx5_core 0002:01:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0 enhanced)
Are my ConnectX-7s functional? Do I have a PCIe power issue? I want to ensure my first GX10 is working before I even consider a second one.