Following up on my first post with what I think settles the cause, plus a better
mitigation than the rollback I was originally going to suggest.
Short version: 6.14.0-1015-nvidia is clean. It is newer than the kernel these units
shipped with, it is in the archive now, and the ConnectX-7 comes up and stays up on it. I
have four GB10 units all running it with all four CX7 functions present and stable. Nobody
needs to sit on the ship-date 6.11 to keep their card.
Same-machine A/B. I rolled the originally faulted unit back to its shipped kernel using
a one-shot grub-reboot, so nothing was uninstalled. Same board, same BIOS, same CX7
firmware 28.45.4028, same cabling, same PSU, same physical placement — the kernel is the
only thing that differs between these two boots:
|
6.17.0-1029-nvidia |
6.11.0-1016-nvidia |
| CX7 functions on the PCIe bus |
0 |
4 |
| CX7 root-port link speed |
2.5 GT/s (Gen1) |
32.0 GT/s (Gen5), equal to max |
| mlx5 netdevs |
none |
all four |
E-Switch: Unload/Disable/cleanup |
yes, from t+5.74s |
none |
Detected insufficient power (27W) |
4x |
4x |
Port module event: Cable unplugged |
4x |
4x |
The window, bounded on both sides. Every row is a boot I actually performed:
| kernel |
units booted |
CX7 functions |
E-Switch teardown |
6.11.0-1016-nvidia (shipped) |
3 |
4 |
none |
6.14.0-1015-nvidia |
4 |
4 |
none |
6.17.0-1021-nvidia |
2 |
0 |
yes |
6.17.0-1029-nvidia |
1 |
0 |
yes |
So it lands after 6.14.0-1015 and by 6.17.0-1021, and it is not specific to one
kernel build — -1021 and -1029 both do it. If anyone has a unit on a build between
those two, that data point is the gap nobody has filled. I have 6.17.0-1008 installed on
one unit and will boot it shortly to try to close it further.
(Correction to my own earlier working notes: I had 6.17.0-1008 listed as faulted. I have
never booted it. It is not part of the evidence above.)
What a healthy 6.14 boot looks like, identical on all four units:
[ 1.205183] mlx5_core 0000:01:00.0: firmware version: 28.45.4028
[ 1.205205] mlx5_core 0000:01:00.0: 126.028 Gb/s available PCIe bandwidth (32.0 GT/s PCIe x4 link)
Binds at t+1.2s, zero E-Switch teardown events, and current_link_speed =
max_link_speed = 32.0 GT/s hours later. The bind is indistinguishable from the faulted
kernels — same moment, same firmware, same full Gen5 bandwidth. The difference is only that
on 6.17 something comes along afterwards and takes the card away.
On the 20-22s vs ~5.7s discrepancy — same event, measured at different ends. The
teardown is sequential, one PCI function at a time:
- one unit: first
Unload vfs at t+7.045s on 0000:01:00.0, final cleanup on
0002:01:00.1 at t+25.370s
- the other: t+5.733s through t+17.466s
My earlier “~5.7 seconds” was the first function going; the 20-22s in the original report
is consistent with the last one. I no longer think there are two failure paths here.
Two things this rules out, from a new direction. Both Detected insufficient power (27W) and Port module event: Cable unplugged fire four times each on every kernel I
have booted, including the ones where everything works — on the very machine that was
faulted an hour earlier. Three kernels, and neither message has tracked the fault on any of
them. If you are chasing either one, you are chasing platform noise. And since the -1021
units bind at full Gen5 before being torn down, the Gen1 link speed is a consequence of
the endpoint disappearing, not a cause.
There was never any firmware problem either. Booted on a healthy kernel with the card
present, nvidia-spark-mlnx-firmware-manager ran to completion by itself — found the
device, reported firmware already current at 28.45.4028, had nothing to install, and
disabled its own service. Its “Cable is not connected” message on the faulted kernel was
reporting device absence, not a cable.
Moving to 6.14
sudo apt-get install -y linux-image-6.14.0-1015-nvidia linux-modules-6.14.0-1015-nvidia \
linux-modules-nvidia-580-open-6.14.0-1015-nvidia
That third package is not optional. The NVIDIA kernel module is built per kernel
version. Install the kernel without it and you boot into a working ConnectX-7 and a dead
GPU — nvidia-smi reports “couldn’t communicate with the NVIDIA driver,” which looks
alarming and is simply a missing module. Check which build matches your userspace first;
mine needed the +6 build, and picking the wrong one drags the whole
nvidia-driver-580-open chain forward and can pull a 6.17 kernel back in behind it. Run
apt-get -s install ... first and read what it intends to do.
Three more things worth knowing:
- Pin the entry explicitly, with the submenu path.
GRUB_DEFAULT=0 means “newest,”
which is how you end up back on 6.17. Per-kernel entries live inside “Advanced options,”
and GRUB only resolves a nested id written as <submenu id>><entry id> — a bare nested
id silently matches nothing and you boot the default anyway. Get the ids with
grep -oP "\$menuentry_id_option .\K[^']+" /boot/grub/grub.cfg as root. Also apt-mark hold the kernel packages, or the next upgrade moves you regardless of GRUB.
- The GPU needs a power cycle, not just
modprobe. After installing the module into a
running system, mine loaded fine and created all five /dev/nvidia* nodes — and
nvidia-smi still said “No devices were found,” with
ksec2PrepareBootCommands: SEC2 secure boot partition timed out in the kernel log. SEC2
bootstraps once per device power cycle. A reboot cleared it completely. Don’t go hunting
for a driver or signing problem; check journalctl -k -b | grep NVRM and reboot.
- Secure Boot can stay enabled. The archive module is signed by Canonical and the shim
already trusts it. If the install pulls dkms in and you get the “Configuring Secure
Boot” MOK prompt, you do not need to enroll anything.
If you would rather test before committing, a one-shot boot changes nothing permanently and
reverts by itself:
sudo grub-reboot "gnulinux-advanced-<uuid>>gnulinux-6.14.0-1015-nvidia-advanced-<uuid>"
sudo reboot
@Lausanne — on the cables and the EEPROM
Your follow-up went down the FS-branded-cable and EEPROM-validation route. I think our data
retires that line: our units log Port module event: Cable unplugged four times per boot on
every kernel, including the ones where the card works perfectly and stays on the bus for
hours. On the faulted kernels the card is gone from the PCIe bus entirely — lspci shows
nothing at 15b3:1021 — which is not something a cable or a module EEPROM can cause. A DAC
that fails validation gets you a port that will not link; it does not get you a PCIe endpoint
that unbinds itself and disappears.
If you still have both kernels installed, the one-shot boot below will tell you in ten minutes
whether yours is the same bug.
Questions still open for NVIDIA
- Is this a known regression between
6.14.0-1015-nvidia and 6.17.0-1021-nvidia, and is
a fixed kernel in flight?
- Is holding at
6.14.0-1015-nvidia a supported interim position, and what is given up?
- Is there a module option or boot parameter that disables the cable-presence/hotplug
behaviour on the newer kernels so the card stays bound regardless of module state?
Happy to provide full journalctl -b -k -o short-monotonic captures from all four units on
every kernel, faulted and healthy, and to run a specific test on request.