Apt upgrade has broken nvidia driver

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
,
1

Just hit a strange issue, just ran an update/upgrade via apt on an MSI EdgeXpert. Firmware is fully up to date and have had no issues with latest before this before updating packages. However, it seems to have broken the nvidia kernel module on all 3 of my units:


user@spark1:$ nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

Could anyone advise how to fix this? Has anyone else hit this issue?

It seems that upgrade to the latest kernel has not included the nvidia kernel module, i.e. nvidia.ko is not available in /lib/modules. Does anyone know what the correct name is of the nvidia kernel driver package?

2

I can see the following packages are available:

$ apt search linux-modules-nvidia | grep linux-modules | grep $(uname -r)

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

linux-modules-nvidia-535-6.17.0-1021-nvidia/noble-updates 6.17.0-1021.21 arm64
linux-modules-nvidia-535-6.17.0-1021-nvidia-64k/noble-updates 6.17.0-1021.21 arm64
linux-modules-nvidia-535-open-6.17.0-1021-nvidia/noble-updates 6.17.0-1021.21 arm64
linux-modules-nvidia-535-open-6.17.0-1021-nvidia-64k/noble-updates 6.17.0-1021.21 arm64
linux-modules-nvidia-535-server-6.17.0-1021-nvidia/noble-updates 6.17.0-1021.21 arm64
linux-modules-nvidia-535-server-6.17.0-1021-nvidia-64k/noble-updates 6.17.0-1021.21 arm64
linux-modules-nvidia-535-server-open-6.17.0-1021-nvidia/noble-updates 6.17.0-1021.21 arm64
linux-modules-nvidia-535-server-open-6.17.0-1021-nvidia-64k/noble-updates 6.17.0-1021.21 arm64
linux-modules-nvidia-580-6.17.0-1021-nvidia/noble-updates 6.17.0-1021.21 arm64
linux-modules-nvidia-580-6.17.0-1021-nvidia-64k/noble-updates 6.17.0-1021.21 arm64
linux-modules-nvidia-580-open-6.17.0-1021-nvidia/noble-updates 6.17.0-1021.21 arm64
linux-modules-nvidia-580-open-6.17.0-1021-nvidia-64k/noble-updates 6.17.0-1021.21 arm64
linux-modules-nvidia-580-server-6.17.0-1021-nvidia/noble-updates 6.17.0-1021.21 arm64
linux-modules-nvidia-580-server-6.17.0-1021-nvidia-64k/noble-updates 6.17.0-1021.21 arm64
linux-modules-nvidia-580-server-open-6.17.0-1021-nvidia/noble-updates 6.17.0-1021.21 arm64
linux-modules-nvidia-580-server-open-6.17.0-1021-nvidia-64k/noble-updates 6.17.0-1021.21 arm64
linux-modules-nvidia-595-6.17.0-1021-nvidia/noble-updates 6.17.0-1021.21 arm64
linux-modules-nvidia-595-6.17.0-1021-nvidia-64k/noble-updates 6.17.0-1021.21 arm64
linux-modules-nvidia-595-open-6.17.0-1021-nvidia/noble-updates 6.17.0-1021.21 arm64
linux-modules-nvidia-595-open-6.17.0-1021-nvidia-64k/noble-updates 6.17.0-1021.21 arm64
linux-modules-nvidia-595-server-6.17.0-1021-nvidia/noble-updates 6.17.0-1021.21 arm64
linux-modules-nvidia-595-server-6.17.0-1021-nvidia-64k/noble-updates 6.17.0-1021.21 arm64
linux-modules-nvidia-595-server-open-6.17.0-1021-nvidia/noble-updates 6.17.0-1021.21 arm64
linux-modules-nvidia-595-server-open-6.17.0-1021-nvidia-64k/noble-updates 6.17.0-1021.21 arm64
linux-modules-nvidia-fs-6.17.0-1021-nvidia/noble-updates,now 6.17.0-1021.21 arm64 \[installed,automatic\]
linux-modules-nvidia-fs-6.17.0-1021-nvidia-64k/noble-updates 6.17.0-1021.21 arm64

If I try to install one of them, I get:

user@spark3:\~$ sudo apt install linux-modules-nvidia-595-server-open-6.17.0-1021-nvidia
\[sudo\] password for user:
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
nvidia-kernel-common-580 : Conflicts: nvidia-kernel-common
nvidia-kernel-common-595-server : Conflicts: nvidia-kernel-common
E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.

I assume I need to just remove nvidia-kernel-common-580 and install 595 variants, but to be honest I’m a bit worried about getting the device into a bad state.

Edit:

I took the plunge and reinstalled 580 variants for now, as 595 came up with errors due to msimatched dependencies:

The following fixed it for now:

sudo apt install 
linux-modules-nvidia-580-open-6.17.0-1021-nvidia 
nvidia-utils-580 
libnvidia-compute-580

Are the 595 drivers ready for use, and could someone from the nvidia team advise how to migrate to it? @eugr_nv

3

Hi @p33zy,

Same issue here on 2× DGX Spark last week. The kernel upgrade to 6.17.0-1021 installed the new kernel but not the matching NVIDIA module — because your old nvidia-kernel-common-580 conflicts with the 595 packages.

Don’t worry, this is fixable without console access.

Step 1: Check your current driver series

dpkg -l | grep nvidia-kernel-common
dpkg -l | grep nvidia-driver

You likely have nvidia-kernel-common-580 installed, which blocks the 595 module.

Step 2: Remove the conflicting 580 package

sudo apt purge nvidia-kernel-common-580

This is safe — it only removes the old common files, not the running module.

Step 3: Install the pre-compiled module for 6.17.0-1021

Option A: Specific module for your current kernel

sudo apt install linux-modules-nvidia-595-open-6.17.0-1021-nvidia

Option B: HWE meta-package (auto-tracks future kernels — recommended)

sudo apt install linux-modules-nvidia-595-open-nvidia-hwe-24.04

Step 4: Reboot

sudo reboot

After reboot:

nvidia-smi  # Should work
uname -r    # Should show 6.17.0-1021-nvidia

Prevention — avoid this on your other 2 units

Before upgrading the kernel on your other units, install the HWE meta-package first:

sudo apt install linux-modules-nvidia-595-open-nvidia-hwe-24.04
sudo apt full-upgrade
sudo reboot

This ensures the pre-compiled module is always present when the kernel updates.

Why this happens

The apt upgrade installs the new kernel but the DKMS build fails on ARM64 (unsupported arch). The pre-compiled modules from Canonical’s repo are the intended path for DGX OS — they bypass DKMS entirely.

If you also have nvidia-dkms-595-open installed, I recommend purging it and putting it on hold to prevent future DKMS loops:

sudo apt purge nvidia-dkms-595-open
sudo apt-mark hold nvidia-dkms-595-open

Hope this helps!

4

[quote=“vince_du_66, post:3, topic:371799”]

sudo apt install linux-modules-nvidia-595-open-nvidia-hwe-24.04

Thnaks for that, I hit the following issue:

user@spark1:\~$ sudo apt install linux-modules-nvidia-595-open-nvidia-hwe-24.04
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
linux-modules-nvidia-595-open-nvidia-hwe-24.04 : Depends: linux-modules-nvidia-595-open-6.17.0-1021-nvidia (= 6.17.0-1021.21) but it is not going to be installed
Depends: nvidia-kernel-common-595 (>= 595.71.05) but it is not going to be installed
nvidia-utils-580 : Depends: libnvidia-compute-580 (>= 580.159.03) but it is not going to be installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

If I try to remove them:

user@spark1:\~$ sudo apt remove libnvidia-compute-580 nvidia-utils-580
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package 'libnvidia-compute-580' is not installed, so not removed
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
nvidia-conf-xconfig : Depends: nvidia-utils or
nvidia-xconfig but it is not going to be installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

I’m not sure if it’s a good idea to keep removing further than that?

5

I installed the driver, but now I’m hitting the following erorr in the kernel. Trying to run step3.7 flash:

\[   65.386217\] NVRM: GPU0 rpcRmApiAlloc_GSP: GspRmAlloc failed: hClient=0xc1e000b0; hParent=0xbaba0045; hObject=0xbaba0046; hClass=0x0000ce97; paramsSize=0x00000000; paramsStatus=0x00000065; status=0x00000065
\[   65.386235\] NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Call timed out \[NV_ERR_TIMEOUT\] (0x00000065) returned from pRmApi->AllocWithHandle(pRmApi, hClientId, KGRAPHICS_CHANNEL_HANDLE_CHANNELID, KGRAPHICS_CHANNEL_HANDLE_3DOBJ, classNum, NULL, 0) @ kernel_graphics.c:2528
\[   65.388561\] NVRM: GPU at PCI:000f:01:00: GPU-f293c40f-a079-591d-f46a-61c8ca35a601
\[   65.388563\] NVRM: Xid (PCI:000f:01:00): 44, pid=1244, name=nv_open_q, channel 0x00000001, intr 00000000
\[   65.401166\] NVRM: GPU0 kgspHealthCheck_TU102: \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* GSP-CrashCat Report \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*
\[   65.401168\] NVRM: GPU0 kgspPrintGspBinBuildId_IMPL: GSP bin buildId: 6332050fe14352839619385982f83700835ef570
\[   65.401171\] NVRM: Xid (PCI:000f:01:00): 120, pid=1244, name=nv_open_q, GSP task exception: load access page fault (cause:0xd) @ pc:0x1367b84, partition:4#0, task:3
\[   65.401174\] NVRM:     Reported by libos partition:4#5 kernel v3.1 \[0\] @ ts:84
\[   65.401175\] NVRM:     RISC-V CSR State:
\[   65.401176\] NVRM:         sstatus:0x0000000200000020  sscratch:0xffffffffa30124f0     sie:0x0000000000000220  sip:0x0000000000000000
\[   65.401177\] NVRM:         sepc:0x0000000001367b84     stval:0x0000000000000250  scause:0x000000000000000d
\[   65.401178\] NVRM:     RISC-V GPR State:
\[   65.401179\] NVRM:         ra:0x000000000147bfe2   sp:0x000000001b80f740   gp:0x0000000000000000   tp:0x000000001bc00000
\[   65.401180\] NVRM:         a0:0x0000000000000000   a1:0x0000ffffffffffff   a2:0x0000000000000002   a3:0x0000000000000000
\[   65.401181\] NVRM:         a4:0x000000000b800000   a5:0x0000000000002e70   a6:0x000000000b807388   a7:0x0000000000000002
\[   65.401182\] NVRM:         s0:0x000000001b80f770   s1:0x0000000014a3b250   s2:0x000000001b80fa70   s3:0x0000000014988b30
\[   65.401183\] NVRM:         s4:0x0000000000000021   s5:0x000000001b80fa20   s6:0x000000000000ffff   s7:0x0000000004165630
\[   65.401184\] NVRM:         s8:0x0000000000000000   s9:0x0000000014b5b3b0  s10:0x0000000004165630  s11:0x0000000014a3b250
\[   65.401184\] NVRM:         t0:0x000000001b80f5bd   t1:0x0002000000000000   t2:0x0000000000000000   t3:0x0000000020000000
\[   65.401185\] NVRM:         t4:0x0000000000000008   t5:0x000000001b80f5a1   t6:0x0000000000000020
\[   65.401186\] NVRM:     Stack Trace:
\[   65.401187\] NVRM:         0x0000000001367b84
\[   65.401187\] NVRM:         0x000000000147bfe2
\[   65.401188\] NVRM:         0x000000000146dce2
\[   65.401189\] NVRM:         0x000000000143b37a
\[   65.401189\] NVRM:         0x0000000001547770
\[   65.401190\] NVRM:         0x0000000001b1322c
\[   65.401191\] NVRM:         0x0000000001b135cc
\[   65.401191\] NVRM:         0x0000000001a24ff4
\[   65.401192\] NVRM:         0x0000000001b72578
\[   65.401192\] NVRM:         0x0000000001a0c6e0
\[   65.401193\] NVRM:     PC Trace:
\[   65.401194\] NVRM:         0x0000000001367b84  0x000000000147bfde  0x0000000001a0c930  0x0000000001a0c930  0x00000000013fde66
\[   65.401195\] NVRM:         0x0000000001b3d930  0x00000000013fde76  0x0000000001430bdc
\[   65.401195\] NVRM:     Local I/O Register State:
\[   65.401196\] NVRM:         0x01450800:0x00000000   0x01450900:0xbadf1002   0x01450a00:0x00000000   0x01450c00:0x00000000
\[   65.401197\] NVRM:         0x01454a00:0x810490d0   0x01454b00:0x010800d0   0x01454c00:0x00080000   0x01400200:0x00000040
\[   65.401198\] NVRM:     ------------\[ end crash report \]------------
\[   65.401215\] NVRM: GPU0 GSP RPC buffer contains function 10 (FREE) sequence 85 and data 0x00000000baba0042 0x0000000000000000.
\[   65.401216\] NVRM: GPU0 RPC history (CPU -> GSP):
\[   65.401217\] NVRM:     entry function                     sequence data0              data1              ts_start           ts_end             duration actively_polling
\[   65.401218\] NVRM:      0    10   FREE                          85 0x00000000baba0042 0x0000000000000000 0x0006530e1414615a 0x0000000000000000          y
\[   65.401220\] NVRM:     -1    10   FREE                          84 0x00000000baba0045 0x0000000000000000 0x0006530e14144f6d 0x0006530e14146054   4327us
\[   65.401221\] NVRM:     -2    103  GSP_RM_ALLOC                  83 0x000000000000ce97 0x0000000000000000 0x0006530e13cf056c 0x0006530e14144f2a   4540ms
\[   65.401223\] NVRM:     -3    76   GSP_RM_CONTROL                82 0x000000002080012b 0x0000000000000230 0x0006530e13ced10a 0x0006530e13cf055a  13392us
\[   65.401224\] NVRM:     -4    103  GSP_RM_ALLOC                  81 0x000000000000ca6f 0x0000000000000170 0x0006530e13cebd9a 0x0006530e13cecefa   4448us
\[   65.401225\] NVRM:     -5    76   GSP_RM_CONTROL                80 0x0000000020802a08 0x0000000000000004 0x0006530e13cebc49 0x0006530e13cebd65    284us
\[   65.401226\] NVRM:     -6    76   GSP_RM_CONTROL                79 0x0000000090f10106 0x00000000000000b8 0x0006530e13ceb5b2 0x0006530e13cebc16   1636us
\[   65.401227\] NVRM:     -7    103  GSP_RM_ALLOC                  78 0x00000000000090f1 0x0000000000000038 0x0006530e13ceb391 0x0006530e13ceb594    515us
\[   65.401228\] NVRM: GPU0 RPC event history (CPU <- GSP):
\[   65.401229\] NVRM:     entry function                     sequence data0              data1              ts_start           ts_end             duration during_incomplete_rpc
\[   65.401230\] NVRM:      0    4100 RC_TRIGGERED                   0 0x0000000000000001 0x000000000000002c 0x0006530e14145ac4 0x0006530e14145aff     59us
\[   65.401231\] NVRM:     -1    4102 OS_ERROR_LOG                   0 0x0000000000000000 0x0000000000000000 0x0006530e14145875 0x0006530e1414587d      8us
\[   65.401233\] NVRM:     -2    4101 MMU_FAULT_QUEUED               0 0x0000000000000000 0x0000000000000000 0x0006530e14145328 0x0006530e1414532b      3us
\[   65.401234\] NVRM:     -3    4124 GSP_LOCKDOWN_NOTICE            0 0x0000000000000000 0x0000000000000000 0x0006530e13cd96c1 0x0006530e13cd96c1
\[   65.401235\] NVRM:     -4    4124 GSP_LOCKDOWN_NOTICE            0 0x0000000000000001 0x0000000000000000 0x0006530e13cd8551 0x0006530e13cd8551
\[   65.401236\] NVRM:     -5    4124 GSP_LOCKDOWN_NOTICE            0 0x0000000000000000 0x0000000000000000 0x0006530e13cd83d9 0x0006530e13cd83d9
\[   65.401237\] NVRM:     -6    4124 GSP_LOCKDOWN_NOTICE            0 0x0000000000000001 0x0000000000000000 0x0006530e13cd7e31 0x0006530e13cd7e31
\[   65.401238\] NVRM:     -7    4124 GSP_LOCKDOWN_NOTICE            0 0x0000000000000000 0x0000000000000000 0x0006530e13ca3790 0x0006530e13ca3790
\[   65.401240\] NVRM: GPU0 krcRcAndNotifyAllChannels_IMPL: RC all user channels for critical error 120.
\[   65.401250\] NVRM: GPU0 kgspHealthCheck_TU102: \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*
\[   65.401269\] NVRM: GPU0 \_issueRpcAndWait: rpcRecvPoll failed with status 0x00000062 for fn 10 sequence 85!
\[   65.401270\] NVRM: GPU0 rpcRmApiFree_GSP: GspRmFree failed: hClient=0xc1e000b0; hObject=0xbaba0042; paramsStatus=0x00000000; status=0x00000062
\[   65.401271\] NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ vaspace_api.c:573
\[   65.401293\] NVRM: GPU0 nvCheckOkFailedNoLog: Check failed: Call timed out \[NV_ERR_TIMEOUT\] (0x00000065) returned from kgraphicsCreateGoldenImageChannel(pGpu, pKernelGraphics) @ kernel_graphics.c:510
\[   65.401294\] NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: 0 @ kernel_fifo.c:3110
\[   65.401303\] NVRM: GPU0 RmInitNvDevice: \*\*\* Cannot load state into the device
\[   65.401304\] NVRM: GPU0 RmInitAdapter: RmInitNvDevice failed, bailing out of RmInitAdapter
\[   65.401326\] NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ fecs_event_list.c:1623
\[   65.401542\] NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_client.c:844
\[   65.401555\] NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: (status == NV_OK) || (status == NV_ERR_GPU_IN_FULLCHIP_RESET) @ rs_server.c:259
\[   65.401562\] NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: rmStatus == NV_OK @ kern_disp.c:298
\[   65.401628\] NVRM: GPU0 nvCheckOkFailedNoLog: Check failed: Reset required \[NV_ERR_RESET_REQUIRED\] (0x00000062) returned from pRmApi->Control(pRmApi, pGpu->hInternalClient, pGpu->hInternalSubdevice, NV2080_CTRL_CMD_INTERNAL_INIT_USER_SHARED_DATA, &params, sizeof(params)) @ gpu_user_shared_data.c:248
\[  120.738781\] NVRM: GPU0 \_threadNodeCheckTimeout: \_threadNodeCheckTimeout: currentTime: f421655fcbb80 >= f421655fcbb80
\[  120.738786\] NVRM: GPU0 \_threadNodeCheckTimeout: \_threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
\[  120.738798\] NVRM: GPU0 \_threadNodeCheckTimeout: \_threadNodeCheckTimeout: currentTime: f421655fcbb80 >= f421655fcbb80
\[  120.738799\] NVRM: GPU0 \_threadNodeCheckTimeout: \_threadNodeCheckTimeout: Timeout was set to: 4000 msecs!
\[  120.738800\] NVRM: GPU0 kflcnWaitForHaltRiscv_GA102: Timeout waiting for RISC-V to halt
\[  120.739233\] NVRM: GPU0 nvAssertFailedNoLog: Assertion failed: rmStatus == NV_OK @ osinit.c:2501
\[  120.743639\] NVRM: GPU0 iovaspaceDestruct_IMPL: 17 left-over mappings in IOVAS 0xf0100
\[  120.743669\] NVRM: GPU 000f:01:00.0: RmInitAdapter failed! (0x25:0x65:1373)
\[  120.745554\] NVRM: GPU 000f:01:00.0: rm_init_adapter failed, device minor number 0
6

I believe this past thread may resolve your accidental override of the official DGXOS nvidia drivers:

Keep in mind that it’s best to stay away from doing a blanket apt update & upgrade and use the DGX dashboard process instead. They moved all the updates in the dedicated “updates” tab, where it shows the green dot badge every time there are new ones (which is A LOT).

You can still use apt for specific updates esp security patches.

7

Is it really necessary to expose the web interface to do this? Previous upgrades have all been fine, it seems like this one has just not been released in a stable way.

This has made my sparks unusable at the moment. Managed to get things installed and somewhat up and running, but the system is now unstable. Can see some PCIE related errors on boot and some kernel tracebacks sometimes. For example:

\[   23.201235\] pcieport 0000:00:00.0: AER: Correctable error message received from 0000:00:00.0
\[   23.201260\] pcieport 0000:00:00.0: PCIe Bus Error: severity=Correctable, type=Physical Layer, (Receiver ID)
\[   23.201263\] pcieport 0000:00:00.0:   device \[10de:22ce\] error status/mask=00000001/0000e000
\[   23.201266\] pcieport 0000:00:00.0:    \[ 0\] RxErr                  (First)
\[   23.201321\] pcieport 0002:00:00.0: AER: Multiple Correctable error message received from 0002:00:00.0
\[   23.201335\] pcieport 0002:00:00.0: PCIe Bus Error: severity=Correctable, type=Physical Layer, (Receiver ID)
\[   23.201337\] pcieport 0002:00:00.0:   device \[10de:22ce\] error status/mask=00000001/0000e000
\[   23.201339\] pcieport 0002:00:00.0:    \[ 0\] RxErr                  (First)
\[   31.638563\] systemd-journald\[626\]: /var/log/journal/0c025d09e0a84c6dbbb9fadd070c5de9/user-1000.journal: Journal file uses a different sequence number ID, rotating.
\[   48.566104\] NVRM: GPU at PCI:000f:01:00: GPU-25eed3ea-2443-55ae-d379-0018a9169f28
\[   48.566111\] NVRM: Xid (PCI:000f:01:00): 32, pid=1258, name=nv_open_q, channel 0x00000001 intr 00008000
\[   48.568550\] NVRM: Xid (PCI:000f:01:00): 32, pid=1258, name=nv_open_q, channel 0x00000001 intr 00008000
\[   69.945060\] NVRM: GPU at PCI:000f:01:00: GPU-25eed3ea-2443-55ae-d379-0018a9169f28
\[   69.945067\] NVRM: Xid (PCI:000f:01:00): 31, pid=1258, name=nv_open_q, channel 0x00000001, intr 00000000. MMU Fault: ENGINE GR_HOST0 HUBCLIENT_ESC0 faulted @ 0x1_20000000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ

I think the problem is I’ve just installed the wrong packages or some configuration step has not happened. Maybe it’s just the firmware image being loaded is wrong.

What I’ve installed is:

sudo apt purge nvidia-dkms-580-open nvidia-utils-580 nvidia-conf-xconfig
sudo apt install linux-modules-nvidia-595-open-6.17.0-1021-nvidia nvidia-utils-595 nvidia-conf-xconfig nvidia-firmware-595-595.71.05
8

@p33zy DKMS is not being used on the Spark. The current Nvidia driver is provided by the metapackage nvidia-driver-580-open

If you really want to use the 59x drivers, you have to install the metapackage nvidia-driver-590-open. It will pull in all the dependencies. However, it’s not recommended to jump to 59x driver, yet.

We’re waiting for an official release!

Update:

After running sudo apt update on one of the Sparks this happened:

elsaco@spark2:/var/log/apt$ modinfo nvidia
modinfo: ERROR: Module nvidia not found.

There’s a new kernel 6.17.0-1021-nvidia but it seems to be an issue with the update.

The relevant packages being:

The following NEW packages will be installed:
  linux-headers-6.17.0-1021-nvidia linux-image-6.17.0-1021-nvidia linux-modules-6.17.0-1021-nvidia
  linux-modules-nvidia-fs-6.17.0-1021-nvidia linux-nvidia-6.17-headers-6.17.0-1021 linux-nvidia-6.17-tools-6.17.0-1021
  linux-tools-6.17.0-1021-nvidia

Time for troubleshooting. There goes my afternoon!

9

I agree with @elsaco . Not a good idea to pull down and install the 590s yet. The nvidia libs are very driver dependent, esp. if you do not have any system snapshots.

In terms of recovery, unless you used Timeshift or similar rsync backup/restore package, you can only roll back from the factory image boot and redo your configurations. At both cases (including with Timeshift), you should copy your home dir if you have files/configs there you need.

10

In my case after running sudo apt update some of the packages were held back resulting in the new kernel being installed (6.17.0-1021-nvidia) but nvidia-driver-580-open not. This is the package with the actual nvidia.ko, and other Nvidia drivers.

sudo apt full-ugrade will fix it.

11

There is definitely some bad advise in this thread. Don’t install the 595 kernel modules - one of my units couldn’t even boot after that. I had to do a full system restore. Furthermore nvidia-driver-580-open resulted in building the module and self-signing it - you’ll need to enroll your key in secure boot. It’s not the best idea to have a kernel module signing key available on the disk of the box.

Was that the same behaviour for other people in this thread? Was the 580 variant that I originally installed before switching to 595 definitely the correct one? It would be best if a member of the Nvidia team could advise the correct way to fix this.

As an aside, the packages that are not compatible with the spark shouldn’t be available in the repositories installed on it. Everyone is running the same hardware, so there should be a seperate package list for it to avoid people bricking their devices. I’m not sure if this bad release is MSI’s fault, or Nvidia’s, but greater care needs to be put in.

12

I have no issues whatsoever. I own one NVIDIA FE and three EdgeXpert devices. So far, I have performed updates by entering commands strictly according to NVIDIA’s instructions. Even after the upgrade, nvidia-smi works well on all devices. I have never once had to go through the hassle of trying to use a different version of the NVIDIA driver.

@spark-00d2:~$ dgx-all "uname -r && nvidia-smi"
=== 169.254.71.59 ===
6.17.0-1021-nvidia
Sun May 31 10:37:45 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.159.03             Driver Version: 580.159.03     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GB10                    On  |   0000000F:01:00.0 Off |                  N/A |
| N/A   44C    P8              4W /  N/A  | Not Supported          |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            2584      G   /usr/lib/xorg/Xorg                       18MiB |
|    0   N/A  N/A            2708      G   /usr/bin/gnome-shell                      6MiB |
+-----------------------------------------------------------------------------------------+
=== 169.254.93.49 ===
6.17.0-1021-nvidia
Sun May 31 10:37:46 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.159.03             Driver Version: 580.159.03     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GB10                    On  |   0000000F:01:00.0 Off |                  N/A |
| N/A   45C    P8              4W /  N/A  | Not Supported          |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            2589      G   /usr/lib/xorg/Xorg                       18MiB |
|    0   N/A  N/A            2691      G   /usr/bin/gnome-shell                      6MiB |
+-----------------------------------------------------------------------------------------+
=== 169.254.218.227 ===
6.17.0-1021-nvidia
Sun May 31 10:37:46 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.159.03             Driver Version: 580.159.03     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GB10                    On  |   0000000F:01:00.0 Off |                  N/A |
| N/A   45C    P8              5W /  N/A  | Not Supported          |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            2576      G   /usr/lib/xorg/Xorg                       18MiB |
|    0   N/A  N/A            2707      G   /usr/bin/gnome-shell                      6MiB |
+-----------------------------------------------------------------------------------------+
=== 169.254.46.240 ===
6.17.0-1021-nvidia
Sun May 31 10:37:47 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.159.03             Driver Version: 580.159.03     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GB10                    On  |   0000000F:01:00.0 Off |                  N/A |
| N/A   45C    P8              4W /  N/A  | Not Supported          |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            2593      G   /usr/lib/xorg/Xorg                       18MiB |
|    0   N/A  N/A            2725      G   /usr/bin/gnome-shell                      6MiB |
+-----------------------------------------------------------------------------------------+
13

I had a a sticky kernel not updating but after a cold reset and it took. You’re probably past that but it’s a harmless test. Good luck.

14

@p33zy It would be useful if you would be specific as to what what you believe to be bad advice. In this thread the consistent and good advice is to stick with the official drivers if you are not comfortable dealing with potential dependency issues with newer 59x drivers.

Nothing is stopping anyone from experimenting and there are some good use cases where the newer drivers and/or CUDA versions are necessary for specific ML/DL workflows. But you should have a good snapshot restore strategy at the OS level, to make your life easier. Timeshift works well or rustic if you want to roll your own scripted backups and restores.

Then you can play around with different drivers, CUDA versions and upstream libs. I have had very few issues with the 580-open (official) for my dev needs. YMMV
dpkg -l|grep nvidia-driver

To see the signing info:
sudo dmesg | grep -iE ‘nvidia|secure|PKCS|signature’

It assumes SecureBoot is enabled.

15

I’m not too sure why it prompted for a key to be enrolled, perhaps when removing/adding packages to swap between the versions something got messed up. I can see after restoring firmware nvidia-driver-580-open is definitely correct.

Main thing which is bad advise is to install greater than 580. 595 looked initially to be working on two of the devices, but it bricked one (kernel panic straight after boot) and causes intermittent crashes on the others. Restoring from vendor firmware has sorted it though.

16

Driver 595 is currently not supported on Spark. Please stick with the 580 drivers provided by the update process