Kernel crashes with Xid 13/31 when run under nsys profile — passes cleanly when run standalone

Developer Tools Nsight Systems Profiling Linux Targets
, ,
1

Issue:

A GPU kernel that launches other pre-instantiated CUDA graphs from device code runs without issue standalone but crashes with Xid 13/31 when profiled under nsys. The only difference between pass and fail is the presence of nsys profile instrumentation.

hardware description & driver info: check attached nvidia-smi.txt file

nsys --version
NVIDIA Nsight Systems version 2025.1.3.140-251335620677v0

CUDA Version: 12.9

OS: Linux

nvidia-smi.txt (1.7 KB)

reproduction source files: refer attached files device_graph_xid_repro_cu.txt main_cu.txt

device_graph_xid_repro_cu.txt (17.7 KB)

main_cu.txt (384 Bytes)

Note: This is a minimal standalone reproducer extracted from a larger real-time signal processing application that exhibits the same crash. The original application uses this same device-graph launch pattern at production scale; the reproducer strips it down to the essential structure that triggers the failure.

build command: (rename repro files to *.cu before executing below command)
nvcc -o xid_repro device_graph_xid_repro.cu main.cu -O3 -dopt=on --generate-line-info --generate-code arch=compute_89,code=sm_89 --forward-unknown-to-host-compiler -g -std=c++17 --default-stream per-thread -L/usr/local/cuda-12.9/lib64 -lcudart_static -lculibos -lcudadevrt -lnvidia-ml -L/usr/lib/x86_64-linux-gnu -lcuda

dmesg logs during the crash attached.

running stand-alone:
$ ./xid_repro

XidRepro: Iter=199000 TickNs=37000 PollNs=100000 Streams=8 Variants=8 Kernels/g=6
XidRepro: DevRuntimePendingLaunchCount requested=16384 actual=16384 status=0
XidRepro: IterDone=199000 FFLaunched=199000 FFCompleted=199000 PeakInFlight=4 LaunchErr=0
XidRepro: LastLaunched stream=7 variant=7 cudaDeviceSync status=0
XidRepro: Result=PASS

running with nsys:

$ nsys profile ./xid_repro
WARNING: Device-side CUDA Event completion trace is currently enabled.
This may increase runtime overhead and the likelihood of false
dependencies across CUDA Streams. If you wish to avoid this, please
disable the feature with --cuda-event-trace=false.
Collecting data…

XidRepro: Iter=199000 TickNs=37000 PollNs=100000 Streams=8 Variants=8 Kernels/g=6
XidRepro: DevRuntimePendingLaunchCount requested=16384 actual=16384 status=0

CUDA error 700 from <device_graph_xid_repro.cu> line 487. Continuing…
XidRepro: IterDone=102767 FFLaunched=99864 FFCompleted=99864 PeakInFlight=4 LaunchErr=0
XidRepro: LastLaunched stream=0 variant=7 cudaDeviceSync status=700
XidRepro: Result=FAIL
Generating ‘/tmp/nsys-report-2528.qdstrm’
[1/1] [========================100%] report3.nsys-rep
Generated:
/root/report3.nsys-rep

repro_exe_run_logs.txt (1.3 KB)

dmesg.txt (22.6 KB)

Please let me know if you need any details.

2

Can you update to the latest Nsys, Nsight Systems | NVIDIA Developer

we are at 2026.3.

@liuyis do you have suggestions?

3

The latest Nsys release is worth trying. In addtion, if possible, I recommend try with the latest CUDA driver (13.3, r610) together, because Nsys will use the latest CUPTI library (the library we rely on for CUDA trace feature) only when we have CUDA 13.x driver; and some fixes in the driver itself might also be related.

If the issue still persists with latest Nsys + latest driver, it will be helpful to experiment with adding a few Nsys CLI options and share the results, since the app has device-launched graphs:

  • --cuda-graph-trace=graph:host-only
  • --cuda-graph-trace=node
4

Hi @liuyis @hwilper

Thank you for your responses.

updated cuda toolkit and driver

here is the details from nvidia-smi

NVIDIA-SMI 595.45.04 Driver Version: 595.45.04 CUDA Version: 13.2

i built and run the same source files attached in this post.

logs:
[74237.707904] perf: interrupt took too long (4317 > 3962), lowering kernel.perf_event_max_sample_rate to 46250
[74237.708384] NVRM: Xid (PCI:0000:25:00): 13, Graphics Exception: SKEDCHECK22_INVALIDATE_ACTIVE_QMD failed
[74237.709350] NVRM: Xid (PCI:0000:25:00): 13, Graphics Exception: ESR 0x407020=0x20000000 0x407028=0x3f 0x40702c=0x206ef0a 0x407030=0x0
[74237.710401] NVRM: Xid (PCI:0000:25:00): 13, pid=17010, name=xid_repro, Graphics Exception: channel 0x00000002, Class 0000cec0, Offset 00000510, Data 0017e2ac
[74500.500898] NVRM: Xid (PCI:0000:25:00): 13, Graphics Exception: SKEDCHECK22_INVALIDATE_ACTIVE_QMD failed
[74500.501165] NVRM: Xid (PCI:0000:25:00): 13, Graphics Exception: ESR 0x407020=0x20000000 0x407028=0x1b00 0x40702c=0x0 0x407030=0x0
[74500.502152] NVRM: Xid (PCI:0000:25:00): 13, pid=17119, name=xid_repro, Graphics Exception: channel 0x00000002, Class 0000cec0, Offset 00000510, Data 0017e2ac
[74500.536328] NVRM: Xid (PCI:0000:25:00): 31, pid=17119, name=xid_repro, channel 0x00000002, intr 00000000. MMU Fault: ENGINE GRAPHICS HUBCLIENT_SKED faulted @ 0x0_00000000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_WRITE

I get the same behaviour when i start exe alone and if i start it with nsys.

5

Just to clarify - do you mean after updating to 595.45.04 driver, the app is failing even without Nsys?

6

yes. thats right.
stand alone app crashes with
Xid 13, Graphics Exception: SKEDCHECK22_INVALIDATE_ACTIVE_QMD failed

7

The error looks like some issue in the kernel mode driver. Unfortunately, we are not experts on that side, could you maybe forward it to CUDA - NVIDIA Developer Forums and get help there?

If the error for the standalone app is resolved, and there is still additional issue when running under Nsys only, we can debug that