NVIDIA L40S 'falls off the bus' (Xid 79) after llama.cpp enters an endless ///// token loop at long context — reproduces on drivers 610.57.04 and 595

Accelerated Computing CUDA CUDA Programming and Performance
, , ,
1

Driver bug report: L40S drops off the bus after runaway “/////” generation (both 610.57.04 and 595.91.07)
Vendor: NVIDIA
Drivers tested: 610.57.04 and 595.91.07 (Linux x86_64)
Hardware: NVIDIA L40S, 46 GB, PCI 0000:01:00.0, GPU-633d3904-9e10-f67b-344e-2acb8e4b80c9

Summary
The two symptoms are causally linked, and both occur on driver 610.57.04 and
595.91.07:

  1. Infinite garbage output `/////` (llama.cpp bug, not a driver bug). Once the
    conversation context grows past ~25k-70k tokens, the model emits an endless
    stream of `/` (or `?`) characters at ~300 tok/s, never hitting EOS. This drives
    the GPU at 100% for minutes (tens of thousands of tokens). It is reported
    separately to llama.cpp .

  2. GPU “falls off the bus” — happens on BOTH drivers, AFTER the `/////` loop
    starts. The runaway token loop sustains 100% GPU compute; shortly afterwards
    the GPU drops off the PCI bus (`Xid 79` on 610.57.04, `Xid 69` channel error on
    595.91.07) and the driver escalates recovery (`Xid 154`). The crash is therefore
    triggered by the sustained full-GPU load of the buggy generation, not by a
    driver-only defect. On 610 recovery required “OS Reboot”; on 595 the card resets
    but recovers without a full reboot. A 235 W power cap reduced crash frequency but
    did not prevent it.

The sequence is consistently:long context → `/////` garbage → GPU falls off the
bus (both drivers).

Environment

  • OS: Debian 13 (trixie), kernel 6.12.73+deb13-amd64
  • GPU: NVIDIA L40S, 46 GB
  • Driver: 610.57.04 (from developer.download.nvidia.com cuda/debian13 repo)
  • CUDA toolkit: 13.3
  • Workload: llama.cpp (Unsloth prebuilt b10472) - GGUF inference, `-ngl 1`,
    `–flash-attn on`, `-c` up to 262144, plus OpenGL/WebKit desktop compositor
    (GNOME/Wayland) sharing the same GPU

Symptom 1 - `/////` garbage

  • Happens with Qwen3 GGUF models (`Qwen3-Coder-30B-A3B-Instruct-Q6_K`,
    `Qwen3.8-27B-Q6_K`) at large context (>~25k tokens).
  • Reproduced identically on 610.57.04 and 595.91.07, with and without
    speculative decoding, with no CUDA errors.
  • Reported separately to llama.cpp.
  • Consequence: this runaway loop holds the GPU at 100% and is the immediate
    precursor of the crash below (Symptom 2) on both driver versions.

Symptom 2 - “GPU has fallen off the bus” (BOTH 610.57.04 and 595.91.07, after the `/////` loop)
Observed events (dmesg) - first crash on 610.57.04, occurring right after the
model entered the `/////` loop (during long-context generation):
NVRM: Xid (PCI:0000:01:00): 69, pid=…, name=eadedCompositor, Class Error: channel 0x00000002, Class 0000c997, Offset 00000364, Data ffffffff, ErrorCode 0000000c
NVRM: Xid (PCI:0000:01:00): 79, GPU has fallen off the bus.
NVRM: GPU 0000:01:00.0: GPU has fallen off the bus.
NVRM: A GPU crash dump has been created. … run nvidia-bug-report.sh
NVRM: Xid (PCI:0000:01:00): 154, GPU recovery action changed from 0x0 (None) to 0x2 (OS Reboot)

Subsequent crashes also produced `RC_TRIGGERED` (ring controller reset) and
`Xid 154 … to 0x1 (PF FLR)`. After a FLR, `nvidia-smi` reported
`power.limit [N/A]` and showed ~42 GB VRAM “in use” with `No running processes
found` - i.e. the driver’s memory accounting was left inconsistent.

A CUDA-side failure was also observed in the workload:
ggml-cuda.cu: CUDA error: unspecified launch failure
current device: 0, in function ggml_backend_cuda_buffer_set_tensor

Impact

  • On 610.57.04: full GPU loss requiring OS reboot (`Xid 79` + `Xid 154 → 0x2`).
  • On 595.91.07: GPU resets (`Xid 69`) after the `/////` loop;
  • After a soft reset (FLR), `nvidia-smi` reported `power.limit [N/A]` and ~42 GB
    VRAM “in use” with `No running processes found` - the driver’s memory accounting
    was left inconsistent.

Workaround / what was tried

  • 235 W power cap (`nvidia-smi -pl 235`) + clock cap (`-lgc 1150,1300`) reduced
    crash frequency but did not stop it (the `/////` loop still destabilizes the GPU).
  • Downgrading 610.57.04 → 595.91.07 changed the failure mode from “OS Reboot” to a
    recoverable reset, but the GPU still drops under the sustained `/////` load.
  • The real fix is to stop the `/////` loop (llama.cpp); the GPU crash is
    its downstream consequence.

Request

  • Investigate why the L40S drops off the bus / resets under sustained 100% CUDA
    compute (the runaway token loop from llama.cpp) on both 610.57.04 and 595.91.07.
  • Confirm whether this is expected behavior for a compute workload that never
    yields, or whether there is a driver/thermal/power protection that can recover
    more gracefully (without VRAM-accounting corruption after FLR).
  • (For context only, not a driver request: the `/////` garbage itself is a
    llama.cpp bug, reproduced on both drivers.)
2

That runaway generation is a good repro case. If you can share the exact llama.cpp launch command and one prompt/context that triggers it, I can put it through a free preflight and give you a command to test the same workload in a clean GPU environment.