[WITHDRAWN] Qwen3.8-Flash-Next NVFP4 on 2x GB10: long-agent crash isolation and historical TP2 measurements

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10 Projects
1

WITHDRAWN (2026-09-01): do not deploy this exact NEXTN/EAGLE profile as a persistent service.

The profile originally described here passed bounded generation, retrieval, semantic, and disconnect tests, but later reproduced a separate invalid-probability crash under real multi-turn long-prefill traffic even with CUDA Graph and overlap scheduling disabled. The repository and v0.1.2-canary are now retained only as isolated failure-analysis assets.

Hi all,

This post documents the evolution—and eventual withdrawal—of a Qwen3.8-Flash-Next ModelOpt NVFP4 TP2 experiment on two GB10 systems. Historical throughput numbers remain useful as bounded measurements, but the serving profile is not service-stable.

Hardware and withdrawn profile

  • Two ASUS Ascent GX10 / NVIDIA GB10 (SM121), TP2 over direct ConnectX-7 RoCEv2
  • RadixArk/Qwen3.8-Flash-Next-NVFP4, native 262,144 context
  • Pinned SGLang Qwen3.8 development path
  • QSA enabled
  • NEXTN/EAGLE 3/1/4 with linear ReplaySSM spec enabled
  • Eager execution; CUDA Graph disabled
  • Overlap scheduling disabled
  • Radix cache disabled
  • FP32 Mamba state, Triton GDN
  • cuDNN dense FP4; FlashInfer CUTLASS MoE
  • Maximum two running requests

Failure history

1. Original failure

The original speed-oriented profile used decode CUDA Graph and overlap. Under a long workload both TP ranks detected invalid sampling probabilities, followed by CUDA device-side assert, Xid 43, TP/NCCL termination, and container restart. This was not OOM.

2. Bounded isolation

Controlled changes showed:

  • Dense CUTLASS was not necessary for the failure.
  • CUDA Graph candidates could silently corrupt 1K non-looping output and 25K marker retrieval while still returning HTTP 200.
  • Eager execution passed those bounded semantic prompts.
  • Disabling overlap crossed several earlier transition tests.
  • QSA draft-extend warmup required the focused field-compatibility fix in SGLang PR #37110.

Historical warmed 2,048-token measurements on the withdrawn eager profile were 41.60 / 42.39 / 42.86 tok/s (median 42.39). A 16,384-token continuous generation reached 48.01 tok/s. These are historical throughput observations, not stability qualification.

3. Streaming-disconnect lifecycle defect

A later client timeout removed tokenizer-side state while scheduler work continued, producing “state was deleted in TokenizerManager” before invalid probabilities and another rank crash. Backporting both commits from SGLang PR #36418 fixed this narrow zombie-request path:

  • Real TCP disconnects reclaimed work in 2.155 and 2.183 seconds.
  • Normal streams afterward completed.
  • A 28,672-token continuous generation completed in 543.56 seconds at 52.76 tok/s.
  • No restart occurred during that bounded validation.

4. Independent eager/no-disconnect recurrence

The lifecycle repair did not stabilize the overall stack. Later real multi-turn traffic reproduced the same service-level crash twice at different context lengths, approximately 29K and 50.6K. The latest incident had:

  • One running request and no queue
  • CUDA Graph disabled and overlap disabled
  • Radix cache disabled
  • Approximately 10% KV/token-pool use
  • Both containers OOMKilled=false
  • No client-disconnect or deleted-state precursor

Both ranks again reached invalid sampling probabilities, then CUDA assert, Xid 43, TP/NCCL termination, and restart. This disproves the earlier conclusion that eager/no-overlap plus the lifecycle fix was a correctness-qualified serving profile.

The sampler detected invalid probabilities; it is not proven to have created them. The unresolved suspect interaction is the repeated long-prefill to speculative-decode state path across NEXTN/EAGLE, ReplaySSM, recurrent GDN/Mamba state, QSA, and ModelOpt NVFP4.

Corrected conclusions

  • SGLang #36418 remains a valid narrow fix for streaming-disconnect zombie requests.
  • PR #37110 remains a valid narrow fix for QSA EAGLE draft-extend row sizing.
  • CUDA Graph remains rejected because it silently corrupted bounded semantic outputs.
  • The eager/no-overlap NEXTN/EAGLE profile is also withdrawn because it later crashed under multi-turn traffic.
  • Continuous-generation and retrieval tests are not sufficient substitutes for repeated 32K–100K prefill/decode/tool-turn transitions.
  • The exact root kernel is still unproven.

The repository launcher now fails closed unless ALLOW_WITHDRAWN_PROFILE=1 is set for isolated reproduction.

Public assets and upstream tracking

Reproductions on newer upstream images are welcome, but this exact profile should not be used as a production service.

2

Update: the disconnect fix remains valid, but the serving profile is withdrawn

A long streamed request exposed a real SGLang request-lifecycle defect: after the client timed out, tokenizer-side state was deleted while scheduler work continued. About two seconds after “state was deleted in TokenizerManager”, both ranks reached invalid probabilities, CUDA assert, Xid 43, TP/NCCL termination, and restart.

We backported both commits from SGLang PR #36418:

Targeted validation showed that real TCP disconnects reclaimed the request in 2.155 and 2.183 seconds, normal streams afterward completed, and a bounded 28,672-token continuous generation completed without a restart. This remains useful evidence for the narrow lifecycle defect addressed by #36418.

2026-09-01 stability correction

The disconnect repair did not make the complete profile stable. Later real multi-turn long-prefill traffic independently reproduced invalid sampling probabilities and dual-rank restart twice. The latest incident had one running request, no queue, CUDA Graph and overlap disabled, radix cache disabled, approximately 10% KV/token-pool use, OOMKilled=false on both containers, and no client-disconnect or deleted-state precursor.

Therefore v0.1.2-canary and the formerly retained eager/no-overlap NEXTN/EAGLE profile are withdrawn. PR #36418 still fixes the zombie-request lifecycle branch, but the separate speculative/numerical state-correctness branch remains open. A bounded continuous-generation pass was not sufficient coverage for repeated long prefill/decode/tool-turn transitions.

Withdrawal advisory:

Please do not treat the earlier reply as production qualification for the complete serving profile.

3

Your withdrawn profile may share a root cause with something we isolated on vLLM at TP4 across four Sparks. Same checkpoint, same traffic shape (repeated deep prefills with decode in between): around the third prompt past ~100k tokens a worker’s GPU stream stops advancing and the engine dies quietly. We eliminated, one variable per boot: prefix caching, MTP (it dies with speculative decoding fully off, which narrows your “speculative/numerical” suspicion), CUDA graphs, allocator settings, page cache, chunk size, and three different PLE paths. py-spy shows the ranks parked at the first stream sync point behind a wedged kernel: GDN, QSA or the EP all-to-all.

Matrix and stack: Qwen3.8-Flash-Next-Quad-DGX-Sparks/docs/OPEN-PROBLEMS.md at main · tsw2k/Qwen3.8-Flash-Next-Quad-DGX-Sparks · GitHub Also reported in the vLLM PR: [Model] Support Qwen3.8-Flash-Next by peakcrosser7 · Pull Request #53896 · vllm-project/vllm · GitHub

Two engines failing on one platform under one traffic shape smells like shared kernels or sm121 numerics rather than engine code. Happy to compare repros.