RTX 5060 Blackwell + WSL2: Periodic 3.1s paravirt stall at exact 35.5s intervals (inference workload)

[yjh970706]

RTX 5060 Blackwell + WSL2: Periodic 3.1s paravirt stall at exact 35.5s intervals (inference workload)

TL;DR

• Hardware: NVIDIA GeForce RTX 5060 (consumer Blackwell)

• Environment: Windows 11 + WSL2 + Docker, ONNX Runtime + TensorRT inference

• Symptom: A stall of 3.1~3.8 seconds occurs at exactly 35.5-second intervals. The GPU itself keeps running; the paravirt layer halts the guest’s new requests.

• NVIDIA driver: 591.86 → 595.97 — nearly no change.

• WSL: 2.6.3.0 → 2.7.5.0 — stall duration reduced by ~17% (3.78s → 3.13s). However, the period and frequency remain unchanged.

• Request: Is NVIDIA aware of this regression? Are there reports of identical symptoms? Are there fix plans in upcoming driver/WSL updates?

---

1. Environment

Item	Value
GPU	NVIDIA GeForce RTX 5060 (8GB, consumer Blackwell)
CPU + motherboard	AMD Ryzen desktop
Windows	Windows 11 (build 26200.8246)
Secondary GPU	AMD Radeon iGPU (separate LUID, used by dwm/G2Client.exe)
WSL kernel	6.18.26.1-microsoft-standard-WSL2
WSL version	2.7.5.0 (preview channel)
NVIDIA driver	595.97 (WDDM mode)
CUDA	13.2 (host) / 12.9 (container)
Container	Ubuntu 24.04 in Docker, gRPC server with TensorRT + ONNX Runtime + Python 3.10
Workload	CCTV face/plate recognition, sustained load of ~5.3 fps plate + ~4.1 fps face
GPU process	The only user of RTX 5060 is vmwp.exe (WSL2 VM Worker)

2. Symptom

On an inference server running 24/7, all GPU requests stall for 3.1 seconds at exact 35.5-second intervals.

Measurement results:

Item	Value
Stall period	35.39s (deviation ±0.5s, CV < 1.5%)
Stall duration	Average 3.13s (range 3.03~3.21s, on WSL 2.7.5 + driver 595.97)
Normal latency before stall (Plate p50)	49ms
Normal latency before stall (Plate p99)	103ms
Max latency during stall (Plate)	3,044ms
Stall count in 30-min measurement	50 (matches the 35.5s period exactly)

The frequency and period of the stall remain identical after container restart, WSL restart, or OS reboot.

3. Mechanism narrowed by diagnostics

Through multi-layered measurement with strace, py-spy, ETW (Windows Performance Recorder), wpr GPU profile, and dxgkrnl event trace:

What stays alive during the stall

• The GPU itself: ETW dxgkrnl event rate stays identical to baseline during the stall (~120,000 events/sec). The GPU never stops.

• WSL guest Linux scheduling: Pure Python sleep, fork+exec, /proc reads all proceed normally during the stall.

• Existing ioctl calls to /dev/dxg: Respond within ≤ 23ms during the stall.

What halts during the stall

• All GPU clients inside the WSL guest: All worker threads in server.py block in futex_wait at the same instant for 3.7 seconds.

• Host’s nvidia-smi: nvidia-smi --query-gpu on the Windows host also hangs for 3.7 seconds at the same time.

• Launching a new nvidia-smi process: If you invoke strace nvidia-smi mid-stall, you get strace: Cannot find executable 'nvidia-smi'. The 9P filesystem mount at /usr/lib/wsl/lib/ becomes unreachable.

Conclusion (from measurements)

The stall does not happen at the dxgkrnl internal lock level — it happens at the Hyper-V VM bus / paravirt layer. The guest’s new GPU/9P requests are halted for about 3 seconds every 35.5 seconds, while in-flight GPU work and host-side dxgkrnl event processing continue normally.

4. Workarounds tried (all ineffective or marginal)

Attempt	Result
nvidia-smi -pm 1 (persistence mode already enabled)	No change
NVCP Power Management “Prefer Maximum Performance”	No change
NVCP Low Latency Mode On/Ultra	No change
GPU clock lock (nvidia-smi -lgc)	No change
GPU keepalive sidecar (cudaMalloc/Free every 300ms)	No change
OpenCV/ORT threadpool tuning	No change
Python GC disable	No change
Docker namespace bypass (--network host)	No change
gRPC keepalive tuning	No change
NVIDIA driver 591.86 → 595.97 update	Negligible (<1% change in stall duration)
WSL 2.6.3.0 → 2.7.5.0 update (preview channel)	~17% reduction in stall duration (3.78s → 3.13s). Period/frequency unchanged.

Only the WSL update produced meaningful improvement. The NVIDIA driver update had negligible effect.

5. Measurement comparison

Stage	Stalls (5min)	Stall duration	Plate p50 / p95 / p99 / max
Baseline (driver 591.86 + WSL 2.6.3)	9	3.74 ~ 3.86s	44 / 89 / 118 / 3,661 ms
After WSL 2.7.5	9	3.09 ~ 3.17s	46 / 88 / 106 / 3,045 ms
WSL 2.7.5 + driver 595.97	9	3.03 ~ 3.13s	49 / 87 / 103 / 3,044 ms

The same pattern reproduces consistently in 30-min long-run measurements (50 stalls, average period 35.39s).

6. Related cases from other users

I found unresolved reports with similar patterns on the NVIDIA Developer Forum:

• RTX PRO 4000/2000 Blackwell + Windows WDDM: 102ms periodic spike, did not occur on Ada Lovelace GPUs in the same PC (forum thread 357475)

• RTX 5060 Ti + WSL2 + Docker Desktop: nvlddmkm.sys DPC latency surge after 8 hours idle (forum thread 369079)

• RTX 5090 + vLLM + WSL2: CUDA graph capture crashes → fixed by WSL 2.7.0 preview (microsoft/WSL issue 14452)

• RTX PRO 6000 + WSL2: System hang → fixed by WSL 2.7.0 preview (Allen Kuo’s Medium post)

These cases do not have our exact 35.5s period, but all suggest a paravirt layer regression in Blackwell + Windows/WSL environments. The partial effectiveness of WSL 2.7.x updates is also a common thread.

7. User impact

In a 24/7 production environment:

• About 2,434 stalls per day (35.5s period × 24 hours)

• Cumulative stall time: about 2.1 hours

• About 124 frames per hour experience 3-second spike latency

• Significant impact on real-time inference workloads (CCTV, access control, etc.)

The GPU itself is fine, but because the paravirt layer is periodically halted, applications requiring real-time responsiveness face severe limitations.

8. Questions and requests

1. Is NVIDIA aware of this regression? Are there other reports of stall patterns occurring at a precise period (35.5s) on consumer Blackwell (RTX 5060/5070/5080/5090) + WSL2 + WDDM paravirt?

2. Are there driver-side fix plans? Neither 595.97 nor 596.49 (released 2026-05-12) release notes mention fixes directly related to our symptom. Can we expect paravirt-related fixes in upcoming driver releases?

3. Willing to provide additional workarounds or diagnostic data: I can provide our measurement data (strace logs, ETW traces, 30-min long-run measurements, etc.) to NVIDIA’s driver team. Please let me know what additional data would help with diagnosis.

4. Users with the same symptom: If any RTX 50 series users reading this experience stalls at an exact 35.5-second period with ~3-second duration, please comment. It is likely the same regression, and collecting cases together would help raise NVIDIA’s priority for this issue.

---

Available data (on request)

• 30-minute G probe log (precise timestamps of stall occurrence and duration)

• Server-side latency distribution (statistics from 10,400+ frames)

• ETW dxgkrnl trace analysis (event distribution within stall windows)

• strace output (nvidia-smi and server.py worker thread during stall)

• Syscall analysis at various stall moments