I’ve built an open-source LLM inference telemetry suite that measures
Tokens Per Joule alongside standard throughput metrics. The NVIDIA
backend uses pynvml for real-time polling of board power (mW → W),
temperature, VRAM utilization, and SM clock speed at 500ms intervals
via a daemon thread.
Reference baseline on RTX 3080 (10GB GDDR6X):
- Qwen-3B Q4_K_M: 0.90 T/J (~198 TPS, ~220W board power)
- Llama-3.1-8B Q8_0 @ 8192 ctx: OOM (needs ~13.7GB)
The NvidiaProvider reads power via nvmlDeviceGetPowerUsage, temperature
via nvmlDeviceGetTemperature, and clocks via
nvmlDeviceGetClockInfo(NVML_CLOCK_SM). All telemetry is logged to
thermal_log.csv with 10-run averaging and 95% confidence intervals.
I’d like to understand how Blackwell’s memory architecture affects
the T/J curve for large-context Q8_0 workloads that hit the VRAM
ceiling on 10GB cards. If you have RTX 5090 or B200 hardware and
15 minutes, the suite auto-detects NVIDIA GPUs and requires no code
changes.
Setup: clone, venv, pip install, python src/orchestrator.py