Long time reader, first time poster. ;)
Please note:
- I prompted most of the changes.
- I’m recalling perf numbers from memory, will revisit and get more final ones later.
AI Summary:
| Area | Result | ||
|---|---|---|---|
| Speculative decode, fast mode | Copy-heavy decode 22.45 → 29.75 t/s; one 61-draft block accepted whole | ||
| Speculative decode, correct mode | Loses at every depth, −14% to −25%; target-only stays default | ||
| n-gram lookup drafting | Copy-heavy 22.45 → 27.50 t/s (+22.5%); prose neutral at a strict gate | ||
| Q3 mixed-quant splice | NLL −5.79% vs Q2 (69/100 cases), −7% decode | ||
| MoE SSD streaming | Fetch-at-depth: shallow prefill +46.8%, deep +20.4%, byte-identical | ||
| RDMA expert tier | 1.75–1.79x over one NIC, byte-exact; remote-RAM promotion measured null | ||
| Auto-promotion to resident | Q2 TP2 decode 14.07 → 21.45 t/s (+52.4%) | ||
| Streaming penalty | Not disk: per-layer probe sync is 41.28 ms/token (58% of decode) |
Background:
All this work is based on antirez’s work.
First of all: the Q2 quant is fully resident on two sparks. :)
- I recall perf is actually the ‘best’ here, around 300 toks/s pp, and 20 toks/s tg.
anitrez added MOE streaming from SSD (based on all the NGRAM SSD work done for QWEN3.8 Flash Next). This means, you can actually run Q2 on ONE Spark as well.
- I recall perf around 200 tok/s pp, 10 tok/s tg.
Further, you can also run Q4 on ONE spark as well.
- I recall perf was also really bad: around 70 tok/s pp, and 5 tok/s tg.
- Still kinda crazy that this works at all though. :)
That is (I believe) where antirez stopped.
Here’s where I picked up and continued:
Q4 TP2 MOE Streaming
This was an easy swap in the code; however, performance didn’t improve that much. (200 tok/s pp, 6 tok/s tg?). Main issue is that if one box stalled on nvme read, the other box had to wait for it. Thus true parallelization didn’t happen. (Curious if there’s another way to do this… though).
DSpark / NGram Spec Decoding
antirez already did this on Mac, so I ported that to Spark / Cuda.
- I got Q2 TG2 up to around 400 tok/s 30 tok/s (for repetitive things).
- Prose didn’t improve still around 400 tok/s 21 tok/s.
- During this dev, it actually got slower, so we aren’t losing anything now.
- I kept pushing by agent, and it implemented a hybrid mode picking in memory NGram vs DSpark draft.
- For ref, I’m unsure if I need to push any data up for this, in order for you to reproduce.
MOE Streaming
I found the Expert streaming quite interesting, and did many experiments on this.
Short story: the Expert streaming blocks any GPU pipelining during TG. This stalls the pipeline due to needing to check if you have things there or not. I couldn’t find anyway to do this async, thus the GPU work would drain and start up again. This caused a large cliff in perf (1 ms per layer); I tried to do Q2 TG2 with streaming ‘on’ (letting one expert stream in if needed) and perf tanked by 25% roughly on both PP and TG. Just checking things are setup stalls a lot.
Even so, there were gains to be found:
I optimized Expert streaming SSD by using prefetch, batching, and running in threads. I believe I landed on 4 threads saturating the Gen5 NVME at 10GB/s.
- During PP, you have to load all MOE’s, so hinting and batching the load improves perf a bit.
Other notes:
- Generally any Expert needed is ‘randomly’ distributed. So prefetching is hard to predict.
- The ‘number of experts’ to keep in cache is actually cyclical over the past 256 tokens. Meaning, if you evict experts and kick them out, you pay performance penalty streaming them in again next token.
RDMA Streaming
For a while, I didn’t realize we were hitting a GPU drain, so I pushed further on ‘what if the SSD was faster?’ After all, based on numbers, 20GB/s on the ConnextX-7 NIC is faster than the NVME. The setup I prototyped with was: Use Spark #1 running Q2 MOE Streaming, and use Spark #2 as a ‘rambox’.
I had Claude try to setup #2 using NVMEoF to stream any weights first; simplest thing since this appears another ‘drive’ on Spark #1. With perf testing, I was able to transfer around 20GB/s via deep thread count (16 IIRC?); however, I found this no faster than NVME due to latency.
After that, I had Claude just build an RDMA service (committed to my repo) on Spark #2, backed against it’s copy of Q2 weights. This actually worked best, since the service could send directly from RAM and latency fell. This did improve perf by about 10% in streaming cases, though still significantly slower than just running Q2 TP2 fully resident.
I did not tset this with Q4 TP2 streaming, as I don’t have another 200 GBE card lying around to stream to both Sparks…
NGRAM Streaming
This was a small speed up. Gains are from implementing same strategy as Expert streaming: hint to drive, and do in batches / threads.
“Q3” Quant
Before the look into DSpark, I did look into other ways to ‘use up the RAM’. By splicing Q2 and Q4 gguf’s together, I had enough RAM to make the last 7 layers as Q4. This maps to 7 of 40 total layers, so perhaps this more like Q2.25. This was able to load in Q2 TP2, and be fully resident w/ 256k context. There were code changes needed for this, available in my repo. Performace was similar to Q2 TP2: around 350 tok/s pp, 20 tok/s tg. Not a huge hit, but slightly slower.
More details in repo’s readme.