I was testing PrismaQuant on various MoE models in the last couple of weeks with various degrees of success. So when PrismaAURA Qwen 3.6 27B was recently released, I set off again with the objective that PrismaAURA would be the baseline to beat, and what could I learn from the process. In order to do that, it would need to have better performance and better quality .
So how do I get better performance ? FP3, FP6, 2:4 sparsity , W4A16 comes to mind. All of which turns out to be a dead end for speed on the GX10 sm121a . I went so far as adding FP6 and 2:4 sparsity, encoders into the vLLM but once you are at 4-bit weight, there’s almost nothing left to win without paying in quality or hitting a hardware or kernel wall.
FP6 - the idea was simple, 6 bits weight should move fewer bytes than 8-bits. I should be able to replace some of the FP8 with FP6 instead and get some savings. The problem was that in order to move the data, it needed to be padded to full byte. Literally FP8, no savings in memory, only diskspace.
W4A16 - in theory, more accurate (16 bits activation) with less storage (4 bits++) , in practice, we hit into a hardware wall, the sm_121a tensor-core MMA is FP4xFP4 only. Which means it needs to be dequant to BF16, and that bring us back 1 full loop.
2:4 sparsity - this was chased the hardest, almost 3 days. I got this serving early but it was slower than dense mode, spend days chasing the crash, then performance. Finally got it to meet dense mode but barely manage to exceed. Did a Call Tools bench, and the drop in quality was simply not worth it.
More generally. Blackwell NVFP4 W4A4 is a perf unlock regardless of which Blackwell chip, just most important in the especially memory bandwidth constrained Spark.
Not sure if you are asking about the Quant or DiffusionGemma.
If about the Quant, each of them are behind a has slightly different issues but essentially the same wall . W4A16 is a hard wall , the tensor core MMA for sm121a only supports FP4xFP4. The path to FP16 is via emulation, basically a dequant process, and that kinds of defeats the purpose of W4A16. The emulation path basically results in added latency, and hence worst performance.
2:4 is memory sparsity, vs model sparsity. I was always curious about this as Nvidia cliam to 1 Pflops for DGX Spark was based on the use of 2:4 sparsity, and folks tends to confuse this with model sparse mode (MoE). But as I start to explore, the barrier is again the kernel, I went from being 4x slower than dense, to essentially performance parity with dense. I wanted to explore that further but decided to do a bench (tool calls) , the performance penalty was so bad, it just doesn’t make senses to chase it further. Basically their 1 Pflops shows nothing because accuracy didn’t meant anything in their bench…
Some of these will probably make senses on other (DC?) GPU but it just don’t make senses from DGX Spark perspective.
Based on what I’ve casually observed, as a general rule of thumb, quants that aren’t powers of 2 are slower than those that are. E.g. Q5 and Q6 are overall slower than Q8, but Q4 is faster. So Q5, Q6 etc only make sense if memory is too tight AND Q4 quality is too terrible. This is a narrow niche and IMO doesn’t make sense in most cases. NVFP4 should solve the quality issue well enough, though I don’t know off the top of my head how fast it is compared to Q8.
Yes, and that generally make senses from a computer perspective because computers are basically binary machines. But the issue is really about the memory bottleneck, and while there was a claimed 275Gb/s performance, most folks see a much lower 200Gb/s range in actual use. 4 bits isn’t too bad if you do it right and selectively, look at PrismaQuant. Memory tight or not, depends on the model that you want to run. Or rather we will always want to run a bigger model than we have bandwidth and memory for. Human nature.
The thing is not every quant that is called q3 or q5 means it has been quantized to 3 or 5 bits, it only means average result, could be 8 and 4 bits in specific proportion. Just like it can be 4.5 bits. It should be impossible yet it exists, because it is just average