The largest binary/ternary quantization attempt so far. Will be interesting to see its quality and performance.
It looks very impressive for its size. Not much use on a spark as we can obviously do much better but for a laptop or even a phone it’s very viable.
There are some benchmarks and memory usage charts here.
There could still be some merit on a Spark. 27B dense is naturally memory bandwidth limited on a Spark for token generation, while its prompt processing speed is quite fast. Bonsai results in a much smaller model memory footprint at 94% the quality, prompt processing should be unaffected but token generation should be significantly faster, especially including MTP.
Yes Spark has a lot of memory but there could be instances where you want to run a small model besides other workflows/models which leave less memory without giving up too much quality. I look forward to test how well this one benchmarks speed wise and quality wise on the Spark compared against my daily driver (Qwen 3.6 27B PrismaScout) and compared against say Qwen 9B which is roughly the same footprint.
exactly the performance boost you get from a smaller footprint could definitely be worth the payoff.
Just gave it a try on my Spark with a recompiled build of llamacpp (prism-b9594-38c66ad) from the PrismML github.
The results from llamabenchy, for the 27B by itself using the following command to spin up the server :
BONSAI_KV4=1 ./build/bin/llama-server \
-m models/Bonsai-27B-Q1_0.gguf \
--spec-type ngram-mod \
-ngl 99 \
-fa 1 \
-c 65536 \
-np 1 --host 0.0.0.0 --port 8080
llama-benchy args :
uvx llama-benchy \
--base-url http://0.0.0.0:8080/v1 \
--model prism-ml/Bonsai-27B-gguf \
--depth 0 4096 8192 16384 32768 \
--latency-mode generation
llama-benchy results with a simple ngram-mod speculative decoding :
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:-------------------------|----------------:|--------------:|-------------:|------------------:|------------------:|------------------:|
| prism-ml/Bonsai-27B-gguf | pp2048 | 950.05 ± 9.32 | | 2105.73 ± 55.31 | 1996.51 ± 55.31 | 2105.73 ± 55.31 |
| prism-ml/Bonsai-27B-gguf | tg32 | 43.19 ± 0.24 | 44.58 ± 0.25 | | | |
| prism-ml/Bonsai-27B-gguf | pp2048 @ d4096 | 961.87 ± 4.54 | | 5999.25 ± 83.72 | 5890.02 ± 83.72 | 5999.25 ± 83.72 |
| prism-ml/Bonsai-27B-gguf | tg32 @ d4096 | 41.43 ± 0.08 | 42.77 ± 0.09 | | | |
| prism-ml/Bonsai-27B-gguf | pp2048 @ d8192 | 957.66 ± 1.53 | | 9819.39 ± 14.34 | 9710.17 ± 14.34 | 9819.39 ± 14.34 |
| prism-ml/Bonsai-27B-gguf | tg32 @ d8192 | 38.82 ± 0.01 | 40.08 ± 0.01 | | | |
| prism-ml/Bonsai-27B-gguf | pp2048 @ d16384 | 933.31 ± 0.71 | | 17829.32 ± 221.39 | 17720.10 ± 221.39 | 17829.32 ± 221.39 |
| prism-ml/Bonsai-27B-gguf | tg32 @ d16384 | 35.80 ± 0.15 | 36.96 ± 0.15 | | | |
| prism-ml/Bonsai-27B-gguf | pp2048 @ d32768 | 893.20 ± 1.60 | | 35461.32 ± 252.71 | 35352.09 ± 252.71 | 35461.32 ± 252.71 |
| prism-ml/Bonsai-27B-gguf | tg32 @ d32768 | 30.56 ± 0.11 | 31.00 ± 0.00 | | | |
With Dspark enabled (Generates a few errors since the Fused Gate Delta Net for Autroregressive is not supported on this llamacpp build for sm121a)
running with the following command :
BONSAI_KV4=1 ./build/bin/llama-server \
-m models/Bonsai-27B-Q1_0.gguf \
--model-draft models/Bonsai-27B-dspark-Q4_1.gguf \
--spec-type draft-dspark \
-ngl 99 -fa 1 -c 65536 -np 1 \
--spec-draft-n-max 4 \
--host 0.0.0.0 --port 8080
and the llama-benchy args :
uvx llama-benchy \
--base-url http://0.0.0.0:8080/v1 \
--model prism-ml/Bonsai-27B-gguf \
--depth 0 4096 8192 16384 32768 \
--latency-mode generation
llama-benchy results :
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:-------------------------|----------------:|--------------:|-------------:|------------------:|------------------:|------------------:|
| prism-ml/Bonsai-27B-gguf | pp2048 | 482.90 ± 4.65 | | 4071.12 ± 51.54 | 3794.83 ± 51.54 | 4071.12 ± 51.54 |
| prism-ml/Bonsai-27B-gguf | tg32 | 29.45 ± 2.94 | 30.19 ± 3.21 | | | |
| prism-ml/Bonsai-27B-gguf | pp2048 @ d4096 | 478.93 ± 1.24 | | 11907.52 ± 137.78 | 11631.23 ± 137.78 | 11907.52 ± 137.78 |
| prism-ml/Bonsai-27B-gguf | tg32 @ d4096 | 24.48 ± 0.73 | 25.00 ± 0.82 | | | |
| prism-ml/Bonsai-27B-gguf | pp2048 @ d8192 | 480.37 ± 1.93 | | 19596.02 ± 436.15 | 19319.73 ± 436.15 | 19596.02 ± 436.15 |
| prism-ml/Bonsai-27B-gguf | tg32 @ d8192 | 16.18 ± 0.25 | 16.67 ± 0.47 | | | |
| prism-ml/Bonsai-27B-gguf | pp2048 @ d16384 | 474.10 ± 2.05 | | 35612.38 ± 301.72 | 35336.09 ± 301.72 | 35612.38 ± 301.72 |
| prism-ml/Bonsai-27B-gguf | tg32 @ d16384 | 6.79 ± 0.53 | 7.33 ± 0.47 | | | |
| prism-ml/Bonsai-27B-gguf | pp2048 @ d32768 | 463.26 ± 1.76 | | 68157.25 ± 609.63 | 67880.96 ± 609.63 | 68157.25 ± 609.63 |
| prism-ml/Bonsai-27B-gguf | tg32 @ d32768 | 3.61 ± 0.17 | 4.00 ± 0.00 |
It is certainly functional. I would say binary (1-bit) from prism is comparable to regular quality-made 2-bit quant, but binary is not really half the size of 2 bit, smaller but not massively. Good solution for someone with smaller cards.
Then you run Qwen 3.6 35B in FP8 - will be fast and I can guarantee it will be much higher quality output that 1 or 2-bit quantized 27b. Prism or not.
BTW interesting observation - 1bit Prism 27b actually generates overall higher quality output than 2bit. But it is extremely verbose in thinking - because of massive quantization its not sure about anything, it keeps brainstorming simplest questions. But overall result is decent. 2bit variant is lot let verbose - it thinks it knows the answers but they are often wrong. It does not feel affected, like 1-bit does and keeps missing the mark.
I ran for comparison 3 models on 5070ti with 16GB ram:
- Binary Prism 27b
- Ternary Prism 27b
- Gemma4 12b QAT (4bits)
They are all roughly same size, plus or minus 1-1.5GB. Not big difference. All can fit in 16 or 12GB with cache to spare. But in speed and quality Gemma4 was running around prisms, beating massively on speed and confidently in quality.
Prism models seems to be targeting mobile devices (what they state) and older/lower grade cards with 6/8 GB VRAM. Then it makes total sense.