5090 speeds on the m5 ultra is an incredible proposition that has so far flown under my radar. I may have to reconsider the gaming card for the speedy tandem teammate to my gb10. Added aspect of EXO combining is another intense proposition.
just trying it and so far it’s looking really good. Also what I have noticed is, that it does not consume so much memory as the other recipes, when running it I have still around 20GB available and I already also run embedding model
Today I tested the new Tensorfold engine and models–Vontra–Qwen3.8-Flash-Next-MLX-4bit-MTP. The engine itself is Very streamlined and minimalist compared to vLLM and does load fully in ~2 minutes.
(Qwen3.8 Flash Next on one DGX Spark with TensorFold · Mia's AI Lab)
GitHub - MiaAI-Lab/Qwen3.8-Flash-Next-Single-DGX-Spark-TensorFold: Qwen3.8 Flash Next on one DGX Spark (TensorFold) · GitHub
It seems the tool eval bench is not fully compatible with Benchy, but the endpoint v1/ was an accurate test. Real world usage on Openclaw and Herms wfeels a tick slower on just all factors> Here are the best side by side results I could get against the INT4 versiion from @azampatti Up to 70tok/s Qwen3.8-Flash-Next-Int4-AutoRound - #211 by jbourny
| Shape | TensorFold (MLX-4bit) | vLLM b12x (INT4-AutoRound) |
|---|---|---|
| real turns (12×512-tok) mean | 34.6 s/turn | 8.6 s/turn |
| real turns p50 / max | 10.3 s / 118 s | 9.4 s / 12 s |
| decode stalls | 5/12 (4–10 t/s) | 0/12 |
| clean-decode p50 | ~55 t/s | ~61 t/s |
| acceptance (counter delta) | 67.9% | 77.3% |
| micro c5 burst p50 e2e | 3.65 s / 5.2 s | 3.30 s / 3.50 s |
TF isn’t uniformly slower; it’s bimodal — near-parade (~55 t/s) when a decode graph path is hot, but 5/12 turns collapse into 4–10 t/s stalls in a memory-quiet state. b12x never stalls. That stall signature, not the eval’s +45% median, is what disqualifies it for production serving. (Still-unverified hypothesis: TF’s startup logged “0 decode graphs captured” — eager decode. If a future experiment ever reopens TF, that knob is the first thing to probe.)
Hey. Just finished setting this up, this fork of my recipe further removed the annoying 0.8s ttft from a prefix cached turn, which is amazing for my smart home use! Quality is also back to the autoround rock solid. My use case contains CJK output so I can’t use the mtp vocab cut speedup though. Thank you for the recommendation, and I stole it into my magi-V3 branch.
Saren-Arterius Is that you?
yep!!!