• Redisgned Multimodal MoE Model: 125B main model parameters, supplemented by an additional 51B N-gram embeddings,and 6B parameters activated per token.
• Comprehensive Architectural Upgrades: Pushing the frontiers of model architecture innovations, across the areas of Attention, Residual, Embedding, and Optimization— enhancing model capabilities.
• Efficient Training and Inference: Significantly reduces training and inference costs. At ~ 1/9th the training cost, Qwen3.8-Flash-Next achieves comparable capability against Qwen3.7-Plus, while being more capable in areas of coding and cowork.
Well I have to eat my hat - I was sure it would be a 397B replacement. They are targeting Mac 96-128GB, DGX Spark, Strix Halo 128GB users - makes total sense. And the structure looks very similar to Ling 3.0 (Kimi K3 Flash)
What is interesting - if the model turns to be on par with 27b, two sparks can run fp8 variant, which is always preferable with lots of kv cache. I doubt it has a chance to dethrone ds4f for me, but ram+speed advantage could be decisive for many. Especially if someone graft a dspark for it. Which will be tricky for multimodal checkpoint
I couldn’t find the source that verifies the size of Qwen3.8-Flash-Next. There are some random tweets claiming that it’s a 125B-A6B model, but I couldn’t find a reliable source backing them up.
I think the “Flash” model is more likely a 35B-A3B model. Below is from Qwen3.5-35B-A3B’s readme:
In particular, Qwen3.5-Flash is the hosted version corresponding to Qwen3.5-35B-A3B with more production features, e.g., 1M context length by default and official built-in tools.
Qwen3.6-35B-A3B was also referred to as Qwen3.6-Flash.
I could be wrong, though. In either case, it could be a very good model for a single Spark (though if it’s 125B, then NVFP4 quantization is a must).
I’ve done a quick research turn on this with a model, it couldn’t find verification on the web about the size. It did assume, that if OPs description is correct then a FP4 quant will fit on 1 GB10, and that FP8 will need 2 GB10 cluster.
125B + 51B, 6B active - this claim is still a rumor, but I think it does fall in-line with the “medium size model” that we’re expecting rather than the 35B, A3B.
Someone said llamacpp said it has 0 day support, meaning there is somewhere a pr lurking with hints on the model arch. Yet to find, maybe private until tomorrow
Alibaba’s Qwen team just teased Qwen4 today. Here’s what you need to know.
ModelScope posted a countdown for Qwen3.8-Flash-Next, an open-weight release built on the next-gen architecture that will power the upcoming Qwen4 model family. Alibaba says it’s shipping this architecture early so developers can prepare before Qwen4 itself lands.
Qwen3.8-Flash-Next is a multimodal MoE model. It pairs a large parameter count with a smaller active count per token, plus a huge n-gram embedding table for fast local token lookups, on top of GDN and QSA mechanisms.
Key numbers:
125B main-model parameters
51B additional N-gram embedding parameters
6B parameters active per token
Open weights expected within about a day of the announcement
No official benchmark numbers or exact release date have been published yet, this is still a preview ahead of the full Qwen4 family.
I sure do hope it’s at Least 512K of context window.
The real interesting part of this and the largest question mark: will the 51B be on the prefill or decode side of memory bus?
At one point I requested that NVIDIA rewrite an LLM, split up the model in this manner, and using some kind of websocket to run between the prefill and decode, removing the BW issue completely.
51B N-gram embeddings is a massive trainable lookup-memory for local token combinations.
The model takes the last few tokens, calculates the hash of their N-grams, fetches a few vectors from tables, and mixes them into the hidden state.
So the main model spends less effort reconstructing frequent local patterns.
Each token uses a tiny fraction of the lookup table, but all 51B parameters still need to be stored somewhere - in VRAM, RAM, or via a dedicated offload.
At ideal 4 bpw, the math goes like this:
125B main weights - 58.2 GiB
51B N-gram tables - 23.7 GiB
total - 82.0 GiB
On a spark, about 40 GiB will be left over. But that’s the best-case scenario. Real Q4 is usually heavier than four bits per parameter, and some quantizers leave embedding tables in FP16 or BF16 altogether.
So the main question of the release isn’t just the model size. We need to see the dtype of these 51B, the actual checkpoint, and support for quantization or offloading.
Since gb10 hit shelves, it’s seen Qwen Coder/Next which barely could keep up, then Qwen 3.5, 3.6 finally kept up with multi round looping cicd. Then DSv4F came out. I started with Gpt 3 quality, to Opus 4.8 now. Less than a year.
This is a wild season for late arrivals. Couldn’t be happier with DSv4F, but if this affords more context space on a single spark I may again hold off on twinning. Plus if the embeds pay off, offloading multi modality to my 5090 is finally done and I can recoup, make a profit, and go on vacation.
