Up to 70tok/s Qwen3.8-Flash-Next-Int4-AutoRound

[azampatti]

Just like I did with my Faster Qwen3.5 122B, I’m basically doing the same with this great awesome new model from Qwen.

Single-line command:

git clone https://github.com/azampatti/Qwen3.8-Flash-Next-Int4-FAST.git && cd Qwen3.8-Flash-Next-Int4-FAST && bash setup.sh

Repos:

• https://github.com/azampatti/Qwen3.8-Flash-Next-Int4-FAST/blob/main/README.md
• azampatti/Qwen3.8-Flash-Next-125B-A5B-INT4-AutoRound · Hugging Face

Speed:
Of course ‘it depends’ but in my tests, between 60-70 tok/s in coding.

Context:

From vLLM’s launcher: GPU KV cache size: 644,732 tokens, Maximum concurrency for 262,144 tokens per request: 2.46x

But HOW?

I halved how many experts the model uses per token, from 10 to 5, then taught its shared experts to make up the difference by imitating the uncut model. Nothing was removed from the model. All 512 experts are still there, I just changed how many of them each token is routed to, which cuts active parameters from 6B to 4.8B and makes it faster, then healed the quality loss by training one small always-on component, 37.75M parameters with everything else frozen, to match the original’s output distribution and quality as close as possible.

MTP Draft head stays in BF16 and Topk10 to predict at the best possible quality with a tiny bit of speed penalty

Try it, enjoy and please provide me with feedback! I’m still improving this model every day, so I will be updating this thread when an improvement is posted.

What’s Next?

• I’m testing new containers as more talented people are working on them. That’s why I wanted to push this model through HuggignFace as well Just make sure you also pay attention to the launch parameters I use to ensure the best performance.
• Quality should be within 2% of the original 4bits models (both NVFP4 and INT4 I tried scored closely from each other), but I’m testing avenues to raise the quality even further like I was able to do with 122B. It’s MUCH harder with this model though , but I’m on it!
• Bookmark this thread, hopefully I can do at least weekly updates with improvements for the next couple of weeks. The idea would be to re-run the one-liner and that would check and update everything.

-Aldo

EDIT: IMPORTANT - The default chat template for this model thinks in EXTRA HIGH. I attached two slightly modified chat templates that are xhigh and medium but with some tweaks to attack some scenarios for stronger coding in OpenCode. I did not included them by default to avoid “cheating” on comparisons, but consider using the “medium” chat template, it’s faster and stronger, but benchmark it on default.

EDIT2: Up to 70tok/s Qwen3.8-Flash-Next-Int4-AutoRound - #13 by ds2atc @ds2atc benchmarked with tool-eval-bench and I’ve just personally validated in my DGX that this results are accurate.

[stever519]

I’m looking forward go giving it a try!

Does this one include vision support?

[azampatti]

I did not stipe anything from its base.

[SVJuno]

Heyo! Very interesting!

I’ve been working on my own Frankenstein build, looks like the following : stock top-10 Flash-Next (RadixArk NVFP4, blazux vLLM fork, MTP-2, full decode CUDA graphs, resident 12 GiB hashed PLE table, bf16 KV at 495k tokens) on a single Spark, run through your sgbench harness, same prompts.

I’m curious what I can pull from you, and will most likely just test 5 experts (not tuned) just to see what the decode does.

I’m curious about a few things:

• For the 57 and the 67 to 70, what are the engine steps/s and the mean accepted tokens per step? That separates the routing gain from acceptance.
  • Would you run the public sgbench against your recent model? That gives everyone a same-prompt comparison.
• Would you share the healed expert tensors as a standalone bf16 delta, and the distillation script? Those 38M parameters should transplant into other quants of the same base, which would let NVFP4 users test the heal without redoing the training.

[azampatti]

@SVJuno

I’m curious what I can pull from you, and will most likely just test 5 experts (not tuned) just to see what the decode does.

I’m curious about a few things:

• For the 57 and the 67 to 70, what are the engine steps/s and the mean accepted tokens per step? That separates the routing gain from acceptance.
  • Would you run the public sgbench against your recent model? That gives everyone a same-prompt comparison.
• Would you share the healed expert tensors as a standalone bf16 delta, and the distillation script? Those 38M parameters should transplant into other quants of the same base, which would let NVFP4 users test the heal without redoing the training.

I’ll run it from you when I finish a few things on the DGX, but let me answer a few questions:

• Yes, you can definitely launch the models forcing Topk=5 in the config to measure speed and it’ll give you a pretty good idea on what will you expect after healing. BUT, MTP should run in K10 for proper function of the drafter and also, once you ‘heal it’ you’ll get a bit more acceptance but also slower decoding speed. So the test will give you a good idea +/- margin of error.
• I don’t have recorded the engine speed and I’m working on the MTP head now, I’m trying to improve it further.
• I can definitely share the BF16 shared experts, but probably after another pass or two. I’m trying to improve the quality a bit more but each additional % points takes multiple days :)
• Finally, We should all measure with “thinking off” for proper decoding speed (Which I haven’t here because is only fair for people to compare apples to apples). This model and its default chat template likes to think A LOT, and that “prose” text is SLOOOWW to draft from the MTP, so your actual perceived speed and benchmarks will suffer from it).

[azampatti]

Look at this acceptance-per-category chart:

┌──────────┬──────┬────────────────────────────────┬
│ category │ rows │   standard acc /               │
├──────────┼──────┼────────────────────────────────┼
│ CODE     │ 15   │ 82.1% /             │
├──────────┼──────┼────────────────────────────────┼
│ MATH     │ 12   │ 76.8% /             │
├──────────┼──────┼────────────────────────────────┼
│ LONGCTX  │ 6    │ 85.5% /             │
├──────────┼──────┼────────────────────────────────┼
│ TOOL     │ 12   │ 68.9% /             │
├──────────┼──────┼────────────────────────────────┼
│ GENERAL  │ 15   │ 55.7% /             │

You can clearly see how it drops heavily in “General” which dominates thinking blocks. But when writing code, it just flies over it

EDIT:

@SVJuno

• Would you run the public sgbench against your recent model? That gives everyone a same-prompt comparison.

Here:

[Q&A] 204 tokens in 3.79s = 53.8 tok/s (prompt: 23)
[Code] 512 tokens in 8.43s = 60.7 tok/s (prompt: 30)
[JSON] 866 tokens in 12.19s = 71.0 tok/s (prompt: 48)
[Math] 64 tokens in .96s = 66.6 tok/s (prompt: 29)
[LongCode] 2048 tokens in 33.27s = 61.5 tok/s (prompt: 37)

And here’s is the same with thinking disabled:

[Q&A] 256 tokens in 4.43s = 57.7 tok/s (prompt: 25)
[Code] 257 tokens in 4.00s = 64.2 tok/s (prompt: 32)
[JSON] 757 tokens in 10.67s = 70.9 tok/s (prompt: 50)
[Math] 9 tokens in .31s = 59.0 tok/s (prompt: 31)
[LongCode] 2048 tokens in 29.08s = 70.4 tok/s (prompt: 39)

You can see how much Code gets penalized by the thinking blocks being slow (the “general” acceptance on my previous post)

[siertum]

Maybe it’s worth including the Tool Eval Bench results for the latest version?
It would be interesting to see the results with a single thread (--parallel 1).

[azampatti]

I have been running 2.3.1 for a while and haven’t updated it to the latest on purpose for A/B testing. I will once I finish doing everything. Tool-eval-bench has been helping me a lot during this as it’s one of the tools I use to analyze behavior (not just score, but also tokens, times and responses).

My median score in 2.3.1 for this model is 86/100 with a +/- 2 variance.

[ds2atc]

╭─────────────────────────── 🏆 Benchmark Complete ────────────────────────────╮
│                                                                              │
│    Model:  vllm                                                              │
│    Score:  94 / 100                                                          │
│    Rating: ★★★★★ Excellent                                                   │
│    Benchmark: tool-eval-bench v2.6.1.dev65+g6be685f0e                        │
│    Engine:       vLLM 0.1.dev20073+g8e685d198                                │
│    Quantization: INT4-AutoRound                                              │
│    Max context:  262,144 tokens                                              │
│    Model root:                                                               │
│  /home/dolidoli/.cache/huggingface/hub/models--azampatti--Qwen3.8-Flash-Nex  │
│  t-125B-A5B-INT4-AutoRound/snapshots/0deb648024edc960900ffef22569eca655b37b  │
│  e5                                                                          │
│                                                                              │
│    ✅ 63 passed   ⚠️  4 partial   ❌ 2 failed                                │
│    Points: 130/138                                                           │
│                                                                              │
│    Quality:        94/100                                                    │
│    Responsiveness: 44/100  (median turn: 3.5s)                               │
│    Deployability:  79/100  (α=0.7)                                           │
│    Weakest: H Instruction Following (80%)                                    │
│                                                                              │
│    Completed in 1048.4s                                                      │
│                                                                              │
│    📊 Token Usage:                                                           │
│    Total: 525,607 tokens  │  Efficiency: 0.2 pts/1K tokens                   │
│                                                                              │
│    ── How this score is calculated ──                                        │
│    • Each scenario: pass=2pt, partial=1pt, fail=0pt                          │
│    • Category %: earned / max per category                                   │
│    • Final score: (total points / max points) × 100                          │
│    • Deployability: 0.7×quality + 0.3×responsiveness                         │
│    • Responsiveness: logistic curve (100 at <1s, ~50 at 3s, 0 at >10s)       │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯

                               Category Breakdown                               
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Category                  ┃   Score   ┃ Bar                       ┃  Earned  ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Tool Selection            │   100%    │ ████████████████████      │   6/6    │
│ Parameter Precision       │   100%    │ ████████████████████      │   6/6    │
│ Multi-Step Chains         │   100%    │ ████████████████████      │   8/8    │
│ Restraint & Refusal       │   100%    │ ████████████████████      │   6/6    │
│ Error Recovery            │   100%    │ ████████████████████      │   6/6    │
│ Localization              │   100%    │ ████████████████████      │   6/6    │
│ Structured Reasoning      │    83%    │ ████████████████░░░░      │   5/6    │
│ Instruction Following     │    80%    │ ████████████████░░░░      │   8/10   │
│ Context & State           │    95%    │ ███████████████████░      │  19/20   │
│ Code Patterns             │   100%    │ ████████████████████      │   6/6    │
│ Safety & Boundaries       │    96%    │ ███████████████████░      │  25/26   │
│ Toolset Scale             │   100%    │ ████████████████████      │   8/8    │
│ Autonomous Planning       │    83%    │ ████████████████░░░░      │   5/6    │
│ Creative Composition      │   100%    │ ████████████████████      │   6/6    │
│ Structured Output         │    83%    │ ████████████████░░░░      │  10/12   │
└───────────────────────────┴───────────┴───────────────────────────┴──────────┘

[azampatti]

What launch parameters did you use? I haven’t yet tested 2.6.1 myself but yours is pretty cool already. Do a “–trials 3” or 5 if you can and check the variance from run to run, I like that average better as a datapoint :)

[ds2atc]

I just asked openhands to do the test. The command was as below(it used fixed seed:
tool-eval-bench --base-url ``http://192.168.1.200:8000`` --model vllm --backend vllm --api-key no --seed 42

and hardmode is(just --hardmode added with same seed):

╭─────────────────────────── 🏆 Benchmark Complete ────────────────────────────╮
│                                                                              │
│    Model:  vllm                                                              │
│    Score:  91 / 100                                                          │
│    Rating: ★★★★★ Excellent                                                   │
│    Benchmark: tool-eval-bench v2.6.1.dev65+g6be685f0e                        │
│    Engine:       vLLM 0.1.dev20073+g8e685d198                                │
│    Quantization: INT4-AutoRound                                              │
│    Max context:  262,144 tokens                                              │
│    Model root:                                                               │
│  /home/dolidoli/.cache/huggingface/hub/models--azampatti--Qwen3.8-Flash-Nex  │
│  t-125B-A5B-INT4-AutoRound/snapshots/0deb648024edc960900ffef22569eca655b37b  │
│  e5                                                                          │
│                                                                              │
│    ✅ 76 passed   ⚠️  8 partial   ❌ 4 failed                                │
│    Points: 160/176                                                           │
│                                                                              │
│    Quality:        91/100                                                    │
│    Responsiveness: 48/100  (median turn: 3.2s)                               │
│    Deployability:  78/100  (α=0.7)                                           │
│    Weakest: K Safety & Boundaries (77%)                                      │
│                                                                              │
│    Completed in 1288.7s                                                      │
│                                                                              │
│    📊 Token Usage:                                                           │
│    Total: 649,066 tokens  │  Efficiency: 0.2 pts/1K tokens                   │
│                                                                              │
│    ── How this score is calculated ──                                        │
│    • Each scenario: pass=2pt, partial=1pt, fail=0pt                          │
│    • Category %: earned / max per category                                   │
│    • Final score: (total points / max points) × 100                          │
│    • Deployability: 0.7×quality + 0.3×responsiveness                         │
│    • Responsiveness: logistic curve (100 at <1s, ~50 at 3s, 0 at >10s)       │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯

I will test -trials 3 or 5 without the seed.

[azampatti]

Dang! 91 in hard-mode it’s nice! I’ll take it :)

I have ~37 more hours of GPU training to go, I’ll update all my tools when I finish.

[ds2atc]

normal: 93 / 93 / 91 / 93 / 93

hard: 89 / 90 / 88 / 89 / 92

╭──────────────────────────── 📊 Trial Statistics ─────────────────────────────╮
│                                                                              │
│    Trials:  5                                                                │
│    Score:   92.6 ± 0.9 / 100                                                 │
│    Median:  93.0                                                             │
│    95% CI:  [91.8, 93.0]                                                     │
│    Points:  128.2 ± 1.8                                                      │
│                                                                              │
│    Pass@5:  98.6%  (capability ceiling)                                      │
│    Pass^5:  79.7%  (reliability floor)                                       │
│    ⚠ Gap:    18.9pp  (high variance — consistency issue)                     │
│                                                                              │
│    Categories with variance:                                                 │
│      E Error Recovery: 97% ± 7.6%                                            │
│      F Localization: 93% ± 14.8%                                             │
│      H Instruction Following: 84% ± 8.9%                                     │
│      I Context & State: 92% ± 9.7%                                           │
│      K Safety & Boundaries: 92% ± 6.7%                                       │
│      N Creative Composition: 97% ± 7.6%                                      │
│      O Structured Output: 86% ± 7.6%                                         │
│                                                                              │
│    ⚡ 13 unstable scenario(s):                                               │
│      TC-14: 1.8 ± 0.5  (2,2,1,2,2)                                           │
│      TC-18: 1.6 ± 0.9  (2,2,0,2,2)                                           │
│      TC-20: 1.8 ± 0.5  (2,1,2,2,2)                                           │
│      TC-21: 1.2 ± 0.5  (1,2,1,1,1)                                           │
│      TC-33: 1.2 ± 1.1  (2,2,0,2,0)                                           │
│      TC-35: 1.4 ± 0.6  (1,2,1,2,1)                                           │
│      TC-43: 1.2 ± 0.5  (1,2,1,1,1)                                           │
│      TC-45: 0.4 ± 0.9  (0,0,0,0,2)                                           │
│      TC-48: 1.6 ± 0.9  (2,0,2,2,2)                                           │
│      TC-50: 1.2 ± 0.5  (1,1,2,1,1)                                           │
│      TC-55: 1.8 ± 0.5  (2,2,2,1,2)                                           │
│      TC-63: 1.6 ± 0.9  (2,0,2,2,2)                                           │
│      TC-68: 0.4 ± 0.9  (0,2,0,0,0)                                           │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯

╭──────────────────────────── 📊 Trial Statistics ─────────────────────────────╮
│                                                                              │
│    Trials:  5                                                                │
│    Score:   89.6 ± 1.5 / 100                                                 │
│    Median:  89.0                                                             │
│    95% CI:  [88.6, 90.8]                                                     │
│    Points:  157.8 ± 2.8                                                      │
│                                                                              │
│    Pass@5:  95.5%  (capability ceiling)                                      │
│    Pass^5:  73.9%  (reliability floor)                                       │
│    ⚠ Gap:    21.6pp  (high variance — consistency issue)                     │
│                                                                              │
│    Categories with variance:                                                 │
│      E Error Recovery: 90% ± 9.3%                                            │
│      F Localization: 87% ± 18.1%                                             │
│      G Structured Reasoning: 80% ± 7.2%                                      │
│      I Context & State: 91% ± 5.5%                                           │
│      K Safety & Boundaries: 82% ± 4.9%                                       │
│      L Toolset Scale: 98% ± 5.4%                                             │
│      M Autonomous Planning: 73% ± 8.8%                                       │
│      N Creative Composition: 93% ± 9.3%                                      │
│      P Hard Mode: 93% ± 3.3%                                                 │
│                                                                              │
│    ⚡ 20 unstable scenario(s):                                               │
│      TC-14: 1.4 ± 0.6  (2,1,1,1,2)                                           │
│      TC-18: 1.2 ± 1.1  (0,2,0,2,2)                                           │
│      TC-21: 0.8 ± 0.5  (1,1,0,1,1)                                           │
│      TC-33: 0.4 ± 0.9  (0,2,0,0,0)                                           │
│      TC-35: 1.2 ± 0.5  (1,1,1,1,2)                                           │
│      TC-40: 1.8 ± 0.5  (1,2,2,2,2)                                           │
│      TC-42: 0.8 ± 1.1  (2,0,0,0,2)                                           │
│      TC-49: 1.8 ± 0.5  (2,2,1,2,2)                                           │
│      TC-50: 1.2 ± 0.5  (1,1,2,1,1)                                           │
│      TC-51: 1.2 ± 0.5  (2,1,1,1,1)                                           │
│      TC-52: 1.8 ± 0.5  (1,2,2,2,2)                                           │
│      TC-53: 1.4 ± 0.6  (1,2,1,1,2)                                           │
│      TC-54: 1.8 ± 0.5  (2,2,1,2,2)                                           │
│      TC-55: 1.8 ± 0.5  (2,2,2,1,2)                                           │
│      TC-57: 1.8 ± 0.5  (2,1,2,2,2)                                           │
│      TC-63: 1.2 ± 1.1  (2,0,2,0,2)                                           │
│      TC-76: 1.8 ± 0.5  (2,2,1,2,2)                                           │
│      TC-80: 0.8 ± 1.1  (0,2,2,0,0)                                           │
│      TC-85: 1.8 ± 0.5  (2,2,1,2,2)                                           │
│      TC-88: 0.8 ± 1.1  (0,0,2,2,0)                                           │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯

[L00kUp]

Hi everyone — this is my first post here, thank you all for the awesome work :)

I’ve had a DGX Spark since early this year, but only since Qwen3.8-27B have local models
become genuinely usable for my daily work, without noticeable quality degradation versus
the larger cloud models.

Over the last few days I tested these two Qwen3.8 Flash-Next deployments:

• azampatti/Qwen3.8-Flash-Next-Int4-FAST
• tonyd2wild/Qwen3.8-Flash-Next-NVFP4-DGX-Spark
  — related forum thread

So far, I have not noticed a meaningful quality difference in daily use. I thought the
real-work numbers might be useful to share.

My setup is DeepSeek Harness with no unusual tuning (medium reasoning effort is my default). The analysis was compiled from session
logs by GPT Sol (high).

Metric	tonyd NVIDIA NVFP4 — Sep 9	INT4-FAST — Sep 10
Pooled decode	24.7 tok/s	43.5 tok/s
Strict c1 decode	36.0 tok/s	51.0 tok/s
Median TTFT	4.15 s	2.10 s

INT4 was 36–42% faster at concurrency 1–4, and 76% faster by pooled median decode rate.

Performance by broad workload category

I also grouped each root session by its initial task. This is not a task-matched
benchmark — the mix differs materially between the two days — but it helps make the
workload behind the pooled result visible. Each entry is calls / root sessions · median decode · median TTFT; — means that model did not receive that category in this window.

Category	tonyd NVIDIA NVFP4 — Sep 9	INT4-FAST — Sep 10
CI, dependencies & versioning	1,310 / 13 · 25.6 tok/s · 4.09 s	64 / 2 · 43.8 tok/s · 1.87 s
Debugging & maintenance	84 / 1 · 18.6 tok/s · 4.36 s	94 / 2 · 39.6 tok/s · 1.72 s
Documentation & reporting	657 / 5 · 22.6 tok/s · 4.24 s	43 / 1 · 30.0 tok/s · 2.44 s
Serving & harness	102 / 1 · 24.4 tok/s · 4.34 s	142 / 3 · 41.2 tok/s · 2.26 s
Other agent work	102 / 4 · 25.1 tok/s · 3.92 s	293 / 3 · 47.7 tok/s · 2.14 s

A few definitions:

• Pooled decode is the median per-request generation speed across all eligible real-work
  requests, including requests made under concurrent load. It is not aggregate GPU throughput.
• Strict c1 decode only includes requests that had no overlap with another request, so it
  is the closest real-work single-stream figure.
• Very short outputs were excluded from decode-rate calculations because they can exaggerate
  tok/s.

EDIT: my DGX spark is running a lower core speed nvidia-smi -lgc 0,2000 - number shall be slightly higher with stock settings

[azampatti]

FYI, you might want to try the “medium” chat template included. Still thinks nicely, still as capable, but the wall-to-wall time drops heavily and some benchmarks actually count this to calculate tok/s

Also, MTP head works best in coding than prose writing, so heavy thinking will get less MTP Acceptance which equals to less tok/s effective decode speed. I’m running medium decode for my daily workflow and it’s perfect (and faster)

[azampatti]

Just included a link to @ds2atc in the OP. I was just able to run a --trials 3 on mine and matches what he benchmarked +/- 1 so it’s accurate :)

[0rand]

Did you have a chance to run it on 2 nodes? Pulling your quant now - goal is to run on 2 as a fast capable model not most complex tasks and at least 60 t/s

[azampatti]

Sadly I don’t have two nodes to be able to test it. So let me know how it goes!

[0rand]

It’s up and running on one - nicely, sampling speed and quality for A/B test then we try to shard it on two

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Test                                                       ┃       c        ┃                      pp t/s ┃                     tg t/s ┃                     TTFT (ms) ┃                   Total (ms) ┃                       Tokens ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ pp2048 tg1024 @ d0                                         │       c1       │                         977 │                       48.9 │                         2,287 │                       23,084 │                    2048+1024 │
│ pp2048 tg1024 @ d0                                         │       c2       │                         700 │                       75.7 │                         4,921 │                       30,137 │                    2048+1024 │
│ pp2048 tg1024 @ d0                                         │       c4       │                       1,089 │                       85.3 │                         7,367 │                       50,702 │                    2048+1024 │
│ pp2048 tg1024 @ d2048                                      │       c1       │                         965 │                       45.4 │                         4,439 │                       26,865 │                    2048+1024 │
│ pp2048 tg1024 @ d2048                                      │       c2       │                       1,004 │                       79.3 │                         8,091 │                       32,720 │                    2048+1024 │
│ pp2048 tg1024 @ d2048                                      │       c4       │                         981 │                       78.2 │                        16,645 │                       64,697 │                    2048+1024 │
│ pp2048 tg1024 @ d8192                                      │       c1       │                       1,017 │                       58.5 │                        10,264 │                       27,632 │                    2048+1024 │
│ pp2048 tg1024 @ d8192                                      │       c2       │                         993 │                       60.7 │                        19,746 │                       52,114 │                    2048+1024 │
│ pp2048 tg1024 @ d8192                                      │       c4       │                       1,023 │                       63.6 │                        34,111 │                       82,165 │                    2048+1024 │
└────────────────────────────────────────────────────────────┴────────────────┴─────────────────────────────┴────────────────────────────┴───────────────────────────────┴──────────────────────────────┴──────────────────────────────┘

[0rand]

Findings: it does not work on TP=2 because of how tensors are packaged - non-divisible. Good news is its cheap and easy to requantize them to make a split. On it.