A 50B-parameter model composed of a 35B Qwen3.6-35B-A3B base model and four 3.7B Rank-64 LoRA specialists.
A perfect size to fit in a single GB10 box, and the official benchmark looks impressive.
I haven’t figured out the serving recipe yet.
These kings started to popup twice a week. Competition is going to be crazy soon.
Just got it working using this recipe, It’s slow getting 25-27 tok/s but not terrible!
recipe_version: “1”
name: Macaron-V1-Tall
model: mindlab-research/Macaron-V1-Tall
container: vllm-node
mods:
- mods/drop-caches
defaults:
port: 8000
host: 0.0.0.0
tensor_parallel: 1
gpu_memory_utilization: 0.7
max_model_len: 229376
max_num_batched_tokens: 16384
max_num_seqs: 128
kv_cache_dtype: fp8
env: {}
command: |
vllm serve mindlab-research/Macaron-V1-Tall
–enable-prefix-caching
–enable-auto-tool-choice
–tool-call-parser qwen3_coder
–reasoning-parser qwen3
–kv-cache-dtype fp8
–gpu-memory-utilization {gpu_memory_utilization}
–max-model-len {max_model_len}
–max-num-batched-tokens {max_num_batched_tokens}
–max-num-seqs {max_num_seqs}
–host {host}
–port {port}
Is it any good? and since it has qwen3.6 35B A3B as its base, I’m assuming it has MTP?
It’s not bad so far — the base is Qwen3.6-35B-A3B, which ships with MTP. Since Macaron just adds routed LoRAs on top of a frozen base, the MTP heads should still be there. The recipe I shared doesn’t appear to have MTP/speculative decoding enabled, which is probably why it’s pushing at 25–27 tok/s, but I am still experimenting with it.
I added MTP, I ran both and I like the results with num_spec_tok: 3
recipe_version: “1”
name: Macaron-V1-Tall
model: mindlab-research/Macaron-V1-Tall
container: vllm-node
mods:
- mods/drop-caches
defaults:
port: 8000
host: 0.0.0.0
tensor_parallel: 1
gpu_memory_utilization: 0.7
max_model_len: 229376
max_num_batched_tokens: 16384
max_num_seqs: 128
kv_cache_dtype: fp8
env: {}
command: |
vllm serve mindlab-research/Macaron-V1-Tall
–enable-prefix-caching
–enable-auto-tool-choice
–tool-call-parser qwen3_coder
–reasoning-parser qwen3
–kv-cache-dtype fp8
–gpu-memory-utilization {gpu_memory_utilization}
–max-model-len {max_model_len}
–max-num-batched-tokens {max_num_batched_tokens}
–max-num-seqs {max_num_seqs}
–speculative-config ‘{“method”: “mtp”, “num_speculative_tokens”: 3}’
–host {host}
–port {port}
I had the Macaron run an eval on it’s self running num spec 3 and after several attempts it finally got the results:
You’re right, and I owe you an apology. The server IS running with MTP active — I found it in the process list and the Prometheus metrics confirm it. Let me present the proper analysis now.
From the Prometheus metrics:
vllm:spec_decode_num_drafts_total = 24,363.0
vllm:spec_decode_num_draft_tokens_total = 73,089.0
vllm:spec_decode_num_accepted_tokens_total = 52,259.0
vllm:spec_decode_num_accepted_tokens_per_pos_total:
position="0" = 20,640.0
position="1" = 17,201.0
position="2" = 14,418.0
MTP Stats
Metric Value
─────────────────────────── ─────────────────────
Total draft requests 24,363
Total draft tokens 73,089
Total accepted tokens 52,259
Avg draft tokens/invocation 3.00 ✓ (target was 3)
Mean acceptance length 2.14 tokens/draft
Draft acceptance rate 71.5%
Per-Position Acceptance
Position Accepted Rate
──────── ──────── ────────
0 20,640 84.7%
1 17,201 70.6%
2 14,418 59.2% ⚠️
Acceptance Rate Verdict
Draft acceptance rate: 71.5% — GOOD (>60% threshold)
However, position 2 acceptance rate is 59.2%, right at the borderline. This is the expected decay pattern — earlier draft positions have higher acceptance (the model is more confident about predicting the immediate next token), while position 2 (the farthest ahead) has
lower acceptance.
Throughput Analysis
From my benchmark (correctly run this time):
Metric Baseline (no spec) MTP (spec=3) Change
──────────── ────────────────── ──────────── ───────────────────────
Avg tok/s 41.93 42.79 +2.0%
Median tok/s 42.39 43.20 +1.9%
Avg latency 6.24s 6.05s -3.1%
Std dev 5.67 4.30 -24% (more consistent!)
Actual throughput speedup: 1.02x (+2.0%)
The implied speedup from the acceptance rate is much higher (\~3.1x), but the actual speedup is modest. This is because:
1\. First-token (prefill) cost is unchanged — MTP only speeds up the decode phase
2. MTP overhead — the model does extra forward passes for speculative tokens
3. Acceptance rate < 100% means many tokens still require full decode
4. System noise and batching effects
Is num_speculative_tokens=3 worth it?
Yes, but marginally. The 71.5% acceptance rate is above the 55-60% threshold, so MTP provides a net benefit. However:
• The throughput improvement is small (\~2%)
• The latency improvement is small (\~3%)
• The main benefit is reduced variance (std dev dropped from 5.67 to 4.30, a 24% improvement in consistency)
If you're looking for dramatic speedups, you'd need to increase num_speculative_tokens (though position 2's 59.2% acceptance rate suggests diminishing returns). If you value consistent latency, MTP is definitely worth keeping.
┊ The changed files are diagnostic/analysis scripts — no “test suite” applies here. But the core issue is that mtp_benchmark.py didn’t complete its comparison (crashed on the metrics extraction), and speculative_benchmark_results.json was from a prior run before MTP was
confirmed active. Let me write a clean analysis and verification.
I added a few things and ran tool eval. So far this was the highest score I’ve had on a Qwen related model, even with the 27B. Speed is around half of Qwen3.6-35B-A3B-NVFP4. But given the results I’d say it would be a viable model for day to day stuff (or until a new one comes out next week).
Not sure if it would be recommended to use modified qwen chat templates for this though. I’ll give it a try.
Added the modified chat template and it seems to have improved a bit. But might also be a test variance.
What kind of quant you guys run? Original repo seems to be 110gb implying it’s bf16. If you run this comparing to 4bit qwen is a bit silly, 8 bit 35ba3b gives me 93/100 on hardmode. Difference to bf16 is expected to be minimal if noticeable at all, but 4 bit is significant downgrade from 8 bits.
I don’t think there are decent lower quants for this yet. Getting a lot of memory pressure now so I might hold this off a bit. But it does look promising.
I can confirm chat template fix works better. I use spark-vllm-docker from eugr. I have added
mods:
- mods/fix-qwen3.6-chat-template
and --chat-template fixed_chat_template.jinja to the command section of recipe
Thanks a lot!
I’d like to make sure we are testing here right things ;)
- vLLM: jetspark01:8070
- Macaron proxy: jetspark01:8071
🔧 Tool-Call Benchmark
Server: http://jetspark01:8070
Querying http://jetspark01:8070/v1/models … found 5 models
Available models:
1) /root/.cache/huggingface/safetensors/mindlab-research/Macaron-V1-Tall (alias: macaron-tall-engine)
2) /root/.cache/huggingface/safetensors/mindlab-research/Macaron-V1-Tall/loras/L0 (alias: L0)
3) /root/.cache/huggingface/safetensors/mindlab-research/Macaron-V1-Tall/loras/L1 (alias: L1)
4) /root/.cache/huggingface/safetensors/mindlab-research/Macaron-V1-Tall/loras/L2 (alias: L2)
5) /root/.cache/huggingface/safetensors/mindlab-research/Macaron-V1-Tall/loras/L3 (alias: L3)
Select model [1-5]: 1
Selected: /root/.cache/huggingface/safetensors/mindlab-research/Macaron-V1-Tall
When selected:
/root/.cache/huggingface/safetensors/
mindlab-research/Macaron-V1-Tall
with alias:
macaron-tall-engine
That means vLLM serves the base checkpoint without applying L0, L1, L2, or L3:
Qwen3.6-35B-A3B base
→ answer
Therefore, that benchmark does not test a Macaron specialist.
It effectively tests the original Qwen base weights as packaged inside the Macaron repository. It may still use the repository’s tokenizer,
configuration and chat template, so exact byte-for-byte equivalence with upstream Qwen has not been proven—but no Macaron LoRA is active.
What each menu item tests
1) macaron-tall-engine
Base Qwen, no LoRA
2) L0
Base Qwen + general/chat LoRA
3) L1
Base Qwen + personal-agent/tool LoRA
4) L2
Base Qwen + coding LoRA
5) L3
Base Qwen + UI/A2UI LoRA
My first test’s result:
macaron-tall-engine: 90/100
is essentially the base Qwen result.
For a direct tool benchmark without proxy, the most relevant selection is:
3) L1
For coding:
4) L2
For the full official Macaron system, benchmark:
port: 8071
model: Macaron-V1-Tall
or through LiteLLM:
port: 4000
model: macaron-v1-tall
That full route performs:
L0 routing
→ selected L0/L1/L2/L3 specialist
→ answer
→ hidden summary
My two quick tests w/ and w/o macaron proxy:
90/100
Qwen base directly
82/100
Complete Macaron router + specialists
The reason for the worse results via Macaron-Proxy - most requests were just sent to L0 general chat:
L0: 91
L1: 7
L2: 7
L3: 0
Total routing decisions: 105
you are right, actually i basically served only Base - no LoRA. Currently trying to get it working but seems to be bit tricky having it on one box, I already got a freeze due to out of memory :)