Thought I’d share my setup from onboarding to my dgx spark since I got such help from you all while lurking this forum.
tl;dr
Which PR are you using? I’m always down for trying new settings that might give me some more performance/quality :)
I totally forgot to include the link to the github repo… :facepalm: and seems I cannot edit my original… :frustrating:
I’m using the PR referenced on the unsloth quants to go with the work, all the details should be in the README
Looking at their code, it’s this one llama + spec: MTP Support by am17an · Pull Request #22673 · ggml-org/llama.cpp · GitHub
I’m currently running some performance on that same PR with Unsloth’s MTP version of Qwen3.6-27B:Q4_K_M
@verdverm I noticed you’re not using Unsloth’s MTP versions here: sparky/util/llama-server.sh at main · verdverm/sparky · GitHub
That is what they suggest: Qwen3.6 - How to Run Locally | Unsloth Documentation
Check out the Ansible, that’s the latest, it supports running with MTP or not-MTP, and should be general enough for any model/quant
My apologies. I see it now, yes very configurable!
the 30t/s (60t/s with MTP) is keeping me happy for the time being
Based off what you said in the other thread ^^, I think you could squeeze more out of it if you want to be happier! I did lots of benchmarking with various llama-server options. The most impactful are the batch sizes. With the same config shared in my other thread, here’s Unsloth’s MXFP4 quant of Qwen3.5-35B-A3B without MTP:
llama-benchy Results
┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Test ┃ c ┃ pp t/s ┃ tg t/s ┃ TTFT (ms) ┃ Total (ms) ┃ Tokens ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ pp2048 tg128 @ d0 │ c1 │ 2,773 │ 62.2 │ 732 │ 2,703 │ 2048+128 │
│ pp2048 tg128 @ d0 │ c2 │ 2,414 │ 90.4 │ 1,464 │ 4,210 │ 2048+128 │
│ pp2048 tg128 @ d0 │ c4 │ 2,360 │ 124.2 │ 2,972 │ 7,005 │ 2048+128 │
│ pp2048 tg128 @ d4096 │ c1 │ 2,576 │ 60.1 │ 2,081 │ 4,124 │ 2048+128 │
│ pp2048 tg128 @ d4096 │ c2 │ 2,557 │ 86.6 │ 4,107 │ 6,977 │ 2048+128 │
│ pp2048 tg128 @ d4096 │ c4 │ 2,508 │ 114.9 │ 8,408 │ 12,774 │ 2048+128 │
│ pp2048 tg128 @ d8192 │ c1 │ 2,589 │ 56.3 │ 3,425 │ 5,610 │ 2048+128 │
│ pp2048 tg128 @ d8192 │ c2 │ 2,538 │ 81.7 │ 6,767 │ 9,814 │ 2048+128 │
│ pp2048 tg128 @ d8192 │ c4 │ 2,393 │ 107.4 │ 14,351 │ 19,029 │ 2048+128 │
└────────────────────────┴───────┴────────────┴────────────┴────────────┴────────────┴────────────┘
So with MTP added on, I think you could go higher than 60 t/s. I’m going to try it once I get past this crashing issue.