MJPansa/MiniMax-M2.7-REAP-172B-A10B-AutoRound-W4A16

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Hey guys,

since over the past days I was eagerly waiting for M2.7 REAPS to come out with quants suitable for single DGX Sparks but I could not find anything and also had holidays from monday to wednesday I created one myself. Unfortunately I only have a single spark plus a 256GB Threadripper with 2x 3090s so my setup is very shitty for calibrating with huge samples sizes and samples lengths.
Also running evals takes forever, I ran GPQA Diamond for over 15 hours just to discover I was too stingy with the max tokens at 16k and 30% of samples didnt finish reasoning within this budget. It still got 60% so the general intelligence seems to be there. Will run some more evals on external compute over the weekend. will need to rent some H200 most likely.

Anyways I thought i still share it here if someone wants to give it a go as well :)
Speed is basically the same as 122B A10B Qwen model with autoround.

There is definitely room for improvements with new kv cache quants landing in vllm. On a single spark with fp8 you can get around 100k context i guess. I havnt pushed it all too far since my agents are also sharing some portion of the ran most of the time

depth prefill tok/s decode tok/s TTFT (ms)
0 2469.3 ± 13.3 29.28 ± 0.05 864.5
4096 2089.9 ± 12.5 27.73 ± 0.05 2784.8
8192 1890.3 ± 5.2 26.28 ± 0.05 5062.3
16384 1601.1 ± 6.5 23.88 ± 0.05 10647.7

Happy for any feedback, this is just a first draft. WIll need to pick some better and bigger datasets for REAP and quantisation calibration. Just wanted to validate everything end to end first before renting out more compute

2

TQ3/TQ4 buys you a little more context, worked when i tested it for 12 hours.
So its possible :)

3

TurboQuant quantised LLM models appear on Hugging Face :-).

The model can be run via

4

This is cool!! Is there a way to run this @eugr

In the meantime I will give it a try to your quant @mjpansa

5

In one single spark? Any scripts or variables?

6

You can give it a try with the most recent build - some of the Turboquant-related PRs have been merged into vLLM recently.

7

My spark is still running aime benchmarks, will try turbo quant tomorrow

8

Not yet.

The main repo now has some TQ options for KV cache, but this requires the whole model too.

You need turboquant-vllm for that one - for now.

It might be worth looking into setting that up for testing. This isn’t the only model that could fit on 1 Spark much more efficiently under such a framework.

9

This is amazing. By far the best model I have run on my spark. I couldn’t get the turboquant version working but 100k context is plenty for my uses

10

Minimax is decent, but it has never been able to surpass the step-3.5-flash model. I’ve used many recently released models with ik_llama.cpp and lvllm (a variant of vllm), and they are all far inferior to step-3.5-flash, which is strange. Perhaps it’s because I’m using roo code.

11

does step-3.5-flash fit in one spark via vLLM? I think that is kind of the benefit of this specific reap model, it fits on one spark with enough headroom for 100k context (or even 110k if you are careful).

12

It fits, but not with vLLM. You have to use llama.cpp for Step-3.5-Flash. Until TurboQuant model weights start hitting prime time anyway.

There’s a thread here for that model, with the IQ4_XS quant by ubergarm recommended.

13

Has anyone been able to try this model on dual spark configuration? I am curious if that configuration would create more output t/s or just allow a greater context size.