100b+ parameter LLM list

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
,
1

Hi everyone. Just curious what sort of success folks have had running models over 100b parameters. Please limit your post to the model (if available) and your preferred method to running it. Model specific topics should be a new post. It may also be worthwhile mentioning what models you’re still struggling with.

I’ll kick things off:

Success:

No success so far:

  • Multinode Llama4
  • RedHatAI/Llama-4-Maverick-17B-128E-Instruct-NVFP4 (not enough GPU memory on dual GB10?)
  • mistralai/Devstral-2-123B-Instruct-2512 on tensor-llm (mistral3 not supported yet?)
  • NVFP4/Qwen3-Coder-480B-A35B-Instruct-FP4 on tensor-llm or vllm (not enough GPU memory on dual GB10?)
2

Thank you for sharing this list!

3

Thank you for the post and for summarizing both your achievements and the areas that didn’t yield results. I’m very interested to see what kind of performance you’ve achieved and whether you’ve done any testing using ShareGPT_V3_unfiltered_cleaned_split.json.

4

Just an update. I’m able to run nvidia/Llama-4-Scout-17B-16E-Instruct-NVFP4 with a 0.5M context length which is handy.

I use this tool by @eugr to get a vLLM cluster running and then use these settings to get it running:

./launch-cluster.sh --nodes 192.168.1.2,192.168.1.3
exec vllm serve nvidia/Llama-4-Scout-17B-16E-Instruct-NVFP4
-tp 2
–enable-auto-tool-choice
–tool-call-parser llama4_pythonic
–kv-cache-dtype fp8
–gpu-memory-utilization 0.9
–max-model-len 524288
–distributed-executor-backend ray
–host 0.0.0.0
–port 8355

I’ve also gotten mistralai/Devstral-2-123B-Instruct-2512 to run, but it was not usable:

./launch-cluster.sh --nodes 192.168.1.2,192.168.1.3
exec vllm serve mistralai/Devstral-2-123B-Instruct-2512
-tp 2
–enforce-eager
–tokenizer_mode mistral
–config_format mistral
–load_format mistral
–enable-auto-tool-choice
–tool-call-parser mistral
–gpu-memory-utilization 0.8
–max-model-len 61440
–distributed-executor-backend ray
–host 0.0.0.0
–port 8355

Regarding Ollama these models run well and can tolerate a large context:
glm-4.7-flash:q8_0
granite4:32b-a9b-h (this seems to suffer a bit of brain-rot, but the big context is nice)

5

Here is the approach I used to get devstral-2-123b model running on my Ascent gx10 device. With speculative decoding, seeing that it is generating ~5 tokens per second. The ministral-3-3b reasoning model seems to be pulling its weight with draft acceptance rates generally >0.7. Without speculative decoding, inference is slower at ~2 tokens per second, but can accommodate full 262144 context with a 128 Gb memory constraint.

docker run -it
–name devstral-2-123b-speculative
–gpus all
-p 3014:8084
-v /home/jsmes/huggingface/unsloth/Devstral-2-123B-Instruct-2512-GGUF:/models
llama.cpp:server-spark
–model /models/UD-Q4_K_XL/Devstral-2-123B-Instruct-2512-UD-Q4_K_XL-00001-of-00002.gguf
–model-draft /models/draft/Ministral-3-3B-Reasoning-2512-Q4_K_M.gguf
–draft 10
–host 0.0.0.0
–port 8084
–ctx-size 64000
–batch-size 4096
–ubatch-size 1024
–flash-attn on
–cache-type-k q4_0
–cache-type-v q4_0
–n-gpu-layers 99
–no-mmap
–prio 2
–temp 0.15
–chat-template-file /models/chat_template_modified_speckit.jinja
–jinja

6

I’ve run GLM-4.7 with one Spark at 2.57 bpw quantization via exllamav3. Need to get some max context length and inference speed stats, then I’ll make a new post with the details.