Running DeepSeek V4 + vision model on 2x DGX Spark (GB10) – memory balancing tips?

Hi everyone,

I recently upgraded from a single ASUS GB10 to a cluster of two connected via QSFP. On the single unit I was running Qwen3-30B-A3B, which worked fine for most things — but FOMO got the better of me, so I went bigger.

With two units, I settled on running DeepSeek V4 via Orand’s repo. Benchmark scores (tool-eval-bench) are similar to what I had before, but I can definitely see the quality bump in some of my automated Hermes Agent jobs, like daily summaries of world cup :) .

My problem is that DS4 lacks vision support, and I need that for browser control and computer-use tasks. I’ve had to cut DS4’s context length and concurrency quite a bit to free up enough VRAM for Qwen3-VL to handle vision. Even so, I’m still teetering on the edge of OOM — I’m down to 256K context with ~0.73 GPU utilization for DS4, and it’s still tight.

How is everyone else solving this? I’m aware I could route vision tasks through an API, but that kind of defeats the purpose of running everything fully locally.

I’m running gemma-4-12B-it-qat-OptiQ-4bit on my MacBook Pro using oMLX and made a custom pi coding agent extension that gives DSV4 a ‘describe_image’ tool so that, it can prompt Gemma 4 with vision requests as many times and however it wants based on the context. I also have another custom extension/tool that DSV4 can use to load/unload an OCR Model on-demand if the context deems it more relevant than gemma 4.

I’d totally recommend hosting a vision model on a separate machine if possible and keep the 2x cluster completely for DSV4.

I have hermes running on an m1 macbook pro with 16gb of ram as a dedicated “AI machine”. Do you think is it worth trying to move the vision model on that machine? will it even fit?

Definitely, give it a shot. Gemma 4 12B should squeeze in there or probably it’s better to play around with different models to see what’s best for your needs and constraints.

Thats exactly what I am doing too. But its possibly to run a small model like Qwen 3.5 4b with vision tower on a worker node that usually has 2-3GB more room. It was working too. Lower quality but okay if you just need it to debug, look at screenshots. Gemma4 Vision is good. But I run Qwen 3.5 122b VL, because I can :)

Deepseek v4 flash (non preview version) is almost definitely being released this Friday, and there is a decent chance it will add vision capabilities, so I may wait until then as to not waste time trying to tackle this problem.

They had some insane vision paper posted online but took it away in 1-2 hours. So likely it is in works.

what do you mean non preview version? are you telling me we’ll get an even better DSV4 Flash?

Yep check the comments here: Reddit - Please wait for verification also deepseek sent out an email telling developers that the API will be deprecated for the official version coming in “mid-July”.

I am serving two separate LLMs on a two-node DGX Spark cluster.

For the front-end multimodal analysis stage, I use Red Hat’s Gemma 12B NVFP4 || RedHatAI/Qwen3.5-9B-FP8-dynamic, while DeepSeek V4 Flash is served with MTP=3 as the text-based reasoning and task execution model. Both models currently achieve an average generation speed of approximately 35–40 tokens per second.

This method allows DeepSeek V4 to process not only visual information but also audio data, significantly expanding its potential use cases. In particular, I use this setup with the official Unity MCP to analyze game screens and gameplay situations, then use the resulting information to control or modify Unity projects.

The ability to directly analyze gameplay footage is another major advantage. However, in a multimodal tower like this, simply connecting the models is not enough. It is especially important to define what information the multimodal model should extract from visual and audio inputs, and how that information should be structured before being passed to the downstream text-based LLM.

Ultimately, the overall performance of the system can vary significantly depending on the analysis criteria and output format assigned to the multimodal tower. Designing an effective multimodal analysis prompt and a well-structured information transfer pipeline is therefore one of the most important aspects of this setup.

If you need strong OCR or vision capabilities, Qwen3.5 4B and Qwen3.5 9B are much stronger options than Gemma 4 12B.

I didn’t know vision is likely on the table, thanks

Thanks for sharing your setup.

For RedHatAI/Qwen3.5-9B-FP8-dynamic on a two-node DGX Spark cluster, could you share the approximate vLLM settings that have been stable for you?

In particular, I am curious about:

  • gpu_memory_utilization for Qwen
  • max_model_len
  • max_num_seqs
  • max_num_batched_tokens
  • KV cache dtype
  • whether Qwen runs on one node or is distributed
  • whether it can run concurrently with DeepSeek V4 Flash without OOM

Would a conservative Qwen profile around 32K context, seq=1, one image per request, and roughly 0.16 GPU utilization be realistic, or did you need more memory headroom?

I am mainly trying to understand the stable memory split between the vision model and DeepSeek.

Thanks!

  1. Use llamacpp, not vllm, much less memory use
  2. Load cluster first, then llamacpp, easy to script

We recommend deploying both models in a distributed serving environment.

In a distributed two node serving configuration, Qwen 3.5 9B can be served with up to 262,144 context length, while DeepSeek V4 Flash can be served with up to 1,048,576 context length. However, for stable operation, we recommend setting Qwen 3.5 9B to 131,072 context length since it will be used primarily for vision analysis. Additionally, when using it as a vision analysis model, we recommend serving it in non-thinking mode rather than thinking mode.

Both models have been verified to support a maximum batch size of 100, and we expect they may support higher batch sizes as well. In actual workloads, stable operation has been confirmed with approximately 10–20 concurrent users. However, a production workload with 100 concurrent users has not yet been validated.

"GPU memory"and “max num batched tokens” settings should be configured as follows:

Qwen 3.5 9B
--gpu-memory-utilization 0.095
--max-num-batched-tokens 8192

DeepSeek V4 Flash
--gpu-memory-utilization 0.785
--max-num-batched-tokens 8192

Additionally, the KV Cache dtype must be set to FP8 for both models.

If DeepSeek V4 Flash is already being served across two nodes, the remaining VRAM on each node is approximately 20 GiB. In this configuration, deploying Qwen 3.5 9B FP8 on one of the existing DeepSeek serving nodes would make it difficult to maintain sufficient context length. Therefore, Qwen 3.5 9B should also be deployed across two nodes in a distributed serving environment.

When running both models simultaneously, the most important requirement is that both models must be served using the “no ray” mode. For detailed configuration and implementation guidance, we recommend referring to eugr’s GitHub repository.

Additionally, the DeepSeek V4 DSpark version consumes more memory than the standard DeepSeek V4 Flash version, which may prevent allocating sufficient context length to Qwen 3.5 9B. Therefore, if you plan to operate both models in a distributed environment, we recommend using the standard DeepSeek V4 Flash version instead of DeepSeek V4 DSpark.

The recommended final serving is to deploy Qwen 3.5 9B as a vision analysis model and use DeepSeek V4 Flash as the main reasoning model. This servning method separates vision analysis and LLM inference workloads, enabling a stable multi-user serving environment capable of handling multiple concurrent users.

I run a small .8b qwen model on mlx