We recommend deploying both models in a distributed serving environment.
In a distributed two node serving configuration, Qwen 3.5 9B can be served with up to 262,144 context length, while DeepSeek V4 Flash can be served with up to 1,048,576 context length. However, for stable operation, we recommend setting Qwen 3.5 9B to 131,072 context length since it will be used primarily for vision analysis. Additionally, when using it as a vision analysis model, we recommend serving it in non-thinking mode rather than thinking mode.
Both models have been verified to support a maximum batch size of 100, and we expect they may support higher batch sizes as well. In actual workloads, stable operation has been confirmed with approximately 10–20 concurrent users. However, a production workload with 100 concurrent users has not yet been validated.
"GPU memory"and “max num batched tokens” settings should be configured as follows:
Qwen 3.5 9B
--gpu-memory-utilization 0.095
--max-num-batched-tokens 8192
DeepSeek V4 Flash
--gpu-memory-utilization 0.785
--max-num-batched-tokens 8192
Additionally, the KV Cache dtype must be set to FP8 for both models.
If DeepSeek V4 Flash is already being served across two nodes, the remaining VRAM on each node is approximately 20 GiB. In this configuration, deploying Qwen 3.5 9B FP8 on one of the existing DeepSeek serving nodes would make it difficult to maintain sufficient context length. Therefore, Qwen 3.5 9B should also be deployed across two nodes in a distributed serving environment.
When running both models simultaneously, the most important requirement is that both models must be served using the “no ray” mode. For detailed configuration and implementation guidance, we recommend referring to eugr’s GitHub repository.
Additionally, the DeepSeek V4 DSpark version consumes more memory than the standard DeepSeek V4 Flash version, which may prevent allocating sufficient context length to Qwen 3.5 9B. Therefore, if you plan to operate both models in a distributed environment, we recommend using the standard DeepSeek V4 Flash version instead of DeepSeek V4 DSpark.
The recommended final serving is to deploy Qwen 3.5 9B as a vision analysis model and use DeepSeek V4 Flash as the main reasoning model. This servning method separates vision analysis and LLM inference workloads, enabling a stable multi-user serving environment capable of handling multiple concurrent users.