Running DeepSeek-V4-Flash and Other Text-Only LLMs as Multimodal with Pilco-mmbridge

[PILCOTHINK]

GitHub - gpdev-Pilcothink/Pilco-mmbridge

I have been running deepseek-ai/DeepSeek-V4-Flash-0731(1M Token) and RedHatAI/Qwen3.5-9B-FP8-dynamic across a 2× NVIDIA DGX Spark cluster.

Turning DeepSeek into a Multimodal Model with Pilco-mmbridge

I actively use Claude Code with Unity MCP and Unreal Engine MCP.

DeepSeek is currently my preferred model because it can process a large amount of context at a relatively low cost. After the July 31 update, its performance improved significantly.

The main limitation is that DeepSeek does not have vision capabilities.

To address this, I started Pilco-mmbridge after testing the setup in real work and using it extensively myself.

The project combines:

Text-Only Model + Multimodal Model

without fine-tuning, so a text-only model can also handle image-based tasks through a separate vision model.

My current recommended combination is:

DeepSeek-V4-Flash-0731
+
Qwen3.5-9B-FP8-dynamic

Qwen handles images and OCR, while DeepSeek handles the main reasoning and final response.

Gemma 4 12B can also be used in a similar omni-style setup, but in my experience its OCR performance is relatively weak. Because I often need to read error messages, warning dialogs, file paths, logs, and UI text accurately, I currently prefer the DeepSeek and Qwen combination.

I mainly use this setup with:

Claude Code
+
MCP or other agent tools
+
Pilco-mmbridge

It can be used in any workflow that requires visual analysis, including screenshots, error messages, documents, user interfaces, charts, application states, game screens, and other information that cannot be fully understood through text alone.

It can also be used outside game development whenever a text-only model needs vision support.

Setup is simple. Serve both models with vLLM, then connect the two API endpoints to Pilco-mmbridge.

The recipes below are configured to serve both models on a 2× DGX Spark cluster.

DeepSeek-V4-Flash-0731

vllm serve /workspace/Model/DeepSeek-V4-Flash-0731 \
  --host 0.0.0.0 --port 8000 \
  --distributed-executor-backend ray \
  --tensor-parallel-size 2 \
  --trust-remote-code \
  --gpu-memory-utilization 0.75 \
  --enable-auto-tool-choice \
  --enable-prefix-caching \
  --reasoning-parser deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --tokenizer-mode deepseek_v4 \
  --max-num-batched-tokens 4096 \
  --max-model-len 1048576 \
  --block-size 256 \
  --kv-cache-dtype fp8 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"greedy"}' \
  --default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
  --max-num-seqs 10

(EngineCore pid=433) INFO 08-01 05:59:31 [kv_cache_utils.py:2168] GPU KV cache size: 1,127,367 tokens
(EngineCore pid=433) INFO 08-01 05:59:31 [kv_cache_utils.py:2169] Maximum concurrency for 1,048,576 tokens per request: 1.08x

(APIServer pid=9) INFO 08-01 06:26:08 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.00, Accepted throughput: 5.40 tokens/s, Drafted throughput: 9.00 tokens/s, Accepted: 54 tokens, Drafted: 90 tokens, Per-position acceptance rate: 0.778, 0.778, 0.556, 0.500, 0.389, Avg Draft acceptance rate: 60.0%
(APIServer pid=9) INFO 08-01 06:26:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 64.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.8%, Prefix cache hit rate: 64.1%

Qwen3.5-9B-FP8-dynamic

vllm serve /workspace/Model/Qwen3.5-9B-FP8-dynamic  \
  --host 0.0.0.0 --port 8001 \
  --distributed-executor-backend ray \
  --trust-remote-code \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.075 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --enable-prefix-caching \
  --tool-call-parser qwen3_coder \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --max-num-batched-tokens 2048 \
  --max-model-len 65536 \
  --kv-cache-dtype fp8 \
  --max-num-seqs 10

(EngineCore pid=254) INFO 08-01 06:09:10 [kv_cache_utils.py:2168] GPU KV cache size: 88,331 tokens
(EngineCore pid=254) INFO 08-01 06:09:10 [kv_cache_utils.py:2169] Maximum concurrency for 65,536 tokens per request: 1.35x

(APIServer pid=9) INFO 08-01 06:10:55 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 33.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%

I will continue updating the project and sharing tested recipes in this thread.

[PILCOTHINK]

When serving the models using the recipe provided in the main text, the system can quickly run out of memory during actual use. Therefore, I recommend serving the models with the following configuration instead.

DSpark DeepSeek Configuration

vllm serve /workspace/Model/DeepSeek-V4-Flash-0731 \
  --host 0.0.0.0 --port 8000 \
  --distributed-executor-backend ray \
  --tensor-parallel-size 2 \
  --trust-remote-code \
  --gpu-memory-utilization 0.77 \
  --enable-auto-tool-choice \
  --enable-prefix-caching \
  --reasoning-parser deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --tokenizer-mode deepseek_v4 \
  --max-num-batched-tokens 2048 \
  --max-model-len 524288 \
  --block-size 256 \
  --kv-cache-dtype fp8 \
  --default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"greedy"}' \
  --max-num-seqs 5

vllm serve /workspace/Model/Qwen3.5-9B-FP8-dynamic \
  --host 0.0.0.0 --port 8001 \
  --distributed-executor-backend ray \
  --gpu-memory-utilization 0.06 \
  --trust-remote-code \
  --tensor-parallel-size 2 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --max-num-seqs 5

[PILCOTHINK]

I found and fixed a critical bug in my project.

The issue was that once an image had been uploaded, the previously uploaded image would continue to be sent to the vision model on subsequent requests, even if the user did not include an image anymore. As a result, the system kept returning image analysis results for old images.

This has now been fixed. Image analysis is only triggered when the current user request actually contains an image.

After testing, I also arrived at a serving configuration that provides a good balance between stability and performance. I’m sharing it here in case it’s useful to others.

The following recipe has only been tested on the DGX Spark FE model.

vllm serve /workspace/Model/DeepSeek-V4-Flash-0731 \
  --host 0.0.0.0 --port 8000 \
  --distributed-executor-backend ray \
  --tensor-parallel-size 2 \
  --trust-remote-code \
  --gpu-memory-utilization 0.75 \
  --kv-cache-memory-bytes 11859478446 \
  --enable-auto-tool-choice \
  --enable-prefix-caching \
  --reasoning-parser deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --tokenizer-mode deepseek_v4 \
  --max-num-batched-tokens 8192 \
  --max-model-len 1048576 \
  --block-size 256 \
  --kv-cache-dtype fp8 \
  --default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"greedy"}' \
  --max-num-seqs 10

vllm serve /workspace/Model/Qwen3.5-9B-quantized.w4a16 \
  --host 0.0.0.0 --port 8001 \
  --distributed-executor-backend ray \
  --gpu-memory-utilization 0.05 \
  --kv-cache-memory-bytes 367001600 \
  --trust-remote-code \
  --tensor-parallel-size 2 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --enable-prefix-caching \
  --tool-call-parser qwen3_coder \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --max-model-len 32768 \
  --max-num-batched-tokens 2048 \
  --kv-cache-dtype fp8 \
  --max-num-seqs 10

Note: The --distributed-executor-backend ray option can be omitted when using the no-Ray mode. However, it is required when running with Ray, so I included it in the recipe for completeness.

=========================================================

[PILCOTHINK]

You can now run multimodal benchmarks with text-only LLMs

With Pilco-mmbridge, a text-only LLM can be connected to a separate vision model and evaluated on standard multimodal benchmarks such as MMMU.

After several rounds of updates and improvements to the project, I ran its first objective multimodal benchmark using MMMU Validation in a 0-shot setting.

MMMU Results

Result	Correct / Evaluated	Accuracy
Raw lmms-eval result	692 / 900	76.89%
Completed-sample accuracy	679 / 848	80.07%

Recorded Token Usage

Metric	Usage
Input tokens	822,868
DeepSeek Output tokens	1,741,570
Total tokens	2,564,438

The evaluation consumed approximately 2.56 million tokens in total.

A significant portion of the output-token usage came from samples that reached the generation limit. The 51 affected samples alone consumed 835,584 output tokens, accounting for approximately 48% of all output tokens used during the evaluation.

Adjustment Details

During this run, I set max_new_tokens to 16,384. Some samples used the entire output-token budget while reasoning and failed to produce or parse a final answer correctly.

The affected samples were:

• 51 samples that reached the maximum output-token limit
• 1 sample that still failed with an API error after all retries
• 52 invalid samples in total

There was one additional evaluation detail: 13 of these invalid multiple-choice samples were accidentally counted as correct because the evaluator’s fallback parser selected an answer even though the actual response contained an error.

For that reason, the completed-sample result removes both the 52 invalid samples from the denominator and the 13 accidentally correct samples from the numerator:

(692 - 13) / (900 - 52) = 679 / 848 = 80.07%

Therefore, 76.89% is the raw result across all 900 samples, while 80.07% is a reference score calculated only from samples that completed normally.

The completed-sample result should not be treated as a directly comparable official full-set score until the failed samples are rerun. Even with that caveat, I think this is a very encouraging level of multimodal performance for the current setup.

Models

• Text model: DeepSeek-V4-Flash-0731
• Vision model: Ornith-1.5-9B-MixedInt4-AutoRound

The vision model is a quantized version of Ornith-1.5-9B that I prepared myself. During quantization, I specifically calibrated it to preserve as much of its Instruct/non-reasoning response quality as possible.

Both models are currently served with vLLM across two DGX Spark systems.

Overall, these results suggest that combining a text-only LLM with a separate vision model through Pilco-mmbridge can provide practical and competitive multimodal performance.

[elvis.castillo]

Now there is a mix of both Jackrong/Qwen3.5-9B-DeepSeek-V4-Flash-GGUF · Hugging Face

It seems to fit into a 16GB and runs surprisingly well for coding.

[PILCOTHINK]

This model was fine-tuned on Qwen 3.5 9B using distilled data from the DeepSeek V4 Flash model you mentioned.

Pilco-mmbridge project is a multimodal bridge that connects a multimodal model to a text-based model, allowing the original text model to process multimodal inputs while preserving its existing reasoning capabilities.

Therefore, the original text model’s reasoning performance remains unaffected, while it can still be used to process multimodal inputs such as images when needed. Depending on the capabilities of the connected multimodal model, it can also be extended to support video and audio.

The project is also supported by both llama.cpp and vLLM, allowing it to be used in a variety of environments.

If you’re looking for a fine-tuned model based on a smaller model, the model you mentioned is certainly a good option. However, I would also recommend trying ornith-ai/Ornith-1.5-9B, as it is a very capable fine-tuned model in the 9B class.

[elvis.castillo]

It seems quite good! thank you, I’ll give it a try.