Stumbled upon this:
Somebody more capable than me wanna take a look?
Indeed! Slightly different approach though, might yield different results.
What I tested was FlyCockpit/DeepSeek-V4-Flash-0731-vision · Hugging Face sitting on top of the official weights.
Summary: it works, but couldn’t get full speed from it, and what the agent gets is worse than Gemma 4 E2B vision (which sits comfortably alongside DS4 Flash on a 2x Spark setup).
Guess we will see more efforts like this. Hopefully we’ll get a decent version with vision. Would make a few things easier.
After rebooting, the speed was better (around 50 tps instead of the 55-60 I usually get)
If any of you want to try it out, here is the guide for it, made by DS4 Flash itself. You can feed that to an agent and they’ll have all gotchas x)
Click here to see full guide
# Eyes for DeepSeek-V4-Flash-0731 on 2Ă— DGX Spark
*AI-generated technical writeup, based on a real deployment.*
DeepSeek-V4-Flash-0731 gained vision on a 2× DGX Spark cluster — with no image rebuild and no sacrifice of the DSpark speculative decoding that makes it fast locally. This post describes the stack, the minimal launch delta, one hard-won fix that matters, and what agents consuming the API should expect.
## The stack
Vision comes from the FlyCockpit DeepEncoderV2 encoder (Apache-2.0/MIT): an 865 MB frozen tower plus a 40 MB trained projector adapter that maps tower features into the 0731 backbone’s embedding space. It plugs into vLLM through a small plugin (`dsv4_vision_vllm`) that registers a wrapper model, `DeepseekV4VisionForCausalLM`, via vLLM’s standard `vllm.general_plugins` entry point. Serving it requires a model directory that is the 0731 snapshot with exactly one change: `architectures` swapped in `config.json` (built as zero-cost symlinks, so the original checkpoint is untouched).
Everything was validated on `aidendle94/sparkrun-vllm-ds4-gb10:production-3.7-reffix`, a flavor of the official `aidendle94/sparkrun-vllm-ds4-gb10:production-3.7` image — the mechanism described here is plain vLLM plugin territory and does not depend on launcher or image internals.
## The launch delta
Everything else is the normal 0731 DSpark serve line. Four concrete changes:
**1. Get the plugin.** It ships in the FlyCockpit playbook:
```bash
# plugin/ is the pip-installable package dsv4_vision_vllm
```
Apply the wrapper-transparency patch from the next section to `plugin/src/dsv4_vision_vllm/model.py`.
**2. Download the encoder assets** (about 900 MB) on each node:
```bash
hf download FlyCockpit/DeepSeek-V4-Flash-0731-vision \
tower/deepencoder_v2_tower.safetensors \
adapter/latest.pt \
–local-dir ~/.cache/huggingface/vision
```
Check the md5s: adapter `d9b3b3bda8f790ecf7cd5a98e6fb93a5`, tower `2d5dba626d816cc367d28b32e744830e`.
**3. Build the vision model dir** — a zero-cost symlink tree of the 0731 snapshot with `architectures` swapped to `DeepseekV4VisionForCausalLM` (adjust the container path in the script to match where your HF cache mounts):
```bash
python3 plugin/make_vision_model_dir.py # → ~/.cache/huggingface/dsv4-0731-vision
```
**4. Launch.** In our setup the HF cache is mounted at `/root/.cache/huggingface` inside the container, so the plugin and assets are already visible there. Install the plugin at container start (or bake it into the image — either works):
```bash
python3 -m pip install --quiet --no-deps --no-build-isolation /root/.cache/huggingface/vision/plugin
```
Then the serve line — normal 0731 DSpark flags, only these change:
```
vllm serve /root/.cache/huggingface/dsv4-0731-vision \
--limit-mm-per-prompt '{"image":8}' \\
--trust-request-chat-template
```
with `DSV4_VISION_TOWER=/root/.cache/huggingface/vision/tower/deepencoder_v2_tower.safetensors` and `DSV4_VISION_ADAPTER=/root/.cache/huggingface/vision/adapter/latest.pt` in the environment.
## The fix that matters
The stock upstream plugin quietly breaks DSpark. The draft keeps running, but acceptance collapses to roughly 1-15% — throughput drops to ~20 tps. The cause: the vision wrapper hides the backbone, cutting off the auxiliary hidden-state flow the DSpark draft feeds on. The fix is to keep the wrapper transparent to the backbone — pass `**kwargs` through in `forward()` and expose an `lm_head` property. With that in place, acceptance recovers to 50-64% with a mean acceptance length around 2.0.
The broken state is recognizable in the logs: `SpecDecoding metrics: Per-position acceptance rate: 0.0x, 0.0`. The fix — edit `plugin/src/dsv4_vision_vllm/model.py` — is to keep the wrapper transparent to the backbone: pass `**kwargs` through in `forward()`, and expose an `lm_head` property that forwards to the language model. If acceptance ever collapses after a plugin update, check the wrapper’s transparency first.
## Agent-facing contract
- Model id: `deepseek-v4-flash`
- Images: `image_url` content parts with base64 data URIs
- **Image requests must send `chat_template_kwargs: {“thinking”: false}`.** The recipe defaults to `thinking:true`, but on image input the model answers without thinking — the answer lands inside an unclosed think block and `content` comes back empty (the text ends up in the `reasoning` field). This one line saves hours of debugging.
- At most 8 images per request, counted across replayed history; a 9th triggers a clean HTTP 400.
## What to expect
- Screenshots, UIs, and on-screen text: **strong** — near-perfect transcription in tests.
- Documents: strong.
- Everyday photos: decent but generic — this is a screenshot specialist, not a general-purpose vision-language model.
- Click-agents and real-world computer use: **not ready** — explicitly unclaimed by the upstream project.
Under the `tiles=2` layout, an image expands to `n_viewsĂ—256+1` tokens (257 / 769 / 1281). Verify the layout is active by checking `[dsv4-vision] checkpoint config.tiles=2` in the logs; a `tiles=0` fallback silently serves the wrong token layout.
## Measured results
On 2× DGX Spark (TP=2): DSpark acceptance 50-64% with mean acceptance length ~2.0, ~40-50 tps after a clean reboot (below 40 before it — worth knowing that a reboot measurably helped). DSpark stays engaged even on image requests (~63% acceptance).
Overall, this is a solid upgrade for UI- and screenshot-heavy agent work. For workloads where raw text speed is the priority, keeping a plain no-vision variant alongside (same model id, different launch config) is the right call.
Edit: it showed issues with context management, this is NOT production-ready.
This popped up. Anyone try it yet ? webbrain-one/DeepSeek-V4-Flash-0731-Vision-NVFP4 · Hugging Face
Adds a whole 9 GB to the weights. Could/should be better quality than the other one which was only about 1 GB - but could be challenging due to the constant struggle with memory constraints.
I’ve a working recipe here with deepseek v4 flash + Qwen3.5-9B NVFP4
Both use dual GPU setup, I got it to work with UI screenshots + improvements etc. please contribute and see if we can optimize memory usage further,