Is anyone else planning to test Inkling Small? I’m downloading it now to try on my two-node cluster. Maybe it can replace Gemma 4 31B for me. The benchmarks look good, and it’s multimodal, unlike Laguna S 2.1.
It looks to be competitive with DeepSeek V4-flash, possibly a small step ahead.
My biggest questions are how the scaling works for KV cache size - can we realize the 1M+ on 2 GB10s? - and if it’s a big enough step to justify the lack of drafting availability. Also, is it enough to stay ahead of the forthcoming final release of DSv4-F.
I’m testing it because it has native vision and audio. But I wouldn’t spend too much time on it because DSV4 GA is expected soon.
Saw it, tempted to download and get it going. DSV4 is good, but the lack of vision is negative for my use case. If DSV4 GA has a vision stack, that’d be cool. @j-montoya please update with results; curious how it goes!
DSFV4 takes up a LOT less memory when the context window starts to fill up. So for just average agentic work DSFV4 is still king
Support has just landed in Spark-vllm-docker: GitHub - eugr/spark-vllm-docker: Docker configuration for running VLLM on dual DGX Sparks · GitHub
./run-recipe.sh inkling-small-nvfp4
It’s still experimental - looks like there are some issues with tool calling in the current build.
I develop games using Unity, and one of the things I have always found disappointing about DeepSeek is its lack of vision capabilities.
This is especially limiting when using Unity’s official MCP, which includes a step where the model captures the game screen and checks the current state on its own. DeepSeek has clear limitations at this stage, making it difficult to take full advantage of the game engine’s MCP functionality.
Of course, it is possible to serve a separate multimodal vision-language model as a vision tower in front of DeepSeek. However, this approach is noticeably less accurate than using a model with native multimodal capabilities.
For people like me who need an LLM with accurate, built-in vision capabilities, I think this is a very promising model.
Thanks!
Yup, general chat with it works fine, but tool calls are spitting out <|content_invoke_tool_json|> tags as content.
I suggest we wait until my pipeline finishes and retest on the most recent build. My current build is 2 days old, so it may be missing some important fixes.
I am in the same boat with Unity and mcp. Without much vision options on a dual spark setup i usually try to get around with the Qwen 3.5 122B using FP8 quant - it can get things done in unity. Qwen 3.5 397B has better visual encoder but only lower quant would fit on a dual spark and so its worse than the 122B FP8 . With that said I also looking forward for a deepseek level inteligence + vision capability this might be indeed a good candidate to test.
IMJONEZZ’s PR on your (@eugr_nv) spark-vllm-docker repository fixed Inkling for me, in order to use it with OpenClaw.
I was a bit disappointed with Inlking Small’s LLM capabilities in OpenClaw. To me it seems very talkative and much prone to looping. I could not yet test its multimodality, which is its main advantage.
But for pure text-agentic AI use, and similar resources, to me DeepSeek V4 Flash 0731 is the much better choice - which I run with sparkrun and the recipe from Add sparkrun recipe for one-command reproducible deployment (closes #9) by jammsen · Pull Request #14 · tonyd2wild/DeepSeek-v4-Flash-0731-DSpark-1M-NVFP4-KV-2x-DGX-Spark · GitHub.
Anyone using this model extensively? I have and am using the recipe from eugr.. The model is very slow even on medium thinking level, it burns tokens. It eventually gets to the right answer, but for the amount of thinking it does, it makes me wish DS4-Flash had vision. :(
Have you tried https://github.com/FlyCockpit/DeepSeek-V4-Vision-2x-DGX-Sparks ?
I have also created spark-vllm recipe for it spark-vllm-docker/recipes/deepseek-v4-flash-0731-vision.yaml at main · hacibeyoglu/spark-vllm-docker · GitHub
The bugs with tool calling and proper handling of reasoning sections had not been fixed in vLLM until the last two days. Even the community made patches did not truly fix the issue, the one by tonyd2wild was better. There is also a problem with CUDA crashes with certain outputs, which was fixed by turning on CUDA codedumping which is discussed on the forum
I’ve used it a lot and it is a very good generalist model, but the bugs are a problem for agentic use and can derail it. Will try the newer vLLM version now and see
Appreciate the DS4 vision tower tip, but I generally have had bad experiences with community contributions where vision is tacked onto the LM after training.
For our case, we are using Inkling-Small in non-agentic scenario (using it to write prose based on some info given) - tool calling hasn’t be a problem yet… Its just the inconsistent thinking patterns that have bit!
After using the NVFP4 quant of the model more, it is excellent for analysis but the vision capability is somewhat compromised. Inkling defaulted to a large patch size by design and instead they state that the model should be given python tools to crop and enlarge images for closer examination.
Also in agentic use it tends to commonly state “in the interest of time,” “given time,” “to save time” or something like that and then immediately conclude its current task or take a shortcut instead of completing the job. Not sure if it is related to the bugs or quant but it is frustrating.
In a chat format it tends to think a lot, ironically
Inkling Small NVFP4 has become my go-to model but vLLM crashes out on occasion, or won’t respond to some requests, but as suggested it’s early in it’s release and hopefully will improve. I use it with --language-model-only since I have no use for vision or other media capabilities.
It is kind of crazy how little support this model has gotten. Llamacpp still has not merged in support for it (Add TML Inkling architecture by danielhanchen · Pull Request #25731 · ggml-org/llama.cpp · GitHub)