Two days ago I released the first GGUF build of Nemotron-3-Omni with working
audio and video ( Nemotron-3-Omni with audio + video in llama.cpp on DGX Spark (GGUF, prebuilt binaries) ). Then,
something stranger: I gave NVIDIA’s Nemotron 3.5 Lightning eyes and ears
WITHOUT ANY TRAINING, and it works.
The trick is architectural: Lightning shares the exact backbone geometry of
Nemotron-3-Nano-Omni (nemotron_h, hidden 2688, 52 layers, same vocabulary).
NVIDIA trained the Omni projectors to feed precisely that geometry, so the
C-RADIO vision tower and the Parakeet audio tower attach to Lightning
zero-shot. The result: a 1M-context, 3B-active agent model that reads
images, understands speech, and watches video, at the same decode speed as
the donor, entirely on a DGX Spark on my desk.
What is released, and how it is tested:
- Image, audio, and video understanding: released and gated. Every
quantization passes text, image, audio, and video tests before upload;
the graded matrix ships in the repository. - Measured transfer cost: on my 38-word ASR gate, the donor Omni scores
2.6 percent word error rate, the zero-shot assembly 7.9 percent (rare
proper nouns). Stated at exactly that scope. - One-pass video WITH soundtrack: works zero-shot, with one prompting
note. I chased this through two wrong hypotheses (temporal-pathway drift,
then irreversible audio dominance) and measurement killed both. What
actually happens: video-only works at every length (2 to 64 frames,
matches the donor), and in one pass with a soundtrack both channels are
fully present. With a neutral single-subject prompt (“what is in the
video?”) Lightning tends to answer from the audio and under-report the
picture. Ask for BOTH and it returns both, every time I tested, temp 0:
“Describe what is shown on screen AND transcribe what the narrator says”
gives the full visual description and the full transcription in one
answer. So it is a released capability with a prompting note, not a
broken feature. The donor Omni, co-trained on interleaved A/V, balances
both even at a neutral prompt; a future Lightning projector tune could
shift that default, but it is polish.
Downloads (quantizations land one by one as each passes its gates; the
matrix grows with them):
Runtime: the same llama.cpp-omni fork as the first release, now rebased
onto current master (release b10598, prebuilt DGX Spark / Jetson Thor
binaries). Image, text and audio also run on stock llama.cpp with these
files:
The narration in my test suite is synthesized by NVIDIA’s Magpie TTS
running as a NIM on the same Spark. Their voice, their models, their desk
computer; the assembly and the graded testing are mine. Happy to answer
questions or fix what breaks.