Nemotron-3.5 Super VL: Key Failure Modes & Path to 100% Reliability

We recently evaluated the hosted API of Nvidia Nemotron-3.5 Super VL across real-world physical verification tasks (robotics assembly and kinetic movement counting), based on the benchmarks published in this analysis.

While the model excels at single-frame classification and basic visual question answering, deploying it end-to-end for zero-shot operational sign-off (e.g., warehouse quality assurance, robotics verification, exercise telemetry) reveals critical reliability bottlenecks. Below is an overview of the observed failure patterns and a proposed hybrid architecture designed to achieve 99.9% production accuracy.

Observed Failure Modes

                  ┌────────────────────────────────────────┐
                  │    Nemotron-3.5 Super VL Video Input   │
                  └───────────────────┬────────────────────┘
                                      │
            ┌─────────────────────────┴────────────────────────┐
            ▼                                                  ▼
 1. Internal Logic Contradiction                    2. Temporal Hallucination
 • Observed: Step 1 Missing [00:02:45]             • Counted: 3 Reps (True: 2) [00:05:07]
 • Verdict: "Verified / Success" [00:03:03]        • Rep 1: 0–1s (Static Standing) [00:05:27]
 └── Disconnect between evidence & verdict         └── Over-confident ungrounded claims [00:05:37]

1. Internal Logic Contradiction (Evidence vs. Verdict Disconnect)
  • Scenario: Robot lamp assembly checklist (Move Base → Place Bulb → Lower Shade).
  • Observed Failure: At [02:45], the model correctly reported in its textual breakdown that the robot base movement was unobserved or stationary (0–30s). However, at [03:03], its final synthesized verdict concluded that the entire procedure was successfully verified.
  • Root Cause: In standard autoregressive generation without token-level decoding constraints, the verdict token prediction is easily biased by general task priors (e.g., assuming typical success in demonstration videos) rather than strictly d-separating and deducing from its own generated intermediate evidence list.
2. Temporal Hallucination & Phantom Action Windows
  • Scenario: Squat count and kinetic cycle identification.
  • Observed Failure: The model reported 3 complete repetitions when only 2 occurred [05:07]. It hallucinated Rep 1 between 00:00:00 and 00:00:01 [05:27], a timestamp window where the subject was merely adjusting their stance.
  • Root Cause: Black-box frame subsampling at the API level creates aliasing over rapid or ambiguous transition frames, leading the temporal attention heads to register micro-movements as completed cycles.
3. Confident Hallucination Under Occlusion
  • Scenario: Kinetic alignment verification (barbell tracking and heel contact).
  • Observed Failure: At [05:37], the model asserted that the lifter’s heels remained flat on the floor and the barbell stayed over the midfoot throughout the movement, despite the feet and rack baseline being visually occluded and cut off by the frame geometry.
  • Root Cause: Lack of calibrated visual epistemic uncertainty; the model defaults to normative expectations of “good form” when visual evidence is absent rather than returning an unresolved_due_to_occlusion status.
4. API Vulnerability Under Negative-Constraint Prompts
  • When prompting with strict fallback schemas (“Output ‘verified’ only if all steps contain explicit visual proof, else ‘insufficient_evidence’”), the API frequently threw HTTP 500 server errors [03:48], indicating friction in handling strict refusal/rejection logic under current hosted parameters.

Engineering Roadmap: Achieving 100% Production Accuracy

To build production-grade automated sign-off pipelines with Nemotron-3.5 Super VL, the system cannot rely on raw end-to-end VLM verdicts. A layered verification architecture is required:

Layer Implementation Strategy Target Failure Addressed
1. Programmatic Guardrails Deterministic AST/schema validation in Python; enforce that len(unobserved_steps) == 0 before verdict is accepted. Evidence-to-verdict contradictions [03:03].
2. Dynamic Temporal Chunking Split video into candidate micro-windows (via scene-change or motion energy) before querying the VLM. Early-frame omissions and phantom rep timestamps [05:27].
3. Multi-Modal Pose Fusion Pipe raw frames through lightweight pose estimators (e.g., MMPose / MediaPipe) to compute joint velocity curves, using Nemotron for context only. False cycle counts and kinematics hallucination under occlusion [05:37].
4. Negative Constraint Calibration Require the model to output visibility scores [0.0 - 1.0] per key object prior to generating assertions. Hallucinating occluded limbs or hidden sub-assemblies.

Open Questions for the NVIDIA Team

  1. Frame-Sampling Transparency: Does the hosted Nemotron-3.5 Super VL endpoint downsample dynamically based on duration, or does it enforce a fixed frame budget (e.g., 32/64 frames)? Can we configure explicit frames-per-second (FPS) parameters via the inference API?
  2. Logit/Grammar-Constrained Decoding: Are there upcoming plans to support constrained grammar decoding (e.g., via Guidance, Outlines, or NeMo Guardrails) directly on the API endpoint to mathematically enforce consistency between intermediate evidence arrays and final boolean verdicts?
  3. Calibrated Uncertainty Outputs: Are token-level log-probabilities or explicit occlusion flags planned for video-domain feature releases?