Building Autonomous Vehicles That Reason with NVIDIA Alpamayo

Technical Blogs & Events Technical Blog
1

Originally published at: Building Autonomous Vehicles That Reason with NVIDIA Alpamayo | NVIDIA Technical Blog

Autonomous vehicle (AV) research is undergoing a rapid shift. The field is being reshaped by the emergence of reasoning-based vision–language–action (VLA) models that bring human-like thinking to AV decision-making. These models can be viewed as implicit world models operating in a semantic space, allowing AVs to solve complex problems step-by-step and to generate reasoning traces…

2

Title: Observation: CoC/Trajectory Mismatch in AlpaSim Closed-Loop

Hi NVIDIA Team,

I’ve been working through the Alpamayo ecosystem and wanted to share an observation from my testing.

Setup:

  • Alpamayo R1 10B with 4-camera configuration
  • AlpaSim with REASONING_OVERLAY
  • Tested on DGX Spark (ARM64), Lambda A100, and Lambda A10
  • Both distributed (remote brain via gRPC tunnel) and local configurations

What works: Part 1 of the tutorial works perfectly. Running inference.py on PhysicalAI-AV dataset clips, the model produces correct trajectories with aligned Chain-of-Causation reasoning. Tested 4 random clips — minADE ranged 0.48m to 2.56m. The model clearly understands the scenes.

My observation: In Part 4 (closed-loop AlpaSim), I’m seeing a disconnect between the CoC reasoning and trajectory execution:

  • Scene clipgt-05bb8212: CoC outputs “Nudge left to pass the parked car” multiple times, but trajectory goes straight/right → collision_rear, wrong_lane
  • Scene clipgt-fa0379c6: CoC outputs “Accelerate through intersection” (correct for green light), but trajectory deviates → offroad, plan_deviation 4.91m

Same behavior across all hardware configs tested, so it doesn’t appear to be hardware or latency related.

1-camera test: Switched to 1-camera config on the same scene. No collision, but CoC said “Keep lane” while trajectory curved left into a gas station → offroad.

Summary: The model reasons correctly in open-loop (Part 1) but the reasoning/trajectory alignment breaks in closed-loop simulation (Part 4). It appears to be something in the model-simulation integration.

Full details, metrics, CoC logs, and reproduction steps in my GitHub issue: [Alpamayo R1] Critical VLA Alignment Failure: Reasoning & Trajectory Heads Decoupled · Issue #20 · NVlabs/alpasim · GitHub

Happy to provide additional testing or information. Thanks for making this ecosystem open — it’s been a great learning experience.