Originally published at: Full-Stack Optimizations for Agentic Inference with NVIDIA Dynamo | NVIDIA Technical Blog
Coding agents are starting to write production code at scale. Stripe’s agents generate 1,300+ PRs per week. Ramp attributes 30% of merged PRs to agents. Spotify reports 650+ agent-generated PRs per month. Tools like Claude Code and Codex make hundreds of API calls per coding session, each carrying the full conversation history. Behind every one…
Interesting work.
I have one quick question:
On the dynamo side, when routing requests, will they consider the different configurations of the inference engine instances? For example, the same degree of parallelism but different TP sizes would yield different performance? Would this situation be considered?
Yes, that’s an important consideration. On the Dynamo side, the router should ideally account for the specific configuration of each inference-engine instance, not just the overall degree of parallelism. Two instances with the same parallelism but different TP sizes can have noticeably different throughput and latency characteristics. So, the routing decision should consider these configuration differences when estimating capacity and selecting the appropriate instance.