Originally published at: DynoSim: Simulating the Pareto Frontier | NVIDIA Technical Blog
Modern LLM serving is hard to tune because each deployment is a stack of interacting choices: model backend, tensor-parallel shape, prefill/decode split, worker counts, scheduler settings, routing policy, KV cache behavior, autoscaling thresholds, and topology. Those choices interact across layers, and a local improvement can shift the bottleneck somewhere else. For larger models, even one…
This is interesting. It’s mentioned it runs on a MacBook… where is the app/source?
Answering myself: Part of dynamo repo… dynamo/docs/dynosim at main · ai-dynamo/dynamo · GitHub