defaults: max_num_batched_tokens: 16384 max_model_len: 262144 host: 0.0.0.0 num_speculative_tokens: 3 gpu_memory_utilization: 0.65 max_num_seqs: 16
I’m working with this. Normally I spin up 8 agents (+ the orchestrator) so 16 is more than enough.. The rest is the same from the linked recipe.
Check my post here where I show/talk quality. 122b is still king, but this is good and fast, read on the hallucinations portion more than anything.