So when I first set up my machine, --kv-cache-dtype fp8 was the default in all my recipes. I had no understanding that bf16 was an option worth exploration. The discussions here, no doubt influenced by the Nvidia DGX Spark marketing was that we should be aiming for --kv-cache-dtype NVFP4. Then I had a discussion with @eugr and @flash3 about all the tool call failures I was experiencing and the eventual outcome was a suggestion to use bfloat16 with int4 AutoRound (which I had never heard of) with Qwen3 Coder Next. This was long before AutoRound had gained popularity. At the time vllm support for this configuration seemed nascent.
Back in Feb the dominant conversations on this forum was almost exclusively focused on t/s. I found this a real time sink because almost everything I tried couldn’t sustain long contexts, instruction following and tool calls. Then Qwen3.5 dropped and while I was attempting to quantise it to AutoRound myself, Intel released their versions in short succession.
It takes a lot of time to download models, wait 10 minutes for vllm to startup, run a 30 min process, tweak a setting, wait 10 minutes for vllm to startup again, re-run a 30 min process, compare results - rinse and repeat.
It may seem perfunctory now to be discussing these settings, but back in Feb this really wasn’t a conversation that was being had – at least not one visible to me. I had clients who needed deliverables and I was lost down endless rabbit holes getting nowhere – mad how fast moving this has been.
I started this thread because I figured other members might also be tired of wasting time and getting nowhere too. I have learned a lot from everyone in the meantime. I have a daily driver setup now that I am satisfied with, and still keenly experimenting with PrismaQuant v2 – so I guess a foot in each camp ;)