Anyone ran the fp8 version of Qwen3.8 Flash on dual dgx sparks? Everywhere I’m seeing nvfp4 recipes for deploying this model on dual sparks.
I also don’t get how so many people are happy with NVFP4 results.
I give this quant another try every few months, but the quality is just not good enough, both on simple and complex tasks. For my use cases (coding, RAG, image description), NVFP4 is actually even worse than INT4 (AWQ).
someone posted nvfp4 results of tool-eval-bench - it was not exciting
I just ran the benchmarks, with the VLLM recommended settings + --ensure-eager to have it load.
Didn’t run it more than just the benchmark.
But I don’t seem to get as good results as other people with NVFP4 gets, but I’ve just ran the vllm/vllm-openai:qwen38-flash-next through eugrs setup with the --use-official-vllm for instanttensors
I feel like it still needs some time to mature.
pay attention to the config they are running - you running a newest 88-question hardmode with 177 point, others are running 138-point mid-level test. Of course they are getting 89/100 - 15 easy question test will get you 98/100 and you can post it too :)
Good point :)
I’m just assuming everyone runs hardmode… that’s probably my bad ^^
nah… it’s scary to run hardmore and dogfooding it - you shall never pick under the curtain if you are afraid what you may see there :D
like in military or business - never give an order you know won’t be followed
agreed
how have you deployed deepseek on dual sparks? I’m running it through tony’s repo, seeing model hallucinations as well as this keeps popping up time to time on logs:
dgx spark [0] transport/net_ib/common.cc:145 NCCL WARN NET/IB : roceP2p1s0f1:1 GID table changed
I’d assume he uses the repo he published, based on @aidendle94 docker:
Thanks, I’ll check it out
back to the topic - FP8 checkpoint should be fine to serve on 2 sparks, nvfp4 is not that much smaller. And serving stack should be the same or easier. Will pull the checkpoint later and spend my openai limit on massaging it. It’s mostly wasted anyways as ds4f does the job
That’s what I was thinking, I thought everyone would be running the fp8 after release since size fits, but everywhere I see, it’s just nvfp4. I guess everyone is chasing for few extra tok/s instead of quality which is weird.
From what I have seen - this is the recipe, looks like FP8 does not require JIT fixes to VLLM than NVFP4 needs
I’m just trying this out on 2 GB10… weights loading…
Please keep us posted ASAP! I dunno what is a problem but my download from HF is so throttled, HF token set, starts at full bandwidth but soon drops, then boosts. No idea. Pulling is very slow. Probably you will get a perfect recipe before I get the weights LOL :D
Please post your recipe!
I’m using @foogitiff worker/master scripts above (just had to change the IPs of the C7x interfaces, if you’re using the outermost connection for your cable).