"Can speculative decoding significantly enhance device performance in low-concurrency inference scenarios?
Given that the device boasts approximately 1,000 TFLOPS of sparse computing power, the theoretical computation latency for generating a single token with a Qwen-3-32B class model is merely around 0.064 ms. This indicates that the primary bottleneck during the decoding phase is the time required to read parameter weights (i.e., the system is memory-bound).
If we adopt speculative decoding with a smaller draft model to leverage this high computing capability, is it theoretically possible to achieve a 2x to 3x performance gain?"
Feasibility of 2-3x Speedup via Speculative Decoding on High-Compute (1000 TFLOPS) / Low-BW Hardware
Lab to support speculative decoding on the spark
Thread with lots of performance numbers for inference in single and cluster mode: