Llama 3 8b hardware

Accelerated Computing GPU-Accelerated Libraries
1

What would it take to make 10k inferences per second of more than 2000 input tokens and 50 output tokens?

2

Data here may be of interest.