Issue with connection to 2 dgx sparks. vllm

[artis1]

I’m trying to connect two dgx sparks using them to run larger models but currently running into issues.

[eugr]

There have been numerous posts about it. This one, for example - you can use my repo or Mark’s - his is a little bit more plug-n-play, I believe, but obscures some parameters. Mine is for more flexibility, but may not be as beginner-friendly. Also, my repo builds from the main branch, Mark’s solution uses official VLLM docker from NVidia (which lags behind in model support and performance).

Install and Use vLLM for Inference on two Sparks does not work

The vLLM scripts have been updated and standardized to support switching between a dozen or so models easily and even running a benchmark against all of the models in the list. The scripts are also now setup to run everything from the head node without the need to manually run a script on the worker. Also tried to clean up the setup process and the setting of the environment variables. Also, since @eugr shared that SGLang can be faster than vLLM for running the models, we created the same s…