A cluster of two nodes is configured in the OpenWebUI + vLLM + Ray stack worked stably on the nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 for one million tokens (–max-model-len 1048576) and I got another node for the opportunity to improve my cluster to 384GB URAM and + another GPU for better token performance per second!). The cluster work plan is as follows - for nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8 I want to use all the performance, that is, all 3 GPUs, since 8960 by 3 is not divided and the problem has not yet been solved (although it would be nice to configure this parameter - if it is not divisible by 3, then I would like to load the head a little less than 2986, while for workers you can give a little more than 2987, for example), while we are using two GPUs. The second model I want to keep in memory is nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16, but I want to run it on three GPUs. That is, the idea is simple - I want to use all GPUs for the text model, and if I make the next request for the visual model, I want not to unload the text model to make the request for the visual model, and not to unload the visual model, return to the text model without starting it again. It turns out I need something to switch the GPU between models… what are my possible options?
PS Cluster of three nodes now works through Mesh Networking with spark-vllm-docker and sparkrun.