We just published a page on how to deploy Nemotron 3 Ultra across 3 DGX Sparks!
Built, in large part, by the one - the only - @eugr_nv !!!
0,1,2,3 = 4?
Is the context length of 256k the max limit
It can fit ~600K on 4x Sparks with these parameters, but we went with 256K to be on the safe side as it really depends on how much stuff you have running there.
Thanks @eugr_nv I had run the recipe as a first thing with 256k contextafter I got my 4 node cluster last week. I will try the 512k and see how it goes :)
Thanks for these instructions, @eugr_nv: works perfectly on my 4x cluster. Testing it with Hermes now.
So I’ve been messing around with this setup this evening, and unfortunately it is not stable for me with Hermes. Within a few minutes, one or two of my Sparks will OOM with this message on the head node repeating on vLLM:
(EngineCore pid=100) INFO 06-26 05:24:31 \[shm_broadcast.py:705\] No available shared memory broadcast block found in 60 seconds. This typically happens when some processes are hanging or doing some time-consuming work (e.g. compilation, weight/kv cache quantization).\
A message like the following shows up in the crashed nodes:
Jun 25 22:46:32 gn100-abcd kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:135
Thanks for the feedback. Do you have anything else running on those nodes?
I’ll see if I can reproduce this on my end with some agentic workloads.
Thanks for taking a look!
For the nodes running other things, no – I even rebooted them and put them in multi-user.target to maximize the free VRAM. So far, these settings seem stable (not scientific – just changed several params at once in attempt to avoid OOM):
gpu_memory_utilization: 0.83
max_model_len: 131072
max_num_seqs: 1
max_num_batched_tokens: 8192
By the by, I should have mentioned: the AIPerf benchmark shown in the guide worked great with the yaml as-is (my numbers were very similar to the screenshot on GitHub).
I see, thanks. Yes, these are very conservative settings, I would not expect OOMs.
One other thing worth checking - what driver version is reported by nvidia-smi?
Of course. Just checked nvidia-smi: 580.159.03 across all the nodes.
Edit: minor update. The conservative settings I posted above are very stable. I’m going to start raising them back and seeing how far they’ll go. The only other minor issue I’m having is related to the occasional tool calling hangup in Hermes, but that’s likely a configuration or settings issue on the Hermes side from what I can tell so far.
Thanks again for putting together this guide – this is a really nice model size for the Sparks!
@eugr_nv: A small update, as I’m actually still diagnosing the stability issue I noticed. Everything about your image works, but the issue is related to a slow vLLM memory leak (maybe a dozen MB every few minutes), which I’ve traced down to the individual TP workers (i.e., VLLM::Worker_TP). Long story short, despite the GPU utilization being fixed as expected, the vLLM workers are slowly eating up host memory (observable even via nvtop) only during active inferencing, which eventually leads to an OOM (which earlyoom thankfully catches at the node level). Interestingly, on my 4 Spark cluster, it’s one or two of the nodes that reliably go out first (although if you change the ordering, it changes which Spark it is, meaning it’s not hardware related). Two of the four always use a few GB more of host memory during a run (e.g., right now, I see 12525MiB, 15173MiB, 12442MiB, 15197MiB across the four [and each continues to grow, e.g., while writing this post, the first one went from 12525 → 12607]).
I’m continuing to diagnose the root cause (which may be related to prefix caching [although posts like this one from Mistral make me feel it may not be so obvious]). In any case, it can be observed by monitoring any of the spark-vllm-docker images during active inferencing (I haven’t observed the growth with idle, but I haven’t looked closely):
while true; do
date +%T
docker exec vllm_node bash -lc '
for pid in $(find /proc -maxdepth 1 -regex "/proc/[0-9]+" -printf "%f\n"); do
[ -r /proc/$pid/smaps_rollup ] || continue
name=$(cat /proc/$pid/comm 2>/dev/null)
anon=$(awk "/^Pss_Anon:/ {print \$2}" /proc/$pid/smaps_rollup)
[ -n "$anon" ] &&
printf "%10.1f MiB pid=%-6s %s\n" \
"$(awk -v x="$anon" "BEGIN {print x/1024}")" \
"$pid" "$name"
done | sort -nr | head -15
'
sleep 60
done
Here is a sample of the output showing the growth in just a few minutes:
12:12:34
5052.3 MiB pid=122 VLLM::Worker_TP
897.9 MiB pid=55 vllm
5.5 MiB pid=121 python3
0.5 MiB pid=4168 sort
0.3 MiB pid=54 exec-script.sh
0.3 MiB pid=49 bash
0.3 MiB pid=34 bash
0.2 MiB pid=4167 bash
0.2 MiB pid=4159 bash
0.1 MiB pid=4169 head
0.1 MiB pid=4158 sleep
0.1 MiB pid=1 sleep
12:13:34
5074.3 MiB pid=122 VLLM::Worker_TP
897.9 MiB pid=55 vllm
5.5 MiB pid=121 python3
0.4 MiB pid=4230 sort
0.3 MiB pid=54 exec-script.sh
0.3 MiB pid=49 bash
0.3 MiB pid=34 bash
0.2 MiB pid=4229 bash
0.2 MiB pid=4221 bash
0.1 MiB pid=4232 head
0.1 MiB pid=4220 sleep
0.1 MiB pid=1 sleep
12:14:34
5106.4 MiB pid=122 VLLM::Worker_TP
897.9 MiB pid=55 vllm
5.5 MiB pid=121 python3
0.5 MiB pid=4292 sort
0.3 MiB pid=54 exec-script.sh
0.3 MiB pid=49 bash
0.3 MiB pid=34 bash
0.2 MiB pid=4291 bash
0.2 MiB pid=4283 bash
0.1 MiB pid=4294 head
0.1 MiB pid=4282 sleep
0.1 MiB pid=1 sleep
12:15:34
5127.4 MiB pid=122 VLLM::Worker_TP
897.9 MiB pid=55 vllm
5.5 MiB pid=121 python3
0.5 MiB pid=4354 sort
0.3 MiB pid=54 exec-script.sh
0.3 MiB pid=49 bash
0.3 MiB pid=34 bash
0.2 MiB pid=4353 bash
0.2 MiB pid=4345 bash
0.1 MiB pid=4356 head
0.1 MiB pid=4344 sleep
0.1 MiB pid=1 sleep
12:16:35
5168.5 MiB pid=122 VLLM::Worker_TP
897.9 MiB pid=55 vllm
5.5 MiB pid=121 python3
0.4 MiB pid=4416 sort
0.3 MiB pid=54 exec-script.sh
0.3 MiB pid=49 bash
0.3 MiB pid=34 bash
0.2 MiB pid=4415 bash
0.2 MiB pid=4407 bash
0.1 MiB pid=4417 head
0.1 MiB pid=4406 sleep
0.1 MiB pid=1 sleep
(In a few more GB worth of growth, eventually one worker will drop off, causing the entire session to fail.)
Thanks, that’s helpful. There is a current vLLM issue with prefix caching for Mamba models that we are looking into currently, could be related to it.