Getting Nemotron embed working on DGX Spark

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
, , ,
1

I have been trying to get the Nemotron-embed model running on my DGX Spark to set up a local RAG server experiment.

After trying to use the official NGC containers which failed due to drivers incompatibility (580 vs 590) I tried to use the vLLM containers and build the code from HF.

I testes the vLLM container with the models already built in the container so I can confirm the officinal vLLM container does work correctly. However, all the models in the vLLM are for reasoning and none for embedding.

After multiple attempts to build the model, cuda driver errors (which lead me to use the vLLM image).

However, it seems that the dependencies in the vLLM container don’t work well with this model. There are loading errors on the transformer package version.

This is the setup I am following:

docker run -d \
  --name nemotron-embed \
  --gpus all \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HF_HOME=/root/.cache/huggingface \
  --shm-size=16g \
  nvcr.io/nvidia/vllm:25.12-py3 \
  vllm serve nvidia/llama-nemotron-embed-1b-v2 \
    --trust-remote-code \
    --runner pooling \
    --override-pooler-config '{"pooling_type": "MEAN"}' \
    --dtype float32 \
    --port 8000 \
    --host 0.0.0.0

any ideas on how to get this going?

I am pretty new to this space and trying to use this personal project as a learning platform

2

I haven’t tested this model, but have you followed all these steps (specifically, the config one)? You don’t have to use vLLM 0.11.0, anything above this is fine (including official NVIDIA NGC container or our community build).

vLLM Usage

  1. Ensure you are using vllm==0.11.0.
  2. Clone this model’s repository.
  3. Overwrite config.json with config_vllm.json.
  4. Start the vLLM server with the following command (replace the <path_to_the_cloned_repository> and <num_gpus_to_use> with your values):
vllm serve \
    <path_to_the_cloned_repository> \
    --trust-remote-code \
    --runner pooling \
    --model-impl vllm \
    --override-pooler-config '{\"pooling_type\": \"MEAN\"}' \
    --data-parallel-size <num_gpus_to_use> \
    --dtype float32 \
    --port 8000

You can now access the model using the OpenAI sdk, for instance:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1")
models = client.models.list()
model_name = models.data[0].id

response = client.embeddings.create(
    input=['query: summit define'],
    model=model_name
)
response.data[0].embedding
3

Thanks @eugr I found a couple of problems. The documentation for the official NGC image for vLLM claims that it has the 0.11 version of vLLM and in fact it has the 0.13 version which lead to all the errors about the transformers 0.5 dependency.

so @nvidia should probably fix their documentation for the image vLLM | NVIDIA NGC for any posterity, the container with vllm 0.11 is the 25.12-py3.

I tried the command inside the container

vllm serve nvidia/llama-nemotron-embed-1b-v2 \
    --trust-remote-code \
    --runner pooling \
    --model-impl vllm \
    --override-pooler-config '{"pooling_type": "MEAN"}' \
    --dtype float32 \
    --port 8000 \
    --host 0.0.0.0

and got this error

(APIServer pid=481)   Value error, Model architectures ['LlamaBidirectionalModel'] are not supported for now. Supported architectures: dict_keys(['AfmoeForCausalLM', 'ApertusForCausalLM', 'AquilaModel', 'AquilaForCausalLM', 'ArceeForCausalLM', 'ArcticForCausalLM', 'BaiChuanForCausalLM', 'BaichuanForCausalLM', 'BailingMoeForCausalLM', 'BailingMoeV2ForCausalLM', 'BambaForCausalLM', 'BloomForCausalLM', 'ChatGLMModel', 'ChatGLMForConditionalGeneration', 'CohereForCausalLM', 'Cohere2ForCausalLM', 'CwmForCausalLM', 'DbrxForCausalLM', 'DeciLMForCausalLM', 'DeepseekForCausalLM', 'DeepseekV2ForCausalLM', 'DeepseekV3ForCausalLM', 'DeepseekV32ForCausalLM', 'Dots1ForCausalLM', 'Ernie4_5ForCausalLM', 'Ernie4_5_MoeForCausalLM', 'ExaoneForCausalLM', 'Exaone4ForCausalLM', 'Fairseq2LlamaForCausalLM', 'FalconForCausalLM', 'FalconMambaForCausalLM', 'FalconH1ForCausalLM', 'FlexOlmoForCausalLM', 'GemmaForCausalLM', 'Gemma2ForCausalLM', 'Gemma3ForCausalLM', 'Gemma3nForCausalLM', 'Qwen3NextForCausalLM', 'GlmForCausalLM', 'Glm4ForCausalLM', 'Glm4MoeForCausalLM', 'GptOssForCausalLM', 'GPT2LMHeadModel', 'GPTBigCodeForCausalLM', 'GPTJForCausalLM', 'GPTNeoXForCausalLM', 'GraniteForCausalLM', 'GraniteMoeForCausalLM', 'GraniteMoeHybridForCausalLM', 'GraniteMoeSharedForCausalLM', 'GritLM', 'Grok1ModelForCausalLM', 'HunYuanMoEV1ForCausalLM', 'HunYuanDenseV1ForCausalLM', 'HCXVisionForCausalLM', 'InternLMForCausalLM', 'InternLM2ForCausalLM', 'InternLM2VEForCausalLM', 'InternLM3ForCausalLM', 'JAISLMHeadModel', 'JambaForCausalLM', 'KimiLinearForCausalLM', 'Lfm2ForCausalLM', 'Lfm2MoeForCausalLM', 'LlamaForCausalLM', 'Llama4ForCausalLM', 'LLaMAForCausalLM', 'LongcatFlashForCausalLM', 'MambaForCausalLM', 'Mamba2ForCausalLM', 'MiniCPMForCausalLM', 'MiniCPM3ForCausalLM', 'MiniMaxForCausalLM', 'MiniMaxText01ForCausalLM', 'MiniMaxM1ForCausalLM', 'MiniMaxM2ForCausalLM', 'MistralForCausalLM', 'MixtralForCausalLM', 'MptForCausalLM', 'MPTForCausalLM', 'MiMoForCausalLM', 'NemotronForCausalLM', 'NemotronHForCausalLM', 'OlmoForCausalLM', 'Olmo2ForCausalLM', 'Olmo3ForCausalLM', 'OlmoeForCausalLM', 'OPTForCausalLM', 'OrionForCausalLM', 'OuroForCausalLM', 'PanguEmbeddedForCausalLM', 'PanguUltraMoEForCausalLM', 'PersimmonForCausalLM', 'PhiForCausalLM', 'Phi3ForCausalLM', 'PhiMoEForCausalLM', 'Plamo2ForCausalLM', 'QWenLMHeadModel', 'Qwen2ForCausalLM', 'Qwen2MoeForCausalLM', 'Qwen3ForCausalLM', 'Qwen3MoeForCausalLM', 'RWForCausalLM', 'SeedOssForCausalLM', 'Step3TextForCausalLM', 'StableLMEpochForCausalLM', 'StableLmForCausalLM', 'Starcoder2ForCausalLM', 'SolarForCausalLM', 'TeleChat2ForCausalLM', 'TeleFLMForCausalLM', 'XverseForCausalLM', 'Zamba2ForCausalLM', 'BertModel', 'BertSpladeSparseEmbeddingModel', 'Gemma2Model', 'Gemma3TextModel', 'GPT2ForSequenceClassification', 'GteModel', 'GteNewModel', 'InternLM2ForRewardModel', 'JambaForSequenceClassification', 'LlamaModel', 'MistralModel', 'ModernBertModel', 'NomicBertModel', 'Qwen2Model', 'Qwen2ForRewardModel', 'Qwen2ForProcessRewardModel', 'RobertaForMaskedLM', 'RobertaModel', 'XLMRobertaModel', 'CLIPModel', 'LlavaNextForConditionalGeneration', 'Phi3VForCausalLM', 'Qwen2VLForConditionalGeneration', 'SiglipModel', 'PrithviGeoSpatialMAE', 'Terratorch', 'BertForSequenceClassification', 'BertForTokenClassification', 'GteNewForSequenceClassification', 'ModernBertForSequenceClassification', 'ModernBertForTokenClassification', 'RobertaForSequenceClassification', 'XLMRobertaForSequenceClassification', 'JinaVLForRanking', 'AriaForConditionalGeneration', 'AyaVisionForConditionalGeneration', 'BeeForConditionalGeneration', 'Blip2ForConditionalGeneration', 'ChameleonForConditionalGeneration', 'Cohere2VisionForConditionalGeneration', 'DeepseekVLV2ForCausalLM', 'DeepseekOCRForCausalLM', 'DotsOCRForCausalLM', 'Ernie4_5_VLMoeForConditionalGeneration', 'FuyuForCausalLM', 'Gemma3ForConditionalGeneration', 'Gemma3nForConditionalGeneration', 'GLM4VForCausalLM', 'Glm4vForConditionalGeneration', 'Glm4vMoeForConditionalGeneration', 'GraniteSpeechForConditionalGeneration', 'H2OVLChatModel', 'InternVLChatModel', 'NemotronH_Nano_VL_V2', 'InternS1ForConditionalGeneration', 'InternVLForConditionalGeneration', 'Idefics3ForConditionalGeneration', 'SmolVLMForConditionalGeneration', 'KeyeForConditionalGeneration', 'KeyeVL1_5ForConditionalGeneration', 'RForConditionalGeneration', 'KimiVLForConditionalGeneration', 'LightOnOCRForConditionalGeneration', 'Llama_Nemotron_Nano_VL', 'Llama4ForConditionalGeneration', 'LlavaForConditionalGeneration', 'LlavaNextVideoForConditionalGeneration', 'LlavaOnevisionForConditionalGeneration', 'MantisForConditionalGeneration', 'MiDashengLMModel', 'MiniMaxVL01ForConditionalGeneration', 'MiniCPMO', 'MiniCPMV', 'Mistral3ForConditionalGeneration', 'MolmoForCausalLM', 'NVLM_D', 'Ovis', 'Ovis2_5', 'PaddleOCRVLForConditionalGeneration', 'PaliGemmaForConditionalGeneration', 'Phi4MMForCausalLM', 'Phi4MultimodalForCausalLM', 'PixtralForConditionalGeneration', 'QwenVLForConditionalGeneration', 'Qwen2_5_VLForConditionalGeneration', 'Qwen2AudioForConditionalGeneration', 'Qwen2_5OmniModel', 'Qwen2_5OmniForConditionalGeneration', 'Qwen3OmniMoeForConditionalGeneration', 'Qwen3VLForConditionalGeneration', 'Qwen3VLMoeForConditionalGeneration', 'SkyworkR1VChatModel', 'Step3VLForConditionalGeneration', 'TarsierForConditionalGeneration', 'Tarsier2ForConditionalGeneration', 'UltravoxModel', 'VoxtralForConditionalGeneration', 'WhisperForConditionalGeneration', 'MiMoMTPModel', 'EagleLlamaForCausalLM', 'EagleLlama4ForCausalLM', 'EagleMiniCPMForCausalLM', 'Eagle3LlamaForCausalLM', 'LlamaForCausalLMEagle3', 'Eagle3Qwen2_5vlForCausalLM', 'EagleDeepSeekMTPModel', 'DeepSeekMTPModel', 'ErnieMTPModel', 'LongCatFlashMTPModel', 'Glm4MoeMTPModel', 'MedusaModel', 'OpenPanguMTPModel', 'Qwen3NextMTP', 'SmolLM3ForCausalLM', 'Emu3ForConditionalGeneration', 'TransformersForCausalLM', 'TransformersMoEForCausalLM', 'TransformersMultiModalForCausalLM', 'TransformersMultiModalMoEForCausalLM', 'TransformersEmbeddingModel', 'TransformersMoEEmbeddingModel', 'TransformersMultiModalEmbeddingModel', 'TransformersForSequenceClassification', 'TransformersMoEForSequenceClassification', 'TransformersMultiModalForSequenceClassification']) [type=value_error, input_value=ArgsKwargs((), {'model': ...rocessor_plugin': None}), input_type=ArgsKwargs]

which leads me to believe this container might not the appropriate to build the model

4

ok this is what made it work

I loaded the bash container and made the changes to config.json directly as follows:

# Inside the container or with the cache mounted
docker run -it --rm \
  --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v ~/models:/models \
  nvcr.io/nvidia/vllm:25.12.post1-py3 \
  bash

Then cloned the model into the models folder

cd /models
hf download nvidia/llama-nemotron-embed-1b-v2 --local-dir /models/llama-nemotron-embed-1b-v2

then did

mv config.json config.json.bk
mv config_vllm.json config.json
jq '. + {"model_impl": "vllm"}' config.json > config_tmp.json && mv config_tmp.json config.json

and I ran the container with this command following the instructions from the repo and this conversation

docker run -d \
  --name nemotron-embed \
  --gpus all \
  -p 8000:8000 \
  -v ~/models:/models \
  --shm-size=16g \
  nvcr.io/nvidia/vllm:25.12-py3 \
  vllm serve /models/llama-nemotron-embed-1b-v2 \
    --trust-remote-code \
    --runner pooling \
    --pooler-config '{"pooling_type": "MEAN"}' \
    --dtype float32 \
    --port 8000 \
    --model-impl vllm \
    --data-parallel-size 1 \
    --host 0.0.0.0