Hi all,
Vol.1 to Vol.3 of this series covered the generation side on one RTX 5090: Llama 3.1 8B and Nemotron 3 Nano as NIMs, and the models you can put behind a NeMo Guardrails rail. Vol.4 is the retrieval side, the part of a RAG pipeline that decides what the model gets to read: the Nemotron embedding NIMs and the Llama Nemotron reranking NIMs, on public data sets, with every passage of the corpus instead of a reduced benchmark corpus, and with the serving rate measured the way NVIDIA’s own load tool measures it. It is written as the five decisions these numbers inform.
Everything here is one model on the card at a time, exact search on the CPU (no vector database, so the index is not a second variable), and public data only: BEIR SciFact and HotpotQA, and MIRACL Chinese and Japanese.
TL;DR: five things to decide before you deploy retrieval on one card
- Expect less than the benchmark on a real corpus, especially in Chinese.
nemotron-3-embed-1b(NIM 2.2.2, fp16) scored nDCG@10 0.662 on the reduced MIRACL zh corpus (81,309 passages) and 0.649 on the whole one (4,934,368); Recall@100 fell from 0.988 to 0.964. On Japanese the drop was small: 0.744 on 185,319 passages, 0.741 on 6,953,614. - Rerank the top 100.
llama-nemotron-rerank-1b-v2lifted nDCG@10 from 0.649 to 0.683 on MIRACL zh, from 0.741 to 0.771 on ja, and from 0.801 to 0.865 on BEIR HotpotQA (5,233,329 passages, 7,405 queries). The 500m reranker added less (+0.020 zh, +0.011 ja) and served 1.6 times the pairs per second at one request at a time. - Size the reranker, not the embedder. With 100 passages per query, the 1b reranker topped out at about 10 queries per second on this card and the 500m one at about 19; embedding one query topped out at 384 requests per second for
nemotron-3-embed-1b. On this card, most embedding NIMs stopped getting faster at 2 concurrent requests and the rerankers at 8–16; past that point more concurrency only adds waiting. These are each NIM alone on the card; two NIMs sharing one card will do less. - Store the vectors as fp16. Every value this NIM returns survives the round trip to 16-bit floats, so the three whole-corpus stores went from 140 GB to 70 GB with identical rankings. Stored as 8-bit integers, nDCG@10 and Recall@100 showed no loss on the three corpora, but the top 10 came back in the same order for only 33–58% of the queries.
- The NIM gives the quality of the open weights. On the reduced MIRACL task, the NIM of
nemotron-3-embed-1band the open weights differ by 0.0011 (zh) and 0.0001 (ja) nDCG@10. The public MTEB value for ja (0.742) is reproduced within 0.002; the public zh value (0.642) is not reproduced by our harness (0.663) or by MTEB itself run here (0.663), and we do not know why.
Two nights repeated 19 levels within 2.1%, and the load tool, checked against a server that answers instantly, went at least 2.99 times faster than any NIM it measured.
nDCG@10 and Recall@100 from pytrec_eval, exact cosine search over every passage. Rates are server-side, measured with AIPerf in a client container on the NIM’s Docker network, closed loop.
One RTX 5090 · one retrieval NIM on the card at a time · BEIR SciFact and HotpotQA, MIRACL zh and ja · exact search · AIPerf 0.11
The arms
| NIM | Image | Precision on this card | Used for |
|---|---|---|---|
nemotron-3-embed-1b |
2.2.2 | fp16 | quality on the reduced and the whole corpora; concurrency |
llama-nemotron-embed-1b-v2 |
1.13.0 | fp8, chosen by the image (fp16 pinned as a check) | quality, concurrency |
llama-nemotron-embed-300m-v2 |
1.13.0 | fp8, chosen by the image | quality, concurrency |
llama-nemotron-embed-vl-1b-v2 |
2.3.0 | chosen by the image | quality, concurrency |
llama-nemotron-rerank-1b-v2 |
1.10.0 | chosen by the image | reranking the top 100 of the whole corpora; concurrency |
llama-nemotron-rerank-500m-v2 |
1.10.0 | chosen by the image | the same |
| reference | open weights of nemotron-3-embed-1b |
bf16 | does our harness ask the model the right way? |
nemotron-3-embed-1b is listed with an NVFP4 profile for the RTX PRO 6000; on this RTX 5090 that profile did not start, so it ran in fp16. The NVIDIA RAG Blueprint (v2.6.2) uses llama-nemotron-embed-vl-1b-v2 image 1.12.0 by default; on this card under Docker Desktop that image reported ready and then answered its first request with HTTP 500, while the same model’s 2.3.0 image served normally, so the tables use 2.3.0 (an issue we have not reproduced elsewhere; this card is not on the verified list for these NIMs).
Results: the whole corpus, and the reranker on top
nDCG@10 (bar) and Recall@100 (dash) for one encoder, nemotron-3-embed-1b: the reduced corpus where one exists, every passage of the corpus, and the same results after each of two rerankers (top 100).
| Corpus | Passages | Queries | nDCG@10 | Recall@100 | after 1b reranker | after 500m reranker |
|---|---|---|---|---|---|---|
| MIRACL zh, reduced | 81,309 | 393 | 0.662 | 0.988 | — | — |
| MIRACL zh, whole | 4,934,368 | 393 | 0.649 | 0.964 | 0.683 | 0.669 |
| MIRACL ja, reduced | 185,319 | 860 | 0.744 | 0.992 | — | — |
| MIRACL ja, whole | 6,953,614 | 860 | 0.741 | 0.984 | 0.771 | 0.752 |
| BEIR HotpotQA, whole | 5,233,329 | 7,405 | 0.801 | 0.916 | 0.865 | not run |
Reranking reorders the first 100 results, so Recall@100 is the same before and after. One call with 100 passages took a median of 143 ms with the 1b reranker and 85 ms with the 500m one on MIRACL zh, one call at a time.
The four embedding NIMs on the reduced sets (nDCG@10):
| NIM | SciFact | MIRACL zh, reduced | MIRACL ja, reduced |
|---|---|---|---|
nemotron-3-embed-1b 2.2.2 |
0.752 | 0.662 | 0.744 |
llama-nemotron-embed-vl-1b-v2 2.3.0 |
0.767 | 0.648 | 0.719 |
llama-nemotron-embed-1b-v2 1.13.0 |
0.775 | 0.645 | 0.716 |
llama-nemotron-embed-300m-v2 1.13.0 |
0.713 | 0.593 | 0.676 |
Pinning llama-nemotron-embed-1b-v2 to fp16 instead of the fp8 the image chooses moved nDCG@10 by −0.004 to +0.0004 on the three sets.
How many requests at once
The full curves for all six NIMs are in the first reply below (two images per post on a new account). Requests per second at one request at a time, the highest rate, and the level where the rate stopped rising (95% of the highest):
| NIM | Request | One at a time | Highest | Stops rising at |
|---|---|---|---|---|
nemotron-3-embed-1b |
1 query | 269 req/s | 384 | 2 |
| 〃 | 64 passages | 252 passages/s | 272 | 2 |
llama-nemotron-embed-vl-1b-v2 |
1 query | 293 req/s | 419 | 2 |
llama-nemotron-embed-1b-v2 |
64 passages | 594 passages/s | 796 | 2 |
llama-nemotron-embed-300m-v2 |
1 query | 170 req/s | 417 | 8 |
| 〃 | 64 passages | 1,073 passages/s | 2,138 | 4 |
llama-nemotron-rerank-1b-v2 |
1 query, 100 passages | 678 pairs/s | 983 | 16 |
llama-nemotron-rerank-500m-v2 |
1 query, 100 passages | 1,113 pairs/s | 1,913 | 8 |
For the query-shaped requests the p99 stayed under 100 ms up to 32 concurrent requests for nemotron-3-embed-1b; past the level where the rate stops rising, latency grows in step with concurrency and the rate stays flat.
Run it
An embedding NIM and a reranking NIM, one at a time on a 32 GB card. These are the commands we ran, with our host paths replaced; /model holds weights that this NIM fetched beforehand in its download-only mode, and we did not test starting it without that mount:
docker run -d --name nemotron-3-embed-1b --gpus all -p 8000:8000 --shm-size=16GB \
-e NGC_API_KEY -e NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER=ngc -e NIM_ENGINE_PRECISION=fp16 \
-v /path/to/weights:/model -v /path/to/nim-cache:/opt/cache \
nvcr.io/nim/nvidia/nemotron-3-embed-1b:2.2.2
docker run -d --name llama-nemotron-rerank-1b-v2 --gpus all -p 8000:8000 --shm-size=16GB \
-e NGC_API_KEY -v /path/to/nim-cache:/opt/nim/.cache \
nvcr.io/nim/nvidia/llama-nemotron-rerank-1b-v2:1.10.0
One level of the concurrency sweep (one query per request, 16 concurrent). We ran AIPerf 0.11.0 in a container on the NIM’s Docker network; run directly on this Windows host it hung at the end of a run:
aiperf profile --model nvidia/nemotron-3-embed-1b --url http://<nim-host>:8000 \
--endpoint-type raw --custom-endpoint /v1/embeddings \
--custom-dataset-type inputs-json --input-file /path/to/emb_query.inputs.json \
--concurrency 16 --benchmark-duration 40 --warmup-request-count 32 \
--tokenizer builtin --use-server-token-count --ui none \
--artifact-dir /path/to/artifacts --no-gpu-telemetry --no-server-metrics
Five things that decide what you get
- Score on the corpus you will serve. The reduced benchmark corpus keeps each query’s judged passages and the hard negatives retrieved for it; the whole corpus adds millions of passages that can still crowd the top 100. On MIRACL zh that cost 0.013 nDCG@10 and 0.024 Recall@100.
- Rerank the top 100. On these three whole corpora the 1b reranker added 0.030 to 0.064 nDCG@10 for a median of 103–154 ms per query at one request at a time; the 500m one gave 37–60% of that gain in about 60% of the time.
- Budget the reranker first, then add instances, not concurrency. At 100 passages per query the reranker sets the ceiling of the retrieval stage (about 10 or 19 queries per second here), far below the embedder. Past each NIM’s saturation level, sending more requests only makes each one wait.
- Store fp16. Check first that the round trip is exact for your encoder; for this one it was, for every value of 17 million vectors. If you go to int8, check the ranking, not only nDCG.
- Send one real request before you trust “ready”. On this card, the Blueprint’s default embedding image reported ready and failed its first request; the newer image of the same model did not.
A few honest notes
- One RTX 5090 on Docker Desktop / WSL2, sessions from 4 to 13 October; the driver moved from 591.86 to 617.42 over that time, and every cell records the driver it ran on. The card is not on NIM’s verified-GPU list for these models.
- Two kinds of rate. Every rate above is server-side, from AIPerf. Our own single-connection Python client, which also parses each 2,048-dimension response, measured up to 26% less on the embedding NIMs; those client numbers are in the repo and labelled end to end.
- The load tool has a ceiling too. Against a server that answers instantly with responses of the same size, AIPerf on this host went at least 2.99 times faster than the fastest NIM at every level, so the plateaus above are the NIMs’. That ceiling itself varied by up to 58% between two runs.
- The zh reference is not reproduced. Our harness and MTEB itself, run here on the open weights, agree with each other (0.6631 and 0.6625) and not with the published 0.6424. Nothing was tuned to make it fit. HotpotQA and the whole MIRACL corpora have no published value for this model; those numbers are reported as measured.
- Exact search, on purpose. An approximate index would add its own recall loss; that belongs to the next volume, which runs the whole Blueprint.
Everything is in the repo
Pre-registrations (frozen before the first container of each session), per-query scores, the run logs, both concurrency nights side by side, the tool-ceiling cell, the figures with their point tables, and a claim-to-evidence table for every number in this post are in the Vol.4 README of NV-benchmark. This is the fourth post in the series; Vol.3 covers the models behind a NeMo Guardrails rail, Vol.2 Nemotron 3 Nano, Vol.1 Llama 3.1 8B with NIM.
None of this says what retrieval does for a given application; that depends on the corpus you serve, which is why the whole-corpus numbers are here. If you run these NIMs on an RTX PRO 6000 or a DGX Spark, we’d like to see where your curves flatten.
