Hi all,
Vol.4 measured the retrieval NIMs one at a time. Vol.5 runs the whole NVIDIA RAG Blueprint (v2.6.2) query path as it ships, with its defaults, on one RTX 5090: Elasticsearch as the default vector store, the Llama Nemotron reranking NIM, and Nemotron 3 Nano as the answer model, against every passage of BEIR HotpotQA (5,233,329 passages, 7,405 questions with official answers) and of MIRACL zh (4,934,368 passages). Three things changed from the shipped defaults, and each one is checked below: the embedding model is nemotron-3-embed-1b so the Vol.4 vectors could be reused, the answer model is Nemotron 3 Nano in NVFP4 because the default 120B model needs more than one card, and the vectors were loaded into the index directly instead of through the Blueprint’s ingestor.
The Blueprint is built for multi-GPU servers; most of what follows is what its defaults do when you bring it down to one card.
TL;DR: nine defaults to check before you deploy the Blueprint on one card
- Retrieval moves the answer from 30% to 53%; exact search in place of the default index takes it to 60%; the right passages would take it to 73%. On 7,405 HotpotQA questions, the answer contained the official answer 30.4% of the time with the knowledge base switched off, 53.4% through the Blueprint’s default path, 60.4% when the same reranker and model got exact search’s top 100 instead, and 72.8% when the two supporting passages were handed to the same model with the same prompt.
- The default vector store costs 0.125 nDCG@10 on 5.2M passages, and the only knob gets 0.05 back. Elasticsearch’s default index (
bbq_hnsw, 100 candidates) scored 0.676 where exact search over the same vectors scores 0.801.VECTOR_DB_TOPK, the one setting the Blueprint exposes, tops out at 400: 0.726, with the median search going from 62 to 107 ms and four times the passages sent to the reranker. Hybrid search does not open another way on the shipped Elasticsearch: the default ranker is refused under its basic licence, and the weighted ranker capsVECTOR_DB_TOPKat 100. On MIRACL zh the loss was small: 0.627 against 0.649. - Text ingestion drops the last token of every chunk and can guess the wrong encoding. Through the default ingestor, 99,721 of 100,000 passages lost their final token (usually the full stop), and 249 passages with a few non-ASCII characters were stored garbled. Both are in the nv-ingest 26.3.0 the Blueprint pins; the first is a regression of a fix merged in 2025. Minimal reproductions are in the repo.
- With the default context length, every default request is refused. The Blueprint sends
max_tokens16256 with every request. An answer NIM started with a 16,384-token context refused all 200 test questions (HTTP 400, “maximum context length”); at 32,768 all 200 answered. - The reasoning budget of 256 tokens has no effect, and by default the reasoning comes back inside the answer. Median reasoning length was 326 tokens with the knowledge base off. The NIM does not separate reasoning from the answer unless started with
--reasoning-parser nemotron_v3; with it, 200 of 200 answers came back clean, and the share containing the answer did not differ significantly (100 against 105 of 200, p = 0.30). - When the supporting passages are not in the corpus, the model refuses at least a third of the time and still answers the rest. With both supporting passages filtered out of the search for 200 questions, 33.5% of answers were refusals by pattern (54% in a 50-question manual check), 12.0% still contained the correct answer, and the remainder answered from whatever came back.
- Three NIMs fit on the card after start-up, at one memory ratio. Embedding, reranking and Nano shared the card at
gpu_memory_utilization0.65 without paging once ready (29,382–31,000 MiB used); at the NIM’s default 0.92 with the answer model started first, 4,239 MiB was paged to system memory in steady state. Context length and sequence count did not change the footprint. - The whole path, co-resident, answers a HotpotQA question in 1.2 s median at one request, 2.2 s at four. Retrieval 62 ms, reranking 150 ms, first token 306 ms, whole answer 1,203 ms (p99 12.9 s) at c=1; 42 / 250 / 408 / 2,249 ms (p99 45.7 s) at c=4.
- Start-up of the answer NIM pages up to 2,980 MiB on this Windows host. It settles once the model is ready; we did not measure a Linux host.
Scores: “contains” = normalized answer contains the official HotpotQA answer; nDCG@10 and Recall@100 from pytrec_eval; Wilson 95% intervals; paired comparisons by exact McNemar. Latencies are cold-start medians (Elasticsearch restarted and page cache dropped before each setting).
One RTX 5090 · NVIDIA RAG Blueprint v2.6.2 defaults · Elasticsearch 9.3.0 · nemotron-3-embed-1b 2.2.2 · llama-nemotron-rerank-1b-v2 1.10.0 · Nemotron 3 Nano NVFP4, NIM 2.0.12 · BEIR HotpotQA and MIRACL zh, every passage
What ran
| Stage | What the Blueprint ships | What ran here | Checked by |
|---|---|---|---|
| Ingestion | nv-ingest 26.3.0 via ingestor-server 2.6.0 | 100,000 passages through the ingestor; the rest loaded directly with the ingestor’s own index mapping | exact search on the directly loaded 100k index agrees with numpy 200/200; the ingestor’s own 100k index overlaps it 0.982 in the exact top 100 (the difference is the two ingestion defects below) |
| Vector store | Elasticsearch 9.3.0, bbq_hnsw, k = candidates = VECTOR_DB_TOPK = 100 |
as shipped | brute force over all 5.2M passages on 5 queries = exact search |
| Embedding | llama-nemotron-embed-vl-1b-v2 1.12.0 |
nemotron-3-embed-1b 2.2.2 fp16 (Vol.4 vectors) |
the Blueprint’s own query embedding of 20 questions matches Vol.4’s to cosine ≥ 0.9999999995 |
| Reranking | llama-nemotron-rerank-1b-v2 1.10.0, top 100 → 10 |
as shipped | requests captured from rag-server match ours field by field, 5/5 |
| Answer | 120B model, two GPUs | Nemotron 3 Nano NVFP4, NIM 2.0.12 | same prompt, same request parameters as rag-server sends (captured, 25/25) |
| Guardrails | off by default | off | — |
Every other setting is the Blueprint’s default, listed with file and line in the repo’s BLUEPRINT_SETTINGS.md (206 rows).
Results: the answer, with and without retrieval
“Contains the official answer” on 7,405 HotpotQA questions, three of the four arms in the table below, Wilson 95% intervals; and the refusal rate when the supporting passages are removed from the search (200 questions).
| Arm | What the model saw | Contains answer | 95% CI | F1 | Refusals |
|---|---|---|---|---|---|
| Knowledge base off | nothing (the Blueprint’s own no-KB template) | 0.304 | 0.294–0.315 | 0.285 | 85 |
| Blueprint default path | top 100 from Elasticsearch → reranked → 10 passages | 0.534 | 0.522–0.545 | 0.460 | 829 (11.2%) |
| Exact search instead of the index | top 100 from exact search over the same vectors → the same reranker → 10 passages | 0.604 | 0.593–0.615 | 0.522 | 298 |
| Supporting passages given | the two official supporting passages, same prompt | 0.728 | 0.717–0.738 | 0.618 | 49 |
Paired: the default path answered 2,090 questions that the no-KB arm missed and lost 389 the other way; exact search answered 800 the default path missed and lost 280; the supporting-passage arm answered 1,157 that exact search missed and lost 241. The ten passages held both supporting passages for 4,271 questions on the default path (57.7%) and 5,707 with exact search (77.1%).
So of the 19 points between the default path and the supporting passages, about 7 are the vector index and the rest come after it: questions whose ten passages hold only one supporting passage even with exact search are the hard ones (24% contain the answer, against 35% for the one-passage questions of the default path).
The default path’s candidates are the 100 passages of the first HotpotQA query (below), and the reranker was called with four requests in flight; at that concurrency the reranking NIM batches requests across questions, so the order of the ten can differ on a repeat run. The ten passages each answer saw are kept in per-question tables outside the repository (supplement 8); the repo holds the counts made from them.
On MIRACL zh (393 questions, no answer strings) the default path put at least one judged-relevant passage in the model’s ten for 90.1% of questions (86.7–92.7%); 26 answers hit the output limit.
Results: the vector store at full scale
| Corpus | Passages | Exact search | ES default | Recall@100 (ES) | Top-100 overlap with exact | Cold p50 |
|---|---|---|---|---|---|---|
| HotpotQA | 5,233,329 | 0.801 | 0.676 | 0.766 | 0.784 | 62.3 ms |
| MIRACL zh | 4,934,368 | 0.649 | 0.627 | 0.912 | 0.900 | 161.9 ms |
The HotpotQA index was first queried right after the bulk load finished, while Elasticsearch was still merging segments. Queried again on the settled index (32 segments, unchanged from 20 October to 1 November), 26.5% of the questions got a different top 100 and the scores moved by less than 0.001: nDCG@10 0.676 both times, Recall@100 0.766 then 0.765 (a difference of 0.0003). The rows below and every HotpotQA cold-start latency were measured on the settled index.
The MIRACL zh index was queried in the 17 minutes after its load finished; a second query 21 to 33 minutes after the load returned a different top 100 for 60% of the questions and nDCG@10 0.628 (Recall@100 0.916). It was not queried again after that, so the zh scores are for an index whose segment merges may not have finished. The zh cold-start latency was taken about 23 hours after the load.
The one knob (HotpotQA, same index):
| Setting | nDCG@10 | Cold p50 | Passages to the reranker |
|---|---|---|---|
Default (VECTOR_DB_TOPK 100) |
0.676 | 62.3 ms | 100 |
VECTOR_DB_TOPK 200 |
0.704 | 81.7 ms | 200 |
VECTOR_DB_TOPK 400 (the maximum the Blueprint accepts) |
0.726 | 106.6 ms | 400 |
Elasticsearch num_candidates 1000, k kept at 100 |
0.718 | 64.4 ms | 100 |
The last row is what a reader would want (most of the gain, no latency cost, nothing extra to rerank), and it cannot be set from the Blueprint: fetch_k is tied to top_k. Raising num_candidates to 200 changed nothing (7,405 identical result lists); the index’s 3× rescoring already covers that many.
Hybrid search is the Blueprint’s other retrieval setting, and it does not get around this on the Elasticsearch it ships. With the default ranker (APP_VECTORSTORE_RANKER_TYPE left at rrf) Elasticsearch refuses the query: basic licence, HTTP 403, “current license is non-compliant for [Reciprocal Rank Fusion (RRF)]”. With weighted, num_candidates is fixed at 100 and k follows VECTOR_DB_TOPK, so any VECTOR_DB_TOPK above 100 is refused (HTTP 400, “[num_candidates] cannot be less than [k]”).
Results: on the card
With gpu_memory_utilization 0.65 on the answer NIM, started last:
| Embedding | Reranking | Answer NIM | All three | |
|---|---|---|---|---|
| Alone on the card | 3,966 MiB | 3,351 MiB | 19,940 MiB | — |
| Co-resident | 29,382 MiB about a minute after ready; paged to host 575 MiB (below the sum of each service’s own pinned pool) |
The positive control for the paging judgement, the answer NIM first at its default 0.92, paged 4,239 MiB in steady state and left the embedding NIM with less than half its memory. Four combinations of context length (8,192 / 16,384) and sequence count (8–26) at 0.65 had the same footprint; only the ratio moves it. We did not search below 0.65.
Stage timings through rag-server, co-resident, 200 questions:
| Retrieval | Reranking | First token | Whole answer | p99 | |
|---|---|---|---|---|---|
| c = 1 | 62.0 ms | 150.1 ms | 306.4 ms | 1,203 ms | 12.9 s |
| c = 4 | 41.7 ms | 250.5 ms | 407.7 ms | 2,249 ms | 45.7 s |
The long tail is reasoning: the 256-token budget the Blueprint sends is not enforced by the NIM, and a few answers reason for thousands of tokens.
Run it
Five commands, in this order, from a folder that holds a clone of NVIDIA-AI-Blueprints/rag at v2.6.2 as rag/ and the NV-benchmark repository as NV-benchmark/. $NGC_ENV_FILE is a file holding your NGC key (NGC_API_KEY=...).
# 1 Elasticsearch, as the Blueprint ships it (the second file only renames the container and the volume); creates the network nvidia-rag
docker compose -p p137es -f rag/deploy/compose/vectordb.yaml -f NV-benchmark/vol5-rag-blueprint/compose_es/override_vectordb.yaml up -d
# 2 the embedding NIM (deviation 1: nemotron-3-embed-1b, fp16)
docker run -d --name embed --gpus device=0 --network nvidia-rag --shm-size=16GB --env-file "$NGC_ENV_FILE" \
-e NIM_ENGINE_MODEL_DOWNLOAD_PROVIDER=ngc -e NIM_ENGINE_PRECISION=fp16 \
-v ~/nim/weights/nemotron-3-embed-1b-nim-2.2.2:/model -v ~/nim/cache/nemotron-3-embed-1b-2.2.2:/opt/cache \
nvcr.io/nim/nvidia/nemotron-3-embed-1b:2.2.2
# 3 the reranking NIM, as shipped
docker run -d --name rerank --gpus device=0 --network nvidia-rag --shm-size=16GB --env-file "$NGC_ENV_FILE" \
-v ~/nim/cache/llama-nemotron-rerank-1b-v2-1.10.0:/opt/nim/.cache \
nvcr.io/nim/nvidia/llama-nemotron-rerank-1b-v2:1.10.0
# 4 the answer NIM (deviation 2), last, with the settings that fit next to the other two and accept the Blueprint's requests
# (NIM_KVCACHE_PERCENT is the share of the card the NIM gives vLLM as gpu_memory_utilization)
docker run -d --name nano --gpus device=0 --network nvidia-rag --shm-size=16GB --env-file "$NGC_ENV_FILE" \
--user 0 --ulimit nofile=65536:65536 \
-e NIM_ENABLE_CHUNKED_PREFILL=1 -e NCCL_NVLS_ENABLE=0 -e VLLM_USE_FLASHINFER_MOE_FP8=0 \
-e NIM_MODEL_PROFILE=1fba9ecfcfb4cde28d4ce3fd55c40bca89a5a613e25e98f057befe6a7e99eada \
-e NIM_MAX_MODEL_LEN=32768 -e NIM_KVCACHE_PERCENT=0.65 \
-e "NIM_PASSTHROUGH_ARGS=--max-num-seqs 26 --reasoning-parser nemotron_v3" \
-v ~/.cache/nim:/opt/nim/.cache \
nvcr.io/nim/nvidia/nemotron-3-nano:2.0.12
# 5 the Blueprint's rag-server, the two model names changed; PROMPT_CONFIG_FILE holds {} (no prompt overridden)
export PROMPT_CONFIG_FILE=$PWD/NV-benchmark/vol5-rag-blueprint/compose_es/prompt_overrides_none.yaml
docker compose -p p137rag --env-file NV-benchmark/vol5-rag-blueprint/compose_es/blueprint_query_path.env --env-file "$NGC_ENV_FILE" \
-f rag/deploy/compose/docker-compose-rag-server.yaml \
-f NV-benchmark/vol5-rag-blueprint/compose_es/override_rag_server.yaml \
up -d rag-server
The index itself is not one of the five: the vectors of Vol.4 were loaded directly (deviation 3, scripts/es_bulk_load.py in the repo). On Docker Desktop with Git Bash, set MSYS_NO_PATHCONV=1 first.
Nine things to check
- Score on your corpus at its real size, and only after the index has settled. The gap between the reduced benchmark and 5.2M passages, and between exact search and the default index, both show up only at scale; and the first queries after a bulk load see a different graph, so wait until
_cat/segmentsstops changing. - Set
VECTOR_DB_TOPKto 400 if the reranker has headroom, or raisenum_candidatesin Elasticsearch directly and keepkat 100. Hybrid search will not do it for you on the shipped Elasticsearch (see above). - Check what the ingestor stored. Read a few chunks back; look for a missing final character and for garbled non-ASCII text. In nv-ingest 26.3.0 the chunk slice ends at the start of the last token (
split_text.py, line 103), and the text reader decodes with the encodingcharset_normalizerguesses without trying UTF-8 first (extract.py, lines 111–118). Minimal reproductions are in the repo. - Start the answer NIM with a context longer than
LLM_MAX_TOKENSplus your prompt. With the Blueprint’s 16256, 16,384 is not enough; 32,768 was. - Start the answer NIM with
--reasoning-parser nemotron_v3, or the model’s reasoning is returned as part of the answer. - Do not rely on the 256-token reasoning budget. Cap
max_tokensyourself if the tail matters. - Expect answers when the corpus does not have the answer. The default prompt refuses about a third of the time; the rest is answered from the nearest passages.
- Set the answer NIM’s memory ratio before co-locating. At the default 0.92 the three NIMs page; at 0.65 they fit.
- Send one real Blueprint request after “ready”. Every refusal and every 400 above came from a request the Blueprint actually sends.
A few honest notes
- Three deviations from the shipped defaults (embedding model, answer model, direct vector load), each with a control above. The numbers are for this combination, not for the Blueprint’s default embedding model or its 120B answer model.
- One RTX 5090 on Docker Desktop / WSL2, driver 617.42, sessions 18–22 October; the exact-search arm on 26 October and 1 November with the same images and settings as each other; it answered at 10 requests in flight, the default path at 7 (each the concurrency the NIM reported for that container when it started). Memory paging is a Windows behaviour; a Linux host would either start or fail.
- The 0.65 ratio was the smallest this card started with, not a documented minimum. We did not search for a smaller one.
- The refusal pattern is a lower bound. In 50 manually labelled answers the pattern missed 4 refusals and flagged no false ones.
- MIRACL zh has no answer strings, so only the retrieval side and the “relevant passage reached the model” rate are reported there. MIRACL ja was not run.
- Ingesting 5.2M passages through the Blueprint’s ingestor would take an estimated 97 hours on this host (1 h 51 min per 100,000 passages, measured) and hold the card throughout; the direct load is what made the full corpus possible.
Everything is in the repo
Pre-registrations (frozen before each session’s first container, with eight addenda), the Blueprint settings table, captured requests, per-arm scores, the ingestion reproductions, the cold-start latency runs including the two voided ones and why, the paging judgement with its positive control, and a claim-to-evidence table for every number here are in the Vol.5 README of NV-benchmark (tag vol5). The scoring module is importable, and the HotpotQA and MIRACL dev sets are evaluation sets here: any system that wants to compare against these numbers should not tune on them. Vol.4 covers the retrieval NIMs alone, Vol.3 the models behind a NeMo Guardrails rail, Vol.2 Nemotron 3 Nano, Vol.1 Llama 3.1 8B with NIM.
If you run the Blueprint on an RTX PRO 6000 or a DGX Spark, we’d like to know which of the nine you hit.
