[SUPPORT] Workbench Example Project: Hybrid RAG

AI & Data Science NVIDIA AI Workbench
103

It looks like a mismatch on Pytorch versions (so libnccl2 is not running)
From inside the project-hybrid-rag container I ran (the latest libnccl2)
apt-get install -y libnccl2=2.26.5-1+cuda12.9 libnccl-dev=2.26.5-1+cuda12.9

then I could see
dpkg -l | grep nccl
ii libnccl-dev 2.26.5-1+cuda12.9 amd64 NVIDIA Collective Communication Library (NCCL) Development Files
ii libnccl2 2.26.5-1+cuda12.9 amd64 NVIDIA Collective Communication Library (NCCL) Runtime

restarted the container /
docker restart project-hybrid-rag-container
docker exec -it project-hybrid-rag-container /bin/bash

Then from inside the container /
python -c “import torch”
Showing there is no longer a lib mismatch
exit container
Then commit the container with a new name /
docker commit project-hybrid-rag-container project-hybrid-rag:fixed-nccl

From inside the container you should see
python -c “import torch; print(torch.version); print(torch.version.cuda); print(torch.cuda.is_available())”

2.4.0
12.4
True
However I then realised I couldn’t create a new project from the fixed container:-)

104

yeah… new containers is a thing.

i’m going to spin up a little project to automate the introspection and label creation for containers so you can just drop them in.

when i get it to an acceptable point, i will share it with you.

105

Hi @btapiasierra @jorge.mijares @lapporatory @andrew319 and all,

Thanks for bringing up this issue. Working on a fix for that now on on branch edwli-dev. Stay tuned.

106

Hi all,

Just pushed a relatively major update to the project, feel free to try it out and let me know of any feedback.

Release Notes:

6/3/2025 Project Overhaul (#24)

  • Bug fix for recurring torch version issue (here)
  • Add a new compose.yaml file:
    • Refactor microservice inference option to utilize NVIDIA NIM via docker/podman compose
    • Swap default NIM option from meta/llama3-8b-instruct to meta/llama-3.1-8b-instruct
  • Remove outdated/deprecated NVIDIA-hosted model endpoints from cloud inference option for better alignment
  • Replace NVCF_RUN_KEY terminology with NVIDIA_API_KEY for better alignment and accuracy
  • Better error messaging in the chatbot UI; replace concise message with full stack trace
  • minor UI layout updates and documentation updates
107

Update fixes that issue / Nice work and thank you @edwli !

108

Hi Edwli,
I was able to run using brev!! thanks for the fix. In my computer didn’t work because I only have 6gb on my GPU, the error was not clear just the API call failed due timeout. On Brev instance all worked great
I tried to use my Ollama server on my local but I couldn’t make it work. I will collect logs and send them here but I was wondering if you have some guide to use ollama?

109

can the workbench be installed for a hybrid RAG on XCP VM (W11) without gpu or with AMD-MxGPU

110

You should be able to clone the hybrid RAG project on any system with workbench installed/running on it.

No NVIDIA GPU is required to run the workflow. For example, you can use the NVIDIA-hosted cloud endpoints for inference, even if you have no GPU or an AMD GPU.

I believe depending on the GPU SKU, AMD GPUs may even be supported for the local TGI inference option as well, but this is untested.

111

(8/12)

Torch / Transformers version bump

113

Hi ,

I am trying to set up the Hybrid RAG example project in NVIDIA AI Workbench on my Asus Ascent GX10 (running Ubuntu on aarch64 architecture).

When attempting to build the project, I encounter the following error indicating a platform mismatch:

Containerfile:1

1 | >>> FROM Package text-generation-inference · GitHub

2 |
3 | WORKDIR /opt/project/build/

ERROR: failed to build: failed to solve: ghcr.io/huggingface/text-generation-inference:2.3.0: failed to resolve source metadata for ghcr.io/huggingface/text-generation-inference:2.3.0: no match for platform in manifest: not found
Build Failed

Any guidance on how to get this running on ARM hardware would be appreciated.

Thanks!

114

Hi @rz.marek ,

Thanks for your interest in this project. Unfortunately, the base container for this project is currently incompatible with an ARM architecture system. This is an issue on the Hugging Face TGI side, and it appears ARM is not on their roadmap for the foreseeable future (see Github issues here, here, and here).

As an alternative, you can try the Agentic RAG example project. There is a playbook for this RAG project available here.