NIM Llama3 8B Instruct - Running container with "CUDA_ERROR_NO_DEVICE"

thanhdc · March 14, 2025, 7:28am

Hi all,

I had tried the on-premise version of llama3-8b-instruct deployment follow the tutorial here Getting Started — NVIDIA NIM for Large Language Models (LLMs).
I had installed and checked all the hardware and software requirements. I have tried running it many times, but still got the same error as shown in picture below.

I am wondering is there any idea about this issue?
Thank you so much.

AakankshaS · March 28, 2025, 8:43pm

Hi @thanhdc
Please check this

Topic		Replies	Views
Aunch NVIDIA NIM (llama3-8b-instruct) for LLMs locally Access/Accounts nim , llama3-8b-instruct	3	107	November 8, 2024
Unable to Run NIM on H100 GPU Due to Profile Compatibility Issue Despite Sufficient GPU Resources Models nim , llama-31-8b-instruct , llama	1	191	November 12, 2024
Reusing a stored model (llama-3.1-8b-instruct) with a proper profile Models nim , llama-31-8b-instruct , llama	0	149	October 30, 2024
Problem with installation of Llama 3.1 8b NIM Models nim , llama3-8b-instruct , llama-31-8b-instruct , llama	1	544	September 4, 2024
NVIDIA NIM Container with CUDA out of Memory Problem Docker and NVIDIA Docker cuda , ubuntu , docker , nim , llama3-8b-instruct	2	492	September 20, 2024
Issues while starting NIM container in A10 VM Models nim , llama3-8b-instruct	4	148	September 4, 2024
How to fix 0 compatible profiles? Where to get compatible profiles? Models nim , llama-31-8b-instruct , llama	4	464	November 26, 2024
NIM does not support llama-3.1-8b-instruct and llama-3.1-70b-instruct on GH200 On-Prem deployment Models nim , llama-31-8b-instruct , llama	1	193	November 7, 2024
Nemollm-inference-microservice failed to deploy Models nim , llama3-8b-instruct , llama	1	148	October 22, 2024
Can I use models like Llama-3-8B-Instruct-Coder with NIM? Models nim , llama	1	59	September 20, 2024

NIM Llama3 8B Instruct - Running container with "CUDA_ERROR_NO_DEVICE"

Related topics