NIM Llama 3.3 70B requirements

renambot · February 17, 2025, 9:53pm

I’m trying to run Llama 3.3 70B, and it cannot find a suitable profile using 2 or 3 H100 (80BG). I thought it would run using 2 H100:
container nvcr.io/nim/meta/llama-3.3-70b-instruct:1.5.2

I’ve been running other NIM llama 3.1 and 3.2 successfully.

Can I pass some options to make it run? fp8 or decreasing the context window?

Thanks,

Luc

sophwats · February 24, 2025, 11:30am

Hi @renambot I’m afraid there are no fp8 profiles for this NIM so it wont run on 2 H100s.

Sophie

renambot · March 21, 2025, 8:36pm

It looks like in v1.8.2, there are more profiles enabled now. Nice.

Topic		Replies	Views
NIM llama3 deploy fp16 Models nim , llama3-70b-instruct	2	128	July 26, 2024
Unable to Run NIM on H100 GPU Due to Profile Compatibility Issue Despite Sufficient GPU Resources Models nim , llama-31-8b-instruct , llama	1	144	November 12, 2024
NIM does not support llama-3.1-8b-instruct and llama-3.1-70b-instruct on GH200 On-Prem deployment Models nim , llama-31-8b-instruct , llama	1	162	November 7, 2024
Profiles doesnt match machine even though specs are correct Models nim , llama-31-405b-instruct , llama	0	25	November 29, 2024
0 Compatible Profiles for Llama 3.1 70B Models nim , llama-31-70b-instruct	6	389	October 28, 2024
How to fix 0 compatible profiles? Where to get compatible profiles? Models nim , llama-31-8b-instruct , llama	4	397	November 26, 2024
NIM TensorRT-LLM on H100 NVL Models nim , llama-31-8b-instruct , llama	2	119	November 22, 2024
NIM - Llama3-8b-Instruct - GPU resource usage is very high Models nim , llama3-8b-instruct	0	16	March 12, 2025
NIM support for GH200? Models nim , llama	2	92	August 21, 2024
Multi-LoRA with LLAMA 3 NIM is not listed in API Models nim , llama3-8b-instruct , llama	2	85	August 21, 2024

NIM Llama 3.3 70B requirements

Related topics