"404 Page Not Found" Error When api used as openai

AI & Data Science NVIDIA NIM Models
1

How to Use API Key in a Way Compatible with OpenAI?

2

Hi @siyuan-l – take a look at one of the examples on build.nvidia.com, like the one here in the Python tab: llama-3.1-405b-instruct Model by Meta | NVIDIA NIM

Your code should look something like the following:

from openai import OpenAI

client = OpenAI(
  base_url = "https://integrate.api.nvidia.com/v1",
  api_key = "$API_KEY_REQUIRED_IF_EXECUTING_OUTSIDE_NGC"
)

completion = client.chat.completions.create(
  model="meta/llama-3.1-405b-instruct",
  messages=[{"role":"user","content":"Write a limerick about the wonders of GPU computing."}],
  temperature=0.2,
  top_p=0.7,
  max_tokens=1024,
  stream=True
)

for chunk in completion:
  if chunk.choices[0].delta.content is not None:
    print(chunk.choices[0].delta.content, end="")

If you’re seeing a 404 error, it might be because the base_url or model name aren’t correctly specified

4

Thanks for your help!
This method works with LLaMA-3.1-405B-Instruct, but it doesn’t work with LLaMA-3.2-90B-Instruct. Is it because the latter hasn’t been adapted yet?