Hello, I am new and missing something. I was able to log into docker via “$ docker login nvcr.io” and was even able top reach the model to begin downloading using the following command and API key, but when attempting to run, the system is saying that it does not have and API key. Any help would be much appreciated!
export NGC_API_KEY=
export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p “$LOCAL_NIM_CACHE”
sudo docker run -it --rm
–gpus all
–shm-size=16GB
-e NGC_API_KEY
-v “$LOCAL_NIM_CACHE:/opt/nim/.cache”
-u $(id -u)
-p 8000:8000
nvcr.io/nim/meta/llama-3.1-8b-instruct:1.1.2
bash: syntax error near unexpected token `newline’
Unable to find image ‘nvcr.io/nim/meta/llama-3.1-8b-instruct:1.1.2’ locally
1.1.2: Pulling from nim/meta/llama-3.1-8b-instruct
5e8117c0bd28: Pulling fs layer
d67fcc6ef577: Pulling fs layer
47ee674c5713: Pulling fs layer
63daa0e64b30: Waiting
d9d9aecefab5: Pulling fs layer
bea35665f412: Pull complete
20c43d9d95aa: Pull complete
b5a0dbeb89d5: Pull complete
0df90329ac80: Pull complete
93312c09429a: Pull complete
0ee063c960a4: Pull complete
ecf7a27c46a1: Pull complete
ed95bdb65fab: Pull complete
74054c067cde: Pull complete
a70adefcda66: Pull complete
d3de2f1518f1: Pull complete
f0dc4708904a: Pull complete
3692bbc2a06f: Pull complete
28a73569c467: Pull complete
00184057dd94: Pull complete
7be533624ee4: Pull complete
91949cd0cd76: Pull complete
0ee5ebedd78d: Pull complete
67dfcde7fb15: Pull complete
da91e163544a: Pull complete
cf8a5f8c91c7: Pull complete
dd5066bd0b4a: Pull complete
30a1e8a7ae03: Pull complete
50b42eddb500: Pull complete
Digest: sha256:c53ac18b1161899e3b7006e4cb24737c218f90c320ca2552a5d75e75d9fc4931
Status: Downloaded newer image for nvcr.io/nim/meta/llama-3.1-8b-instruct:1.1.2
===========================================
== NVIDIA Inference Microservice LLM NIM ==
NVIDIA Inference Microservice LLM NIM Version 1.1.2
Model: nim/meta/llama-3.1-8b-instruct
Container image Copyright (c) 2016-2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
The use of this model is governed by the NVIDIA AI Foundation Models Community License Agreement (found at NVIDIA Agreements | Enterprise Software | NVIDIA AI Foundation Models Community License Agreement.
ADDITIONAL INFORMATION: Llama 3.1 Community License Agreement, Built with Llama.
INFO 09-20 21:37:21.423 ngc_profile.py:222] Running NIM without LoRA. Only looking for compatible profiles that do not support LoRA.
INFO 09-20 21:37:21.423 ngc_profile.py:224] Detected 2 compatible profile(s).
INFO 09-20 21:37:21.423 ngc_injector.py:132] Valid profile: 6a3ba475d3215ca28f1a8c8886ab4a56b5626d1c98adbfe751025e8ff3d9886d (vllm-fp16-tp2) on GPUs [0, 1]
INFO 09-20 21:37:21.423 ngc_injector.py:132] Valid profile: 3bb4e8fe78e5037b05dd618cebb1053347325ad6a1e709e0eb18bb8558362ac5 (vllm-fp16-tp1) on GPUs [0, 1]
INFO 09-20 21:37:21.423 ngc_injector.py:190] Selected profile: 6a3ba475d3215ca28f1a8c8886ab4a56b5626d1c98adbfe751025e8ff3d9886d (vllm-fp16-tp2)
INFO 09-20 21:37:21.425 ngc_injector.py:198] Profile metadata: feat_lora: false
INFO 09-20 21:37:21.425 ngc_injector.py:198] Profile metadata: llm_engine: vllm
INFO 09-20 21:37:21.425 ngc_injector.py:198] Profile metadata: precision: fp16
INFO 09-20 21:37:21.425 ngc_injector.py:198] Profile metadata: tp: 2
INFO 09-20 21:37:21.425 ngc_injector.py:218] Preparing model workspace. This step might download additional files to run the model.
[09-20 21:37:21.425 ERROR nim_sdk::hub::repo rust/nim-sdk/src/hub/repo.rs:117] One or more errors fetching files:
[09-20 21:37:21.425 ERROR nim_sdk::hub::repo rust/nim-sdk/src/hub/repo.rs:119] The requested operation requires an API key, but none was found
[09-20 21:37:21.425 ERROR nim_sdk::hub::repo rust/nim-sdk/src/hub/repo.rs:119] The requested operation requires an API key, but none was found
[09-20 21:37:21.425 ERROR nim_sdk::hub::repo rust/nim-sdk/src/hub/repo.rs:119] The requested operation requires an API key, but none was found
[09-20 21:37:21.425 ERROR nim_sdk::hub::repo rust/nim-sdk/src/hub/repo.rs:119] The requested operation requires an API key, but none was found
[09-20 21:37:21.425 ERROR nim_sdk::hub::repo rust/nim-sdk/src/hub/repo.rs:119] The requested operation requires an API key, but none was found
[09-20 21:37:21.425 ERROR nim_sdk::hub::repo rust/nim-sdk/src/hub/repo.rs:119] The requested operation requires an API key, but none was found
[09-20 21:37:21.425 ERROR nim_sdk::hub::repo rust/nim-sdk/src/hub/repo.rs:119] The requested operation requires an API key, but none was found
[09-20 21:37:21.426 ERROR nim_sdk::hub::repo rust/nim-sdk/src/hub/repo.rs:119] The requested operation requires an API key, but none was found
[09-20 21:37:21.426 ERROR nim_sdk::hub::repo rust/nim-sdk/src/hub/repo.rs:119] The requested operation requires an API key, but none was found
[09-20 21:37:21.426 ERROR nim_sdk::hub::repo rust/nim-sdk/src/hub/repo.rs:119] The requested operation requires an API key, but none was found
[09-20 21:37:21.426 ERROR nim_sdk::hub::repo rust/nim-sdk/src/hub/repo.rs:119] The requested operation requires an API key, but none was found
[09-20 21:37:21.426 ERROR nim_sdk::hub::repo rust/nim-sdk/src/hub/repo.rs:119] The requested operation requires an API key, but none was found
[09-20 21:37:21.426 ERROR nim_sdk::hub::repo rust/nim-sdk/src/hub/repo.rs:119] The requested operation requires an API key, but none was found
[09-20 21:37:21.426 ERROR nim_sdk::hub::repo rust/nim-sdk/src/hub/repo.rs:119] The requested operation requires an API key, but none was found
Traceback (most recent call last):
File “/usr/lib/python3.10/runpy.py”, line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File “/usr/lib/python3.10/runpy.py”, line 86, in _run_code
exec(code, run_globals)
File “/opt/nim/llm/vllm_nvext/entrypoints/launch.py”, line 99, in
main()
File “/opt/nim/llm/vllm_nvext/entrypoints/launch.py”, line 42, in main
inference_env = prepare_environment()
File “/opt/nim/llm/vllm_nvext/entrypoints/args.py”, line 143, in prepare_environment
engine_args, extracted_name = inject_ngc_hub(engine_args)
File “/opt/nim/llm/vllm_nvext/hub/ngc_injector.py”, line 220, in inject_ngc_hub
cached = repo.get_all()
Exception: The requested operation requires an API key, but none was found