AF2 NIM cannot authenticate on apptainer

tyrone_lee · December 11, 2024, 10:09pm

I’m trying to run the alphafold2 nim on a DGX that’s in an HPC. There are no docker modules, instead i pulled the container with singularity/apptainer:

export APPTAINER_DOCKER_USERNAME='$oauthtoken'
export APPTAINER_DOCKER_PASSWORD='nvapi-*****************************'
apptainer pull af2_nim-2-1-0.sif docker://nvcr.io/nim/deepmind/alphafold2:latest

Trying to run NIM on the DGX within an interactive terminal or with SLURM sbatch using the following commands in apptainer throws an authentication error:

(base) [user@dgx09  home]$ apptainer run --nv --bind $LOCAL_NIM_CACHE:/opt/nim/.cache  --env NGC_API_KEY=$NGC_API_KEY af2_nim-2-1-0.sif
INFO:    underlay of /usr/bin/nvidia-smi required more than 50 (480) bind mounts
WARNING: group: unknown groupid 2190919
15:4: not a valid test operator:
15:4: not a valid test operator: 12.6
21:4: not a valid test operator: (
21:4: not a valid test operator: 550.54.15
"timestamp": "2024-12-11 17:04:36,147", "level": "INFO", "message": "Starting nimlib 0.3.7"
"timestamp": "2024-12-11 17:04:36,147", "level": "INFO", "message": "NIM VERSION:"
1.0.0
"timestamp": "2024-12-11 17:04:36,148", "level": "INFO", "message": "NIM LICENSE:"
***LISCENCE***
"timestamp": "2024-12-11 17:04:36,148", "level": "INFO", "message": "NIM banner.txt:"
AlphaFold2 was originally developed by Google DeepMind.
This NIM has been packaged and optimized by NVIDIA to support sequence-to-structure, sequence-to-msa, and msa-to-structure inference.
"timestamp": "2024-12-11 17:04:36,175", "level": "INFO", "message": "Matched profile_id in manifest from env NIM_MODEL_PROFILE to: 4df59ce8aa1ff5a0d88e808316d0c83ea0c3bde259e9da3a12a1cdc7a54e2635"
"timestamp": "2024-12-11 17:04:36,175", "level": "INFO", "message": "Using the profile selected by the profile selector: 4df59ce8aa1ff5a0d88e808316d0c83ea0c3bde259e9da3a12a1cdc7a54e2635"
"timestamp": "2024-12-11 17:04:36,175", "level": "INFO", "message": "Using the workspace specified during init: /opt/nim/workspace"
"timestamp": "2024-12-11 17:04:36,175", "level": "INFO", "message": "Downloading manifest profile: 4df59ce8aa1ff5a0d88e808316d0c83ea0c3bde259e9da3a12a1cdc7a54e2635"
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:128] One or more errors fetching files:
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.397 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
[12-11 17:04:42.398 ERROR  nim_sdk::hub::repo  rust/nim-sdk/src/hub/repo.rs:130] Authentication Error
"timestamp": "2024-12-11 17:04:42,399", "level": "ERROR", "message": "Error downloading models, ignoring and continuing startup"
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/nimlib/nim_sdk.py", line 313, in download_models
    cache = repo.get_all()  # download model artifacts to cache
Exception: Authentication Error

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/nimlib/nimutils.py", line 45, in download_models
    model_manifest.download_models()
  File "/usr/local/lib/python3.10/dist-packages/nimlib/nim_sdk.py", line 339, in download_models
    raise ManifestDownloadError(
nimlib.exceptions.ManifestDownloadError: Error downloading manifest to cache: /opt/nim/workspace error: Authentication Error
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/nimlib/nim_sdk.py", line 313, in download_models
    cache = repo.get_all()  # download model artifacts to cache
Exception: Authentication Error

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/start_server", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/nimlib/start_server.py", line 73, in main
    nimutils.download_models(model_manifest)
  File "/usr/local/lib/python3.10/dist-packages/nimlib/nimutils.py", line 45, in download_models
    model_manifest.download_models()
  File "/usr/local/lib/python3.10/dist-packages/nimlib/nim_sdk.py", line 339, in download_models
    raise ManifestDownloadError(
nimlib.exceptions.ManifestDownloadError: Error downloading manifest to cache: /opt/nim/workspace error: Authentication Error

I am passing in the right environment flags but I’ve tried different generated api keys and personal keys but nothing works. Did something break in the apptainer conversion or is there an additional authentication key I’m missing to get this NIM to start?

mayk · January 28, 2025, 1:19am

I am getting the same error while trying to run the container nvcr.io/nim/deepmind/alphafold2:latestin modal.

Could you solve that?

mayk · February 12, 2025, 6:19pm

Hey @tyrone_lee .

I am not sure I solved the problem. But I could get past this error by exporting NGC_API_KEY to my container. I’m unfamiliar with apptainer, but is it possible that --env NGC_API_KEY=$NGC_API_KEY isn’t working correctly?

Topic		Replies	Views
NIM embedding model downloads but fails with auth error on startup Access/Accounts nim , nv-embedqa-e5-v5	27	517	February 11, 2025
Alphafold2 Models nim	8	45	January 27, 2025
Alphafold2 bug Models nim	4	107	February 12, 2025
Getting Started With NVIDIA NIM Tutorial Issues with NGC Registry Access/Accounts ubuntu , nim , llm , llama3-8b-instruct	7	1107	July 24, 2024
[SUPPORT] Workbench Example Project: Hybrid RAG NVIDIA AI Workbench workbench-example-project	88	1850	February 20, 2025
I have an issue when installing NVIDIA NIM on laptop Models nim	6	412	September 18, 2024
Running NIM llama-3_1-8b-instruct fails in On-Prem deployment Models nim , llama	6	218	February 24, 2025
Model says there is a compatible profile but fails on data type Models nim , mistral-7b-instruct-v03	4	537	August 21, 2024
NIM API key not Found Models nim , llama-31-8b-instruct , llama	4	409	September 21, 2024
DeepStream 2.0 error in running gstreamer，Can anyone solve this problem？ thanks DeepStream SDK	16	2726	September 7, 2023

AF2 NIM cannot authenticate on apptainer

Related topics