When running this command: “docker run --rm --name alphafold2 --runtime=nvidia
-e NGC_API_KEY
-v $LOCAL_NIM_CACHE:/opt/nim/.cache
-p 8000:8000
nvcr.io/nim/deepmind/alphafold2:2.0.0”
I got errors like below:
nim_hub_ngc::api::tokio /home/mmorais/.cargo/registry/src/urm.nvidia.com-53d20a091fff2cec/nim-hub-ngc-0.4.5/src/api/tokio.rs:814] Error creating symlink: I/O error No such file or directory (os error 2)
Hi @yongjinhou please can you share the full logs from the docker run command? Thanks!
I didn’t save the log for that run. Later, I tried with 1.0.0 and failed and got different error messages. Here it is the command and complete log with 1.0.0 failure.
docker run --rm --name alphafold2 --runtime=nvidia -e NGC_API_KEY -v $LOCAL_NIM_CACHE:/opt/nim/.cache -p 8002:8000 nvcr.io/nim/deepmind/alphafold2:1.0.0
Below is the log of deploying alpha2:1.0.0 failure
2025-01-23 06:49:03 {“timestamp”: “2025-01-23 11:49:03,140”, “level”: “INFO”, “message”: “NIM banner.txt:”}
2025-01-23 06:49:03 AlphaFold2 was originally developed by Google DeepMind.
2025-01-23 06:49:03 This NIM has been packaged and optimized by NVIDIA to support sequence-to-structure, sequence-to-msa, and msa-to-sequence inference.
2025-01-23 06:49:03 {“timestamp”: “2025-01-23 11:49:03,142”, “level”: “INFO”, “message”: “Using model config profile ID ‘default’”}
2025-01-23 06:49:03 {“timestamp”: “2025-01-23 11:49:03,868”, “level”: “INFO”, “message”: “Initializing NGC SDK client with NGC org 0601755190998017”}
2025-01-23 06:49:04 {“timestamp”: “2025-01-23 11:49:04,484”, “level”: “INFO”, “message”: “Downloading model to: /opt/nim/.cache/alphafold2-data_v1.0.0”}
2025-01-23 06:49:04 {“timestamp”: “2025-01-23 11:49:04,485”, “level”: “INFO”, “message”: “Downloading model ‘nim/deepmind/alphafold2-data:1.0.0’ from NGC to /opt/nim/.cache/”}
2025-01-23 20:58:17 {
2025-01-23 20:58:17 “download_end”: “2025-01-24 01:58:12”,
2025-01-23 20:58:17 “download_start”: “2025-01-23 11:49:05”,
2025-01-23 20:58:17 “download_time”: “14h 9m 8s”,
2025-01-23 20:58:17 “files_downloaded”: 29,
2025-01-23 20:58:17 “local_path”: “/opt/nim/.cache/alphafold2-data_v1.0.0”,
2025-01-23 20:58:17 “size_downloaded”: “145.21 GB”,
2025-01-23 20:58:17 “status”: “FAILED”
2025-01-23 20:58:17 }
2025-01-23 20:58:17 {“timestamp”: “2025-01-24 01:58:17,014”, “level”: “INFO”, “message”: “Starting NIM inference server”}
2025-01-23 20:58:18 File not found: /opt/nim/.cache/alphafold2-data_v1.0.0/mgnify/mgy_clusters_2022_05.fa
2025-01-23 20:58:18 File not found: /opt/nim/.cache/alphafold2-data_v1.0.0/pdb_mmcif/pdb_mmcif.hdf5
2025-01-23 20:58:18 {“timestamp”: “2025-01-24 01:58:18,793”, “level”: “INFO”, “message”: “Serving HTTP server on port 8000”}
2025-01-24 09:52:51 File not found: /opt/nim/.cache/alphafold2-data_v1.0.0/mgnify/mgy_clusters_2022_05.fa
2025-01-24 09:52:51 INFO: 172.17.0.1:34798 - “GET /v1/health/ready HTTP/1.1” 503 Service Unavailable
2025-01-23 20:58:18 INFO: Started server process [1]
2025-01-23 20:58:18 INFO: Waiting for application startup.
2025-01-23 20:58:18 INFO: Application startup complete.
2025-01-23 20:58:18 INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
Ran When running this command: “docker run --rm --name alphafold2 --runtime=nvidia
-e NGC_API_KEY
-v $LOCAL_NIM_CACHE:/opt/nim/.cache
-p 8000:8000
nvcr.io/nim/deepmind/alphafold2:2.0.0". Got the following error in the log: “This NIM has been packaged and optimized by NVIDIA to support sequence-to-structure, sequence-to-msa, and msa-to-structure inference.
“timestamp”: “2025-01-24 22:25:54,370”, “level”: “INFO”, “message”: “Matched profile_id in manifest from env NIM_MODEL_PROFILE to: 4df59ce8aa1ff5a0d88e808316d0c83ea0c3bde259e9da3a12a1cdc7a54e2635”
“timestamp”: “2025-01-24 22:25:54,370”, “level”: “INFO”, “message”: “Using the profile selected by the profile selector: 4df59ce8aa1ff5a0d88e808316d0c83ea0c3bde259e9da3a12a1cdc7a54e2635”
“timestamp”: “2025-01-24 22:25:54,370”, “level”: “INFO”, “message”: “Using the workspace specified during init: /opt/nim/workspace”
“timestamp”: “2025-01-24 22:25:54,370”, “level”: “INFO”, “message”: “Downloading manifest profile: 4df59ce8aa1ff5a0d88e808316d0c83ea0c3bde259e9da3a12a1cdc7a54e2635”
[01-24 23:01:49.520 ERROR nim_hub_ngc::api::tokio /home/mmorais/.cargo/registry/src/urm.nvidia.com-53d20a091fff2cec/nim-hub-ngc-0.4.5/src/api/tokio.rs:814] Error creating symlink: I/O error No such file or directory (os error 2)
[01-24 23:01:57.704 ERROR nim_hub_ngc::api::tokio /home/mmorais/.cargo/registry/src/urm.nvidia.com-53d20a091fff2cec/nim-hub-ngc-0.4.5/src/api/tokio.rs:814] Error creating symlink: I/O error No such file or directory (os error 2)
[01-24 23:06:45.290 ERROR nim_hub_ngc::api::tokio /home/mmorais/.cargo/registry/src/urm.nvidia.com-53d20a091fff2cec/nim-hub-ngc-0.4.5/src/api/tokio.rs:814] Error creating symlink: I/O error No such file or directory (os error 2)
[01-24 23:06:47.083 ERROR nim_hub_ngc::api::tokio /home/mmorais/.cargo/registry/src/urm.nvidia.com-53d20a091fff2cec/nim-hub-ngc-0.4.5/src/api/tokio.rs:814] Error creating symlink: I/O error No such file or directory (os error 2)
[01-24 23:08:26.476 ERROR nim_hub_ngc::api::tokio /home/mmorais/.cargo/registry/src/urm.nvidia.com-53d20a091fff2cec/nim-hub-ngc-0.4.5/src/api/tokio.rs:814] Error creating symlink: I/O error No such file or directory (os error 2)
[01-24 23:35:15.408 ERROR nim_hub_ngc::api::tokio /home/mmorais/.cargo/registry/src/urm.nvidia.com-53d20a091fff2cec/nim-hub-ngc-0.4.5/src/api/tokio.rs:814] Error creating symlink: I/O error No such file or directory (os error 2)
[01-24 23:35:42.708 ERROR nim_hub_ngc::api::tokio /home/mmorais/.cargo/registry/src/urm.nvidia.com-53d20a091fff2cec/nim-hub-ngc-0.4.5/src/api/tokio.rs:814] Error creating symlink: I/O error No such file or directory (os error 2)
[01-24 23:56:19.001 ERROR nim_hub_ngc::api::tokio /home/mmorais/.cargo/registry/src/urm.nvidia.com-53d20a091fff2cec/nim-hub-ngc-0.4.5/src/api/tokio.rs:814] Error creating symlink: I/O error No such file or directory (os error 2)
[01-25 00:04:39.276 ERROR nim_hub_ngc::api::tokio /home/mmorais/.cargo/registry/src/urm.nvidia.com-53d20a091fff2cec/nim-hub-ngc-0.4.5/src/api/tokio.rs:814] Error creating symlink: I/O error No such file or directory (os error 2)
[01-25 00:05:07.909 ERROR nim_hub_ngc::api::tokio /home/mmorais/.cargo/registry/src/urm.nvidia.com-53d20a091fff2cec/nim-hub-ngc-0.4.5/src/api/tokio.rs:814] Error creating symlink: I/O error No such file or directory (os error 2)
[01-25 00:14:27.638 ERROR nim_hub_ngc::api::tokio /home/mmorais/.cargo/registry/src/urm.nvidia.com-53d20a091fff2cec/nim-hub-ngc-0.4.5/src/api/tokio.rs:814] Error creating symlink: I/O error No such file or directory (os error 2)
[01-25 00:14:29.461 ERROR nim_hub_ngc::api::tokio /home/mmorais/.cargo/registry/src/urm.nvidia.com-53d20a091fff2cec/nim-hub-ngc-0.4.5/src/api/tokio.rs:814] Error creating symlink: I/O error No such file or directory (os error 2)
[01-25 00:14:31.609 ERROR nim_hub_ngc::api::tokio /home/mmorais/.cargo/registry/src/urm.nvidia.com-53d20a091fff2cec/nim-hub-ngc-0.4.5/src/api/tokio.rs:814] Error creating symlink: I/O error No such file or directory (os error 2)
[01-25 00:24:09.961 ERROR nim_hub_ngc::api::tokio /home/mmorais/.cargo/registry/src/urm.nvidia.com-53d20a091fff2cec/nim-hub-ngc-0.4.5/src/api/tokio.rs:814] Error creating symlink: I/O error No such file or directory (os error 2)
[01-25 01:20:52.072 ERROR nim_hub_ngc::api::tokio /home/mmorais/.cargo/registry/src/urm.nvidia.com-53d20a091fff2cec/nim-hub-ngc-0.4.5/src/api/tokio.rs:814] Error creating symlink: I/O error No such file or directory (os error 2)
[01-25 01:20:54.262 ERROR nim_hub_ngc::api::tokio /home/mmorais/.cargo/registry/src/urm.nvidia.com-53d20a091fff2cec/nim-hub-ngc-0.4.5/src/api/tokio.rs:814] Error creating symlink: I/O error No such file or directory (os error 2)
[01-25 01:21:45.651 ERROR nim_hub_ngc::api::tokio /home/mmorais/.cargo/registry/src/urm.nvidia.com-53d20a091fff2cec/nim-hub-ngc-0.4.5/src/api/tokio.rs:814] Error creating symlink: I/O error No such file or directory (os error 2)
[01-25 01:22:24.602 ERROR nim_hub_ngc::api::tokio /home/mmorais/.cargo/registry/src/urm.nvidia.com-53d20a091fff2cec/nim-hub-ngc-0.4.5/src/api/tokio.rs:814] Error creating symlink: I/O error No such file or directory (os error 2)
[01-25 01:34:23.499 ERROR nim_hub_ngc::api::tokio /home/mmorais/.cargo/registry/src/urm.nvidia.com-53d20a091fff2cec/nim-hub-ngc-0.4.5/src/api/tokio.rs:814] Error creating symlink: I/O error No such file or directory (os error 2)”
Hi @yongjinhou, it looks like there’s some issue with the way the $LOCAL_NIM_CACHE is mounted. Is this on a local drive, and does your user have permissions to modify it?
yes, local on my Ubuntu machine mounted to a local drive
Can you also try using the “latest” tag container tag? I.e. nvcr.io/nim/deepmind/alphafold2:latest
I tried latest initially and didn’t work either.
Ok, we’ll try to see if we can reproduce this internally. In the meantime, a few more troubleshooting thoughts – could you be running out of disk space? Can you also try deleting the cache directory and retrying the deployment?