EDIT: solved, was a silly mistake, when configuring the .credentials file, the $oauthtoken does not go into the command. So the correct command is
srun --container-image='nvidia/cuda:11.2.2-devel-ubuntu20.04' --gres=gpu:a100:1 --pty nvidia-smi -L
And it works.
– OLDER MESSAGE –
Having the same problem here,
Did what matthias.leopold mentioned, but still failing.
I followed the documentation and made sure ENROOT_CONFIG_PATH environment variable is set.
➜ ~ echo $ENROOT_CONFIG_PATH
/home/MYUSER/.config/enroot
My Credential file is
➜ ~ cat $HOME/.config/enroot/.credentials
# NVIDIA GPU Cloud (both endpoints are required)
machine nvcr.io login $oauthtoken password MYNGCTOKEN
machine authn.nvidia.com login $oauthtoken password MYNGCTOKEN
# DockerHub
# machine auth.docker.io login <login> password <passord>
# Google Container Registry
# machine gcr.io login oauth2accesstoken password $(gcloud auth print-access-token)
# machine gcr.io login _json_key password $(jq -c '.' $GOOGLE_APPLICATION_CREDENTIALS | sed 's/ /\\u0020/g')
Finally, running a test example fails
➜ ~ srun --container-image='docker://$oauthtoken@nvcr.io#nvidia/cuda:11.2.2-devel-ubuntu20.04' --gres=gpu:a100:1 --pty nvidia-smi -L
pyxis: importing docker image ...
slurmstepd: error: pyxis: child 810178 failed with error code: 1
slurmstepd: error: pyxis: failed to import docker image
slurmstepd: error: pyxis: printing contents of log file ...
slurmstepd: error: pyxis: [INFO] Querying registry for permission grant
slurmstepd: error: pyxis: [INFO] Authenticating with user: docker://$oauthtoken
slurmstepd: error: pyxis: [ERROR] URL https://nvcr.io/proxy_auth returned error code: 401 Unauthorized
slurmstepd: error: pyxis: couldn't start container
slurmstepd: error: pyxis: if the image has an unusual entrypoint, try using --no-container-entrypoint
slurmstepd: error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
slurmstepd: error: Failed to invoke spank plugin stack
srun: error: nodeGPU01: task 0: Exited with exit code 1
any ideas what is missing? thanks in advance