Slurmstepd error URL 21.09-tf2-py3 returned error code: 401 Unauthorized

Hi,
I’ve set up a Slurm-cluster following by deepops/docs/slurm-cluster at master · NVIDIA/deepops · GitHub. After completing the slurm-cluster installation I attempted to run the command ansible-playbook -l slurm-cluster playbooks/slurm-cluster/slurm-validation.yml with non-root user. However, I encountered an issue about pulling the image.

TASK [Compiling NCCL tests] **********************************************************************************************************************************************************************************
fatal: [31-control]: FAILED! => changed=true
cmd: |-
srun -p batch -N 1
–ntasks-per-node=1
–cpus-per-task=10
–container-image=“nvcr.io/nvidia/tensorflow:21.09-tf2-py3
–container-save=“${HOME}/enroot_images/nccl_tests_slurm_val.sqsh”
–container-remap-root

bash -c 'mkdir -p /opt/nccl_tests
cd /opt/nccl_tests
NCCL_TESTS_COMMITISH=f773748b46
wget -q -O - https://github.com/NVIDIA/nccl-tests/archive/${NCCL_TESTS_COMMITISH}.tar.gz |
tar --strip-components=1 -xzf -
&& CC=mpicc CXX=mpicxx make MPI=1
cp -R /opt/nccl_tests/build/* /usr/local/bin/

delta: ‘0:00:04.535931’
end: ‘2024-01-11 03:33:13.533776’
msg: non-zero return code
rc: 1
start: ‘2024-01-11 03:33:08.997845’
stderr: |-
pyxis: importing docker image …
slurmstepd: error: pyxis: child 27145 failed with error code: 1
slurmstepd: error: pyxis: failed to import docker image
slurmstepd: error: pyxis: printing contents of log file …
slurmstepd: error: pyxis: [INFO] Querying registry for permission grant
slurmstepd: error: pyxis: [INFO] Authenticating with user:
slurmstepd: error: pyxis: [INFO] Authentication succeeded
slurmstepd: error: pyxis: [INFO] Fetching image manifest list
slurmstepd: error: pyxis: [INFO] Fetching image manifest
slurmstepd: error: pyxis: [ERROR] URL https://registry-1.docker.io/v2/nvcr.io/nvidia/tensorflow/manifests/21.09-tf2-py3 returned error code: 401 Unauthorized
slurmstepd: error: pyxis: couldn’t start container
slurmstepd: error: pyxis: if the image has an unusual entrypoint, try using --no-container-entrypoint
slurmstepd: error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
slurmstepd: error: Failed to invoke spank plugin stack
srun: error: 2-compute: task 0: Exited with exit code 1
stderr_lines:
stdout: ‘’
stdout_lines:

I’m using ubuntu 22.04.1 LTS. And the cluster comprises 1 control node and 2 compute nodes. I have set up credential on all these three nodes for both root and non-root user. While the command docker pull nvcr.io/nvidia/tensorflow:21.09-tf2-py3 works successfully, so I curious to identify the step I may missed? Thanks.