Installation issues clara train - Unable to run rootless container via nvidia runtime

Hello everyone,

I am trying to install nvidia clara train platform. However, I am facing an issue while trying to run rootless container via nvidia runtime. My docker deamon is running successfully. But when I try to run the 2nd line of command for the client, I get the error as shown below

"OCI runtime create failed: container_linux.go:345: starting container process caused “process_linux.go:424: container init caused “process_linux.go:407: running prestart hook 1 caused \“fork/exec /data/volume02/selva/usernetes/bin/nvidia-container-runtime-hook: exec format error\”””: unknown.
"

I have shown the two lines of command. I get the above error when I execute the 2nd line of command

./run.sh default-docker-nokube  # this works fine and daemon initialization is success
                                               
docker -H unix://$XDG_RUNTIME_DIR/docker.sock run --runtime=nvidia -- rm -it nvidia/cuda:10.0-devel nvidia-smi  #this command throws error

I thought that the above issue might be due to cgroups. However I created a config.toml file (with cgroups = True) but still I get the same error. Can anyone help me resolve this issue?

Since the error happened on the cuda container, it’s likely a driver issue. Can you post your host nvidia driver information?

And please try to follow https://github.com/NVIDIA/nvidia-docker to install nvidia docker runtime. Thank you.

Hello,

The above issue is resolved. Thank you