Issue with Docker instantiation while converting Tfrecords for KITTI trainval dataset in TAO Toolkit

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

I’m encountering an error while attempting to convert Tfrecords for the KITTI trainval dataset using the TAO Toolkit in a Jupyter Notebook environment. The error message I’m receiving is as follows:

Converting Tfrecords for KITTI trainval dataset
2024-04-04 13:02:23,884 [TAO Toolkit] [INFO] root 160: Registry: ['nvcr.io']
2024-04-04 13:02:24,010 [TAO Toolkit] [INFO] nvidia_tao_cli.components.instance_handler.local_instance 361: Running command in container: nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf1.15.5
2024-04-04 13:02:24,033 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 301: Printing tty value True
Docker instantiation failed with error: 500 Server Error: Internal Server Error ("failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'csv'
invoking the NVIDIA Container Runtime Hook directly (e.g. specifying the docker --gpus flag) is not supported. Please use the NVIDIA Container Runtime (e.g. specify the --runtime=nvidia flag) instead.: unknown")

I’m seeking assistance in resolving this issue. From my research, it appears to be related to Docker instantiation and the use of the NVIDIA Container Runtime. However, I’m unsure how to resolve it.

Could someone please provide guidance on how to properly configure Docker to use the NVIDIA Container Runtime with the --runtime=nvidia flag? Any insights or troubleshooting steps would be greatly appreciated.

Thank you!

2

Please use below.

$ curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | \
sudo apt-key add -

$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
$ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
sudo tee /etc/apt/sources.list.d/nvidia-docker.list
$ sudo apt-get update
$ sudo apt-get install -y nvidia-docker2
$ sudo pkill -SIGHUP dockerd
3

Thank you for the instructions. I followed the steps you provided, including adding the NVIDIA Docker repository, updating, installing nvidia-docker2, and restarting Docker daemon. However, I’m still encountering the same error when attempting to convert Tfrecords for the KITTI trainval dataset in the TAO Toolkit.

Do you have any additional suggestions or troubleshooting steps that I can try to resolve this issue?

Thank you for your assistance.

4

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

Which gpu did you run? Could you share the result of $nvidia-smi ?

Please try with New computer install GPU Docker error - #6 by david9xqqb , especially, sudo systemctl restart docker.service

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.