TAO data services Error response from daemon: No such container dataset convert error from kitti to COCO

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Please provide the following information when requesting support.

• Hardware NVIDIA Geoforce RTX 3060
• Network Type (tao dataset convert KITTI To COCO)

• TLT Version

tao info
Configuration of the TAO Toolkit Instance
task_group: [‘model’, ‘dataset’, ‘deploy’]
format_version: 3.0
toolkit_version: 5.3.0
published_date: 03/14/2024

prerequisites installed
(tao_launcher) quest@INTVMLT3947:~$ docker --version
Docker version 26.1.0, build 9714adc
(tao_launcher) quest@INTVMLT3947:~$ docker version --format ‘{{.Server.APIVersion}}’
1.45
(tao_launcher) quest@INTVMLT3947:~$ dpkg -l | grep nvidia-container-toolkit
ii nvidia-container-toolkit 1.15.0-1 amd64 NVIDIA Container toolkit
ii nvidia-container-toolkit-base 1.15.0-1 amd64 NVIDIA Container Toolkit Base
(tao_launcher) quest@INTVMLT3947:~$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
(tao_launcher) quest@INTVMLT3947:~$ nvidia-smi --query-gpu=driver_version --format=csv,noheader
535.171.04
(tao_launcher) quest@INTVMLT3947:~$ lsvirtualenv
tao_launcher

)

SPEC FILE:
data:
input_format: “KITTI”
output_format: “COCO”
output_dir: “/results”
kitti:
image_dir: “/data/images”
label_dir: “/data/labels”
coco:
ann_file: “/results/data.json”

Errors/How to reproduce error:
command:

Convert KITTI to COCO

!tao dataset annotations convert -e $SPECS_DIR/convert.yaml
Error:
2024-04-24 16:39:34,801 [TAO Toolkit] [INFO] root 160: Registry: [‘nvcr.io’]
2024-04-24 16:39:34,862 [TAO Toolkit] [INFO] nvidia_tao_cli.components.instance_handler.local_instance 360: Running command in container: nvcr.io/nvidia/tao/tao-toolkit:5.3.0-data-services
2024-04-24 16:39:34,872 [TAO Toolkit] [WARNING] nvidia_tao_cli.components.docker_handler.docker_handler 288:
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the “user”:“UID:GID” in the
DockerOptions portion of the “/home/quest/.tao_mounts.json” file. You can obtain your
users UID and GID by using the “id -u” and “id -g” commands on the
terminal.
2024-04-24 16:39:34,872 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 301: Printing tty value True
Error response from daemon: No such container: 970ab60aec24bf8da4efb1a7722d6e0fb586c62b4b6b6d8a6bdb365aa9108cf5
2024-04-24 16:39:37,204 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 363: Stopping container.

3

Can you run below successfully?
! tao dataset annotations run cat $SPECS_DIR/convert.yaml

To narrow down, you can open a terminal instead to use the following command to login the docker, and then run command without tao dataset in the beginning.
$ tao dataset annotations run /bin/bash
Then,
# annotations convert xxx

4

(launcher) quest@quest-ROG-Strix-G533ZX-G533ZXZ:~/getting_started_v5.3.0/notebooks/tao_data_services$ tao dataset
usage: tao dataset [-h]
{list,stop,info,dataset,deploy,model} …
{augmentation,auto_label,annotations,analytics} …

optional arguments:
-h, --help show this help message and exit

task_groups:
{list,stop,info,dataset,deploy,model}

task:
{augmentation,auto_label,annotations,analytics}
(launcher) quest@quest-ROG-Strix-G533ZX-G533ZXZ:~/getting_started_v5.3.0/notebooks/tao_data_services$ tao dataset annotations run /bin/bash
2024-05-07 17:24:26,965 [TAO Toolkit] [INFO] root 160: Registry: [‘nvcr.io’]
2024-05-07 17:24:27,002 [TAO Toolkit] [INFO] nvidia_tao_cli.components.instance_handler.local_instance 361: Running command in container: nvcr.io/nvidia/tao/tao-toolkit:5.3.0-data-services
2024-05-07 17:24:27,008 [TAO Toolkit] [WARNING] nvidia_tao_cli.components.docker_handler.docker_handler 293:
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the “user”:“UID:GID” in the
DockerOptions portion of the “/home/quest/.tao_mounts.json” file. You can obtain your
users UID and GID by using the “id -u” and “id -g” commands on the
terminal.
2024-05-07 17:24:27,008 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 301: Printing tty value True
Error response from daemon: No such container: 7ba0b88a485fa41b4b6e0fcbcdbde6673b145ea58657eadfc96794e8e080593a
2024-05-07 17:24:29,355 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 363: Stopping container.
(launcher) quest@quest-ROG-Strix-G533ZX-G533ZXZ:~/getting_started_v5.3.0/notebooks/tao_data_services$

5

initially when i run the command it didn’t show any issue related to this… it was showing the data was not there in the directory then i updated the directory with coco json dataset then it is showing the container issue. I had reinstalled all the requirements from start, for the first time running the command there was no issue but after this issue pops up…

6

Could you login the docker via below way?
$ docker run --runtime=nvidia -it --rm nvcr.io/nvidia/tao/tao-toolkit:5.3.0-data-services /bin/bash

7

docker run --runtime=nvidia -it --rm nvcr.io/nvidia/tao/tao-toolkit:5.3.0-data-services /bin/bash
docker: Error response from daemon: unknown or invalid runtime name: nvidia.
See ‘docker run --help’.

8

Please install nvidia-docker. Refer to New computer install GPU Docker error - #6 by david9xqqb.

9

(launcher) quest@quest-ROG-Strix-G533ZX-G533ZXZ:~/getting_started_v5.3.0/notebooks/tao_data_services$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nvcr.io/nvidia/tao/tao-toolkit 5.3.0-data-services a64cab8c3779 2 months ago 23.9GB
hello-world latest d2c94e258dcb 12 months ago 13.3kB
(launcher) quest@quest-ROG-Strix-G533ZX-G533ZXZ:~/getting_started_v5.3.0/notebooks/tao_data_services$ tao dataset auto_label generate
2024-05-14 16:30:06,779 [TAO Toolkit] [INFO] root 160: Registry: [‘nvcr.io’]
2024-05-14 16:30:06,814 [TAO Toolkit] [INFO] nvidia_tao_cli.components.instance_handler.local_instance 361: Running command in container: nvcr.io/nvidia/tao/tao-toolkit:5.3.0-data-services
2024-05-14 16:30:06,822 [TAO Toolkit] [WARNING] nvidia_tao_cli.components.docker_handler.docker_handler 293:
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the “user”:“UID:GID” in the
DockerOptions portion of the “/home/quest/.tao_mounts.json” file. You can obtain your
users UID and GID by using the “id -u” and “id -g” commands on the
terminal.
2024-05-14 16:30:06,822 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 301: Printing tty value True
Error response from daemon: No such container: 935e44b7ae5b7e7bab5cb925b77c0be88928a6837fc484738728d4e3e07731f6
2024-05-14 16:30:09,154 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 363: Stopping container.
(launcher) quest@quest-ROG-Strix-G533ZX-G533ZXZ:~/getting_started_v5.3.0/notebooks/tao_data_services$

still its not working, each time it is trying to fetch new container and the container is not created

10

docker pull nvcr.io/nvidia/tao/tao-toolkit:5.3.0-data-services

used this command to pull docker containers for 5.3.0-data-services

11

Can you
$ docker run --runtime=nvidia -it --rm nvcr.io/nvidia/tao/tao-toolkit:5.3.0-data-services /bin/bash

Then inside the docker, run
# annotations convert xxx

12

root@ff8b96e19013:/opt/nvidia# (launcher) quest@quest-ROG-Strix-G533ZX-G533ZXZ:~/getting_started_v5.3.0/notebooks/tao_data_services$
(launcher) quest@quest-ROG-Strix-G533ZX-G533ZXZ:~/getting_started_v5.3.0/notebooks/tao_data_services$ docker run --runtime=nvidia -it --rm nvcr.io/nvidia/tao/tao-toolkit:5.3.0-data-services /bin/bash

=================================
=== TAO Toolkit Data-Services ===

NVIDIA Release 5.3.0-DataServices (build 76438008)
TAO Toolkit Version 5.3.0

Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the TAO Toolkit End User License Agreement.
By pulling and using the container, you accept the terms and conditions of this license:

WARNING: CUDA Minor Version Compatibility mode ENABLED.
Using driver version 535.171.04 which has support for CUDA 12.2. This container
was built with CUDA 12.3 and will be run in Minor Version Compatibility mode.
CUDA Forward Compatibility is preferred over Minor Version Compatibility for use
with this container but was unavailable:
[[Forward compatibility was attempted on non supported HW (CUDA_ERROR_COMPAT_NOT_SUPPORTED_ON_DEVICE) cuInit()=804]]
See 1. Why CUDA Compatibility — CUDA Compatibility r550 documentation for details.

NOTE: The SHMEM allocation limit is set to the default of 64MB. This may be
insufficient for TAO Toolkit. NVIDIA recommends the use of the following flags:
docker run --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 …

root@6aaf92f27c07:/opt/nvidia# annotations convert
usage: annotation [-h] [-r RESULTS_DIR] -e EXPERIMENT_SPEC_FILE [-g GPUS] {convert,merge,slice}
annotation: error: the following arguments are required: -e/–experiment_spec_file
root@6aaf92f27c07:/opt/nvidia# annotations convert -h
usage: annotation [-h] [-r RESULTS_DIR] -e EXPERIMENT_SPEC_FILE [-g GPUS] {convert,merge,slice}

Annotation entrypoint

positional arguments:
{convert,merge,slice}
Subtask for a given task/model.

options:
-h, --help show this help message and exit
-r RESULTS_DIR, --results_dir RESULTS_DIR
Path to a folder where the experiment outputs should be written. (DEFAULT: ./)
-e EXPERIMENT_SPEC_FILE, --experiment_spec_file EXPERIMENT_SPEC_FILE
Path to the experiment spec file.
-g GPUS, --gpus GPUS Number of GPUs or gpu index to use.
root@6aaf92f27c07:/opt/nvidia#

Do I need to change my CUDA versionto 12.3?

13

Can you share the result of
$ nvidia-smi

14

It is not needed.

15

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

16

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.