My docker was working before. Today run command below and have error. !tao mask_rcnn train -e $SPECS_DIR/maskrcnn_train_resnet50.txt \ -d $USER_EXPERIMENT_DIR/experiment_dir_unpruned\ -k $KEY \ --gpus 1 The whole message before contain…

Oh got it, thanks! Changed it to tao-toolkit-lm:v3.22.05-py3 and it works now

For workaround, please try below in terminal. $ docker run --runtime=nvidia -it --rm --entrypoint “” -v yourlocaldolder:dockerfolder nvcr.io/nvidia/tao/tao-toolkit-tf:v3.22.05-tf1.15.5-py3 /bin/bash

Why I can’t run from jupyter as before and how can I fix back? The above mentioned command works. But I prefer using running command from Jupyter

[image] Morganh: --entrypoint “” Since yesterday, there is a new update in ngccli which results in this issue. Internal team is working on that to update the launcher with a fix. Above is just a workaround. I will check another workaround for jupyter.

That means rather than running as normal in Jupyter, I run docker in bash and work inside container? What should I set for yourlocaldolder:dockerfolder?

For triggering jupyter notebook, see below example, $ docker run --runtime=nvidia -it --rm --entrypoint “” -v ~/demo_3.0:/workspace/demo_3.0 -p 8888:8888 nvcr.io/nvidia/tao/tao-toolkit-tf:v3.22.05-tf1.15.5-py3 /bin/bash then, root@8d6c08489e41:/workspace# jupyter notebook --ip 0.0.0.0 --allow-r…

This command !tao info gives /bin/sh: 1: tao: not found

For this workaround, please ignore tao launcher. For example, if you run training, root@03d61af590ba:/workspace# mask_rcnn train --help Using TensorFlow backend. usage: mask_rcnn train [-h] [--num_processes NUM_PROCESSES] [--gpus GPUS] [--gpu_index GPU_INDEX [GPU_INDEX ...]]…

[image] Morganh: docker run --runtime=nvidia -it --rm --entrypoint “” -v ~/demo_3.0:/workspace/demo_3.0 -p 8888:8888 nvcr.io/nvidia/tao/tao-toolkit-tf:v3.22.05-tf1.15.5-py3 /bin/bash This workaround will always work. Even after team has fixed the changes, does this workaround still can be use…

[image] edit_or: Even after team has fixed the changes, does this workaround still can be used? Yes.

Thanks start training. Let see how what else I still need to fix. Yesterday issue still have. I am going to remove images giving the issue.

Tlt.components.docker_handler.docker_handler: Stopping container

Accelerated Computing Intelligent Video Analytics TAO Toolkit

edit_or June 16, 2022, 5:42am 15

The issue is solved using wrokaround method

Training with multiple GPUs has error using TAO toolkit

Topic		Replies	Views
Tlt.components.docker_handler.docker_handler: Stopping container TAO Toolkit	18	1462	July 6, 2022
Error running tao container image TAO Toolkit	4	646	June 15, 2022
Issue with tlt.components.docker_handler.docker_handler: Stopping container TAO Toolkit	16	1384	June 20, 2022
CLI update TAO Toolkit	13	1367	June 23, 2022
TAO Toolkit Container(for Conversational AI) Setup Issue TAO Toolkit	11	819	November 30, 2022
While installing TAO, ERROR: nvidia-docker not found TAO Toolkit tao	10	369	August 8, 2025
Issues Running Jupyter Notebook While setting up TAO TAO Toolkit	8	534	March 10, 2025
LPRNet Error TAO Toolkit	12	496	June 5, 2024
About tao_mounts.json and docker container stop in traning cell TAO Toolkit	6	1034	July 6, 2022
Error setting up TAO Toolkit - 'nvidia-docker not found' TAO Toolkit ubuntu , python	27	653	June 12, 2025

Tlt.components.docker_handler.docker_handler: Stopping container

Related topics