Hi there, it’s me again. [image] Train with my own tlt model TAO Toolkit I have already trained a tlt model and I want to use it as a pretrained model to train a new model. However, when I try to run it directly from the commands I have given, the following err…

It is normal. For experiment, you can set below to check the mAP at the beginning. soft_start_annealing_schedule { min_learning_rate: 10e-10 max_learning_rate: 10e-10

Did you ever create ~/.tlt_mounts.json or ~/.tao_mounts.json ? More, how did you meet above error? Can you share the detailed steps? How did you trigger the docker? Use tao launcher or not? The full command when you run into error.

Thanks for answering. I trigger the docker through typing the command above. Than I use create a jupyter notebook and download the example. I follow the step in the example " Object Detection using TAO SSD" In the first block “Set up env variables and map drives” I run the following cod: # Mappi…

[image] tmid0819: sudo docker run Could you add “--runtime=nvidia” when you trigger docker? Refer to an old version of TLT 2.0 user guide. https://docs.nvidia.com/tao/archive/tlt-20/tlt-user-guide/text/requirements_and_installation.html#running-the-transfer-learning-toolkit

Thank you for your reply. After adding the “–untime=nvidia”, a different error occurred when I ran the tfrecords program. Log: Converting the training set to TFRecords. 2021-12-29 05:32:26,430 [INFO] root: Registry: ['nvcr.io'] 2021-12-29 05:32:26,563 [INFO] tlt.components.instance_handler.local_…

May I know which gpu did you run on?

[image] tmid0819: cuda.init() Please check if you are running on Ubuntu. There is also a similar topic which has the same error. No CUDA-capable device is detected on tao detectnet_v2 dataset convert - #4 by NilsAI . For that topic, adding the tag --privileged helped run without the issue…

Thanks. I try to use the method you gave me in the article. I also remove Tao from the instruction. !ssd dataset_convert \ -d $SPECS_DIR/ssd_tfrecords_kitti_train.txt \ -o $DATA_DOWNLOAD_DIR/tfrecords/kitti_train It works! But why? What went wrong in the process that caused me t…

[image] tmid0819: I also remove Tao from the instruction. I see. You were triggering the docker via "docker run xxx ". In this way, there is not needed to use “tao”. See TAO Launcher — Tao Toolkit For example, to train a detectnet_v2 model in the interactive session, run the following com…

Hi @Morganh , after debugging, I can perform model training smoothly. But facing the initial problem, I was still stuck. In tao 3.21.11, I try to use my 38 class model to train 52 new classes. This is my code: !ssd train --gpus 1 --gpu_index=$GPU_INDEX \ -e $SPECS_DIR/ssd_train_re…

Train with my own tlt model #2

Accelerated Computing Intelligent Video Analytics TAO Toolkit

tmid0819 December 29, 2021, 6:29am 17

Sure, I’m using GeForce RTX 2080 Ti.

pycuda._driver.Error: cuInit failed: unknown error

Topic		Replies	Views
Train with my own tlt model TAO Toolkit	13	931	November 23, 2021
Training Peoplent on custom data TAO Toolkit ai-training	18	3150	July 5, 2021
TLT - retrain trafficcamnet with customized data precision is 0 TAO Toolkit	19	1053	June 16, 2021
TLT V2.0 Classification TAO Toolkit	25	3253	December 30, 2020
Change prediction layer of pretrained model TAO TAO Toolkit tao	7	375	July 8, 2024
SSD: custom tlt training result in AP:0 for all my classes TAO Toolkit ssd , ai-training	5	1199	August 28, 2020
Unable to train SSD-Resnet-18 TAO Toolkit	15	2197	November 8, 2019
Error wile using TLT pretrained model tlt_semantic_segmentation:resnet101 TAO Toolkit	6	729	June 28, 2021
License Plate Recognition TAO Toolkit	13	1507	June 20, 2022
Original error: could not get source code TAO Toolkit	7	807	July 6, 2022

Train with my own tlt model #2

Related topics