Please provide the following information when requesting support.
• Hardware GeForce RTX 4080
• Network Type LPRnet
• TLT Version Configuration of the TAO Toolkit Instance
task_group: [‘model’, ‘dataset’, ‘deploy’]
format_version: 3.0
toolkit_version: 5.5.0
published_date: 08/26/2024
• Training spec file:
random_seed: 42
lpr_config {
hidden_units: 512
max_label_length: 8
arch: "baseline"
nlayers: 18 #setting nlayers to be 10 to use baseline10 model
}
training_config {
batch_size_per_gpu: 32
num_epochs: 24
learning_rate {
soft_start_annealing_schedule {
min_learning_rate: 1e-6
max_learning_rate: 1e-5
soft_start: 0.001
annealing: 0.5
}
}
regularizer {
type: L2
weight: 5e-4
}
}
eval_config {
validation_period_during_training: 5
batch_size: 1
}
augmentation_config {
output_width: 96
output_height: 48
output_channel: 3
max_rotate_degree: 5
rotate_prob: 0.5
gaussian_kernel_size: 5
gaussian_kernel_size: 7
gaussian_kernel_size: 15
blur_prob: 0.5
reverse_color_prob: 0.5
keep_original_prob: 0.3
}
dataset_config {
data_sources: {
label_directory_path: "/workspace/tao-experiments/data/openalpr/train/label"
image_directory_path: "/workspace/tao-experiments/data/openalpr/train/image"
}
characters_list_file: "/workspace/tao-experiments/lprnet/specs/us_lp_characters.txt"
validation_data_sources: {
label_directory_path: "/workspace/tao-experiments/data/openalpr/val/label"
image_directory_path: "/workspace/tao-experiments/data/openalpr/val/image"
}
}
• How to reproduce the issue ? No training… !tao model lprnet train --gpus=1 --gpu_index=$GPU_INDEX
-e $SPECS_DIR/tutorial_spec.txt
-k $KEY
-r $USER_EXPERIMENT_DIR/experiment_dir_unpruned
-m $USER_EXPERIMENT_DIR/pretrained_lprnet_baseline18/lprnet_vtrainable_v1.0/us_lprnet_baseline18_trainable.tlt
log: For multi-GPU, change --gpus based on your machine.
2025-11-02 15:43:55,465 [TAO Toolkit] [INFO] root 160: Registry: [‘nvcr.io’]
2025-11-02 15:43:55,575 [TAO Toolkit] [INFO] nvidia_tao_cli.components.instance_handler.local_instance 360: Running command in container: nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf1.15.5
2025-11-02 15:43:56,209 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 301: Printing tty value True
2025-11-02 18:43:59.698170: I tensorflow/stream_executor/platform/default/dso_loader.cc:50] Successfully opened dynamic library libcudart.so.12
2025-11-02 18:44:00,152 [TAO Toolkit] [WARNING] tensorflow 40: Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
2025-11-02 18:44:07,992 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use sklearn by default. This improves performance in some cases. To enable sklearn export the environment variable TF_ALLOW_IOLIBS=1.
2025-11-02 18:44:08,175 [TAO Toolkit] [WARNING] tensorflow 42: TensorFlow will not use Dask by default. This improves performance in some cases. To enable Dask export the environment variable TF_ALLOW_IOLIBS=1.
2025-11-02 18:44:08,197 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use Pandas by default. This improves performance in some cases. To enable Pandas export the environment variable TF_ALLOW_IOLIBS=1.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
WARNING:tensorflow:TensorFlow will not use sklearn by default. This improves performance in some cases. To enable sklearn export the environment variable TF_ALLOW_IOLIBS=1.
WARNING: TensorFlow will not use sklearn by default. This improves performance in some cases. To enable sklearn export the environment variable TF_ALLOW_IOLIBS=1.
WARNING:tensorflow:TensorFlow will not use Dask by default. This improves performance in some cases. To enable Dask export the environment variable TF_ALLOW_IOLIBS=1.
WARNING: TensorFlow will not use Dask by default. This improves performance in some cases. To enable Dask export the environment variable TF_ALLOW_IOLIBS=1.
WARNING:tensorflow:TensorFlow will not use Pandas by default. This improves performance in some cases. To enable Pandas export the environment variable TF_ALLOW_IOLIBS=1.
WARNING: TensorFlow will not use Pandas by default. This improves performance in some cases. To enable Pandas export the environment variable TF_ALLOW_IOLIBS=1.
WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/lprnet/scripts/train.py:78: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.
WARNING: From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/lprnet/scripts/train.py:78: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.
WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/lprnet/scripts/train.py:81: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
WARNING: From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/lprnet/scripts/train.py:81: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/lprnet/scripts/train.py:82: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.
WARNING: From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/lprnet/scripts/train.py:82: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.
INFO: Log file already exists at /workspace/tao-experiments/lprnet/experiment_dir_unpruned/status.json
INFO: Merging specification from /workspace/tao-experiments/lprnet/specs/tutorial_spec.txt
INFO: Loading pretrained weights. This may take a while…
INFO: Training was interrupted
INFO: Training was interrupted.
Telemetry data couldn’t be sent, but the command ran successfully.
[WARNING]: HTTPSConnectionPool(host=‘telemetry.metropolis.nvidia.com’, port=443): Max retries exceeded with url: /api/v1/telemetry (Caused by SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1131)’)))
Execution status: PASS
2025-11-02 15:44:25,984 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 363: Stopping container.