Nvidia TAO Yolo_v3 training failure

Swapnadip_Moni · May 14, 2024, 11:19am

Please provide the following information when requesting support.

• Hardware (T4/V100/Xavier/Nano/etc)
Tesla T4
• Network Type (Detectnet_v2/Faster_rcnn/Yolo_v4/LPRnet/Mask_rcnn/Classification/etc)
Yolo_v3 ResNet_18
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here)
• Training spec file(If have, please share here)

yolo_v3_train_resnet18_tfrecord.txt (1.9 KB)

• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)

Morganh · May 14, 2024, 2:26pm

There is not specific error. Can you share the full log?

Swapnadip_Moni · May 14, 2024, 6:01pm

I am using the below configuration for training:
random_seed: 42
yolov3_config {
big_anchor_shape: “[(114.94, 60.67), (159.06, 114.59), (297.59, 176.38)]”
mid_anchor_shape: “[(42.99, 31.91), (79.57, 31.75), (56.80, 56.93)]”
small_anchor_shape: “[(15.60, 13.88), (30.25, 20.25), (20.67, 49.63)]”
matching_neutral_box_iou: 0.7
arch: “resnet”
nlayers: 18
arch_conv_blocks: 2
loss_loc_weight: 0.8
loss_neg_obj_weights: 100.0
loss_class_weights: 1.0
freeze_bn: false
#freeze_blocks: 0
force_relu: false
}
training_config {
batch_size_per_gpu: 10
num_epochs: 20
enable_qat: false
checkpoint_interval: 10
learning_rate {
soft_start_annealing_schedule {
min_learning_rate: 1e-6
max_learning_rate: 1e-4
soft_start: 0.1
annealing: 0.2
}
}
regularizer {
type: L1
weight: 3e-5
}
optimizer {
adam {
epsilon: 1e-7
beta1: 0.9
beta2: 0.999
amsgrad: false
}
}
pretrain_model_path: “/content/drive/MyDrive/results/yolo_v3/pretrained_resnet18/pretrained_object_detection_vresnet18/resnet_18.hdf5”
}
eval_config {
average_precision_mode: SAMPLE
batch_size: 8
matching_iou_threshold: 0.5
}
nms_config {
confidence_threshold: 0.001
clustering_iou_threshold: 0.5
top_k: 200
force_on_cpu: True
}
augmentation_config {
hue: 0.1
saturation: 1.5
exposure:1.5
vertical_flip:0
horizontal_flip: 0.5
jitter: 0.3
output_width: 960
output_height: 544
output_channel: 3
randomize_input_shape_period: 0
}
dataset_config {
data_sources: {
tfrecords_path: “/content/drive/MyDrive/Final_Nvidia_Cable_dataset/tfrecords/kitti_trainval/kitti_trainval*”
image_directory_path: “/content/drive/MyDrive/Final_Nvidia_Cable_dataset/”
}
include_difficult_in_training: true
image_extension: “jpg”
target_class_mapping {
key: “break”
value: “break”
}
target_class_mapping {
key: “thunderbolt”
value: “thunderbolt”
}
validation_fold: 0
}

Swapnadip_Moni · May 14, 2024, 6:03pm

I am using the Google Colab notebook provided by Nvidia Tao

after this command :

print(“To run with multigpu, please change --gpus based on the number of available GPUs in your machine.”)
!tao model yolo_v3 train -e $SPECS_DIR/yolo_v3_train_resnet18_tfrecord.txt
-r $EXPERIMENT_DIR/experiment_dir_unpruned
-k $KEY
–gpus 1

getting below error :

To run with multigpu, please change --gpus based on the number of available GPUs in your machine.
Using TensorFlow backend.
2024-05-14 17:55:27.488314: I tensorflow/stream_executor/platform/default/dso_loader.cc:50] Successfully opened dynamic library libcudart.so.12
2024-05-14 17:55:27,543 [TAO Toolkit] [WARNING] tensorflow 40: Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
2024-05-14 17:55:28,551 [TAO Toolkit] [WARNING] root 329: Limited tf.compat.v2.summary API due to missing TensorBoard installation.
2024-05-14 17:55:29,116 [TAO Toolkit] [WARNING] root 329: Limited tf.compat.v2.summary API due to missing TensorBoard installation.
2024-05-14 17:55:31,641 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.export.trt_utils 36: Failed to import TensorRT package, exporting TLT to a TensorRT engine will not be available.
2024-05-14 17:55:31,641 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.export.base_exporter 44: Failed to import TensorRT package, exporting TLT to a TensorRT engine will not be available.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:root:Limited tf.compat.v2.summary API due to missing TensorBoard installation.
WARNING:root:Limited tf.compat.v2.summary API due to missing TensorBoard installation.
WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/yolo_v3/scripts/train.py:54: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/yolo_v3/scripts/train.py:54: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/yolo_v3/scripts/train.py:57: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:2018: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

WARNING:tensorflow:The operation tf.image.convert_image_dtype will be skipped since the input and output dtypes are identical.
WARNING:tensorflow:The operation tf.image.convert_image_dtype will be skipped since the input and output dtypes are identical.
WARNING:tensorflow:The operation tf.image.convert_image_dtype will be skipped since the input and output dtypes are identical.
WARNING:tensorflow:The operation tf.image.convert_image_dtype will be skipped since the input and output dtypes are identical.
WARNING:tensorflow:The operation tf.image.convert_image_dtype will be skipped since the input and output dtypes are identical.
WARNING:tensorflow:The operation tf.image.convert_image_dtype will be skipped since the input and output dtypes are identical.
/usr/local/lib/python3.8/dist-packages/third_party/keras/tensorflow_backend.py:367: UserWarning: Seed 42 from outer graph might be getting used by function Dataset_map__map_func_set_random_wrapper, if the random op has not been provided any seed. Explicitly set the seed in the function if this is not the intended behavior.
return old_map(
WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/yolo_v3/data_loader/data_loader.py:188: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/yolo_v3/data_loader/data_loader.py:188: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/third_party/keras/tensorflow_backend.py:199: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:3295: The name tf.log is deprecated. Please use tf.math.log instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

/usr/local/lib/python3.8/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was not compiled. Compile it manually.
warnings.warn('No training configuration found in save file: ’

Layer (type) Output Shape Param # Connected to

Input (InputLayer) (None, 3, None, None 0

conv1 (Conv2D) (None, 64, None, Non 9408 Input[0][0]

bn_conv1 (BatchNormalization) (None, 64, None, Non 256 conv1[0][0]

activation_2 (Activation) (None, 64, None, Non 0 bn_conv1[0][0]

block_1a_conv_1 (Conv2D) (None, 64, None, Non 36864 activation_2[0][0]

block_1a_bn_1 (BatchNormalizati (None, 64, None, Non 256 block_1a_conv_1[0][0]

block_1a_relu_1 (Activation) (None, 64, None, Non 0 block_1a_bn_1[0][0]

block_1a_conv_2 (Conv2D) (None, 64, None, Non 36864 block_1a_relu_1[0][0]

block_1a_conv_shortcut (Conv2D) (None, 64, None, Non 4096 activation_2[0][0]

block_1a_bn_2 (BatchNormalizati (None, 64, None, Non 256 block_1a_conv_2[0][0]

block_1a_bn_shortcut (BatchNorm (None, 64, None, Non 256 block_1a_conv_shortcut[0][0]

add_9 (Add) (None, 64, None, Non 0 block_1a_bn_2[0][0]
block_1a_bn_shortcut[0][0]

block_1a_relu (Activation) (None, 64, None, Non 0 add_9[0][0]

block_1b_conv_1 (Conv2D) (None, 64, None, Non 36864 block_1a_relu[0][0]

block_1b_bn_1 (BatchNormalizati (None, 64, None, Non 256 block_1b_conv_1[0][0]

block_1b_relu_1 (Activation) (None, 64, None, Non 0 block_1b_bn_1[0][0]

block_1b_conv_2 (Conv2D) (None, 64, None, Non 36864 block_1b_relu_1[0][0]

block_1b_conv_shortcut (Conv2D) (None, 64, None, Non 4096 block_1a_relu[0][0]

block_1b_bn_2 (BatchNormalizati (None, 64, None, Non 256 block_1b_conv_2[0][0]

block_1b_bn_shortcut (BatchNorm (None, 64, None, Non 256 block_1b_conv_shortcut[0][0]

add_10 (Add) (None, 64, None, Non 0 block_1b_bn_2[0][0]
block_1b_bn_shortcut[0][0]

block_1b_relu (Activation) (None, 64, None, Non 0 add_10[0][0]

block_2a_conv_1 (Conv2D) (None, 128, None, No 73728 block_1b_relu[0][0]

block_2a_bn_1 (BatchNormalizati (None, 128, None, No 512 block_2a_conv_1[0][0]

block_2a_relu_1 (Activation) (None, 128, None, No 0 block_2a_bn_1[0][0]

block_2a_conv_2 (Conv2D) (None, 128, None, No 147456 block_2a_relu_1[0][0]

block_2a_conv_shortcut (Conv2D) (None, 128, None, No 8192 block_1b_relu[0][0]

block_2a_bn_2 (BatchNormalizati (None, 128, None, No 512 block_2a_conv_2[0][0]

block_2a_bn_shortcut (BatchNorm (None, 128, None, No 512 block_2a_conv_shortcut[0][0]

add_11 (Add) (None, 128, None, No 0 block_2a_bn_2[0][0]
block_2a_bn_shortcut[0][0]

block_2a_relu (Activation) (None, 128, None, No 0 add_11[0][0]

block_2b_conv_1 (Conv2D) (None, 128, None, No 147456 block_2a_relu[0][0]

block_2b_bn_1 (BatchNormalizati (None, 128, None, No 512 block_2b_conv_1[0][0]

block_2b_relu_1 (Activation) (None, 128, None, No 0 block_2b_bn_1[0][0]

block_2b_conv_2 (Conv2D) (None, 128, None, No 147456 block_2b_relu_1[0][0]

block_2b_conv_shortcut (Conv2D) (None, 128, None, No 16384 block_2a_relu[0][0]

block_2b_bn_2 (BatchNormalizati (None, 128, None, No 512 block_2b_conv_2[0][0]

block_2b_bn_shortcut (BatchNorm (None, 128, None, No 512 block_2b_conv_shortcut[0][0]

add_12 (Add) (None, 128, None, No 0 block_2b_bn_2[0][0]
block_2b_bn_shortcut[0][0]

block_2b_relu (Activation) (None, 128, None, No 0 add_12[0][0]

block_3a_conv_1 (Conv2D) (None, 256, None, No 294912 block_2b_relu[0][0]

block_3a_bn_1 (BatchNormalizati (None, 256, None, No 1024 block_3a_conv_1[0][0]

block_3a_relu_1 (Activation) (None, 256, None, No 0 block_3a_bn_1[0][0]

block_3a_conv_2 (Conv2D) (None, 256, None, No 589824 block_3a_relu_1[0][0]

block_3a_conv_shortcut (Conv2D) (None, 256, None, No 32768 block_2b_relu[0][0]

block_3a_bn_2 (BatchNormalizati (None, 256, None, No 1024 block_3a_conv_2[0][0]

block_3a_bn_shortcut (BatchNorm (None, 256, None, No 1024 block_3a_conv_shortcut[0][0]

add_13 (Add) (None, 256, None, No 0 block_3a_bn_2[0][0]
block_3a_bn_shortcut[0][0]

block_3a_relu (Activation) (None, 256, None, No 0 add_13[0][0]

block_3b_conv_1 (Conv2D) (None, 256, None, No 589824 block_3a_relu[0][0]

block_3b_bn_1 (BatchNormalizati (None, 256, None, No 1024 block_3b_conv_1[0][0]

block_3b_relu_1 (Activation) (None, 256, None, No 0 block_3b_bn_1[0][0]

block_3b_conv_2 (Conv2D) (None, 256, None, No 589824 block_3b_relu_1[0][0]

block_3b_conv_shortcut (Conv2D) (None, 256, None, No 65536 block_3a_relu[0][0]

block_3b_bn_2 (BatchNormalizati (None, 256, None, No 1024 block_3b_conv_2[0][0]

block_3b_bn_shortcut (BatchNorm (None, 256, None, No 1024 block_3b_conv_shortcut[0][0]

add_14 (Add) (None, 256, None, No 0 block_3b_bn_2[0][0]
block_3b_bn_shortcut[0][0]

block_3b_relu (Activation) (None, 256, None, No 0 add_14[0][0]

block_4a_conv_1 (Conv2D) (None, 512, None, No 1179648 block_3b_relu[0][0]

block_4a_bn_1 (BatchNormalizati (None, 512, None, No 2048 block_4a_conv_1[0][0]

block_4a_relu_1 (Activation) (None, 512, None, No 0 block_4a_bn_1[0][0]

block_4a_conv_2 (Conv2D) (None, 512, None, No 2359296 block_4a_relu_1[0][0]

block_4a_conv_shortcut (Conv2D) (None, 512, None, No 131072 block_3b_relu[0][0]

block_4a_bn_2 (BatchNormalizati (None, 512, None, No 2048 block_4a_conv_2[0][0]

block_4a_bn_shortcut (BatchNorm (None, 512, None, No 2048 block_4a_conv_shortcut[0][0]

add_15 (Add) (None, 512, None, No 0 block_4a_bn_2[0][0]
block_4a_bn_shortcut[0][0]

block_4a_relu (Activation) (None, 512, None, No 0 add_15[0][0]

block_4b_conv_1 (Conv2D) (None, 512, None, No 2359296 block_4a_relu[0][0]

block_4b_bn_1 (BatchNormalizati (None, 512, None, No 2048 block_4b_conv_1[0][0]

block_4b_relu_1 (Activation) (None, 512, None, No 0 block_4b_bn_1[0][0]

block_4b_conv_2 (Conv2D) (None, 512, None, No 2359296 block_4b_relu_1[0][0]

block_4b_conv_shortcut (Conv2D) (None, 512, None, No 262144 block_4a_relu[0][0]

block_4b_bn_2 (BatchNormalizati (None, 512, None, No 2048 block_4b_conv_2[0][0]

block_4b_bn_shortcut (BatchNorm (None, 512, None, No 2048 block_4b_conv_shortcut[0][0]

add_16 (Add) (None, 512, None, No 0 block_4b_bn_2[0][0]
block_4b_bn_shortcut[0][0]

block_4b_relu (Activation) (None, 512, None, No 0 add_16[0][0]

yolo_expand_conv1 (Conv2D) (None, 512, None, No 2359296 block_4b_relu[0][0]

yolo_expand_conv1_bn (BatchNorm (None, 512, None, No 2048 yolo_expand_conv1[0][0]

yolo_expand_conv1_lrelu (LeakyR (None, 512, None, No 0 yolo_expand_conv1_bn[0][0]

yolo_conv1_1 (Conv2D) (None, 256, None, No 131072 yolo_expand_conv1_lrelu[0][0]

yolo_conv1_1_bn (BatchNormaliza (None, 256, None, No 1024 yolo_conv1_1[0][0]

yolo_conv1_1_lrelu (LeakyReLU) (None, 256, None, No 0 yolo_conv1_1_bn[0][0]

yolo_conv1_2 (Conv2D) (None, 512, None, No 1179648 yolo_conv1_1_lrelu[0][0]

yolo_conv1_2_bn (BatchNormaliza (None, 512, None, No 2048 yolo_conv1_2[0][0]

yolo_conv1_2_lrelu (LeakyReLU) (None, 512, None, No 0 yolo_conv1_2_bn[0][0]

yolo_conv1_3 (Conv2D) (None, 256, None, No 131072 yolo_conv1_2_lrelu[0][0]

yolo_conv1_3_bn (BatchNormaliza (None, 256, None, No 1024 yolo_conv1_3[0][0]

yolo_conv1_3_lrelu (LeakyReLU) (None, 256, None, No 0 yolo_conv1_3_bn[0][0]

yolo_conv1_4 (Conv2D) (None, 512, None, No 1179648 yolo_conv1_3_lrelu[0][0]

yolo_conv1_4_bn (BatchNormaliza (None, 512, None, No 2048 yolo_conv1_4[0][0]

yolo_conv1_4_lrelu (LeakyReLU) (None, 512, None, No 0 yolo_conv1_4_bn[0][0]

yolo_conv1_5 (Conv2D) (None, 256, None, No 131072 yolo_conv1_4_lrelu[0][0]

yolo_conv1_5_bn (BatchNormaliza (None, 256, None, No 1024 yolo_conv1_5[0][0]

yolo_conv1_5_lrelu (LeakyReLU) (None, 256, None, No 0 yolo_conv1_5_bn[0][0]

yolo_conv2 (Conv2D) (None, 128, None, No 32768 yolo_conv1_5_lrelu[0][0]

yolo_conv2_bn (BatchNormalizati (None, 128, None, No 512 yolo_conv2[0][0]

yolo_conv2_lrelu (LeakyReLU) (None, 128, None, No 0 yolo_conv2_bn[0][0]

upsample0 (UpSampling2D) (None, 128, None, No 0 yolo_conv2_lrelu[0][0]

concatenate_3 (Concatenate) (None, 384, None, No 0 upsample0[0][0]
block_3b_relu[0][0]

yolo_conv3_1 (Conv2D) (None, 128, None, No 49152 concatenate_3[0][0]

yolo_conv3_1_bn (BatchNormaliza (None, 128, None, No 512 yolo_conv3_1[0][0]

yolo_conv3_1_lrelu (LeakyReLU) (None, 128, None, No 0 yolo_conv3_1_bn[0][0]

yolo_conv3_2 (Conv2D) (None, 256, None, No 294912 yolo_conv3_1_lrelu[0][0]

yolo_conv3_2_bn (BatchNormaliza (None, 256, None, No 1024 yolo_conv3_2[0][0]

yolo_conv3_2_lrelu (LeakyReLU) (None, 256, None, No 0 yolo_conv3_2_bn[0][0]

yolo_conv3_3 (Conv2D) (None, 128, None, No 32768 yolo_conv3_2_lrelu[0][0]

yolo_conv3_3_bn (BatchNormaliza (None, 128, None, No 512 yolo_conv3_3[0][0]

yolo_conv3_3_lrelu (LeakyReLU) (None, 128, None, No 0 yolo_conv3_3_bn[0][0]

yolo_conv3_4 (Conv2D) (None, 256, None, No 294912 yolo_conv3_3_lrelu[0][0]

yolo_conv3_4_bn (BatchNormaliza (None, 256, None, No 1024 yolo_conv3_4[0][0]

yolo_conv3_4_lrelu (LeakyReLU) (None, 256, None, No 0 yolo_conv3_4_bn[0][0]

yolo_conv3_5 (Conv2D) (None, 128, None, No 32768 yolo_conv3_4_lrelu[0][0]

yolo_conv3_5_bn (BatchNormaliza (None, 128, None, No 512 yolo_conv3_5[0][0]

yolo_conv3_5_lrelu (LeakyReLU) (None, 128, None, No 0 yolo_conv3_5_bn[0][0]

yolo_conv4 (Conv2D) (None, 64, None, Non 8192 yolo_conv3_5_lrelu[0][0]

yolo_conv4_bn (BatchNormalizati (None, 64, None, Non 256 yolo_conv4[0][0]

yolo_conv4_lrelu (LeakyReLU) (None, 64, None, Non 0 yolo_conv4_bn[0][0]

upsample1 (UpSampling2D) (None, 64, None, Non 0 yolo_conv4_lrelu[0][0]

concatenate_4 (Concatenate) (None, 192, None, No 0 upsample1[0][0]
block_2b_relu[0][0]

yolo_conv5_1 (Conv2D) (None, 64, None, Non 12288 concatenate_4[0][0]

yolo_conv5_1_bn (BatchNormaliza (None, 64, None, Non 256 yolo_conv5_1[0][0]

yolo_conv5_1_lrelu (LeakyReLU) (None, 64, None, Non 0 yolo_conv5_1_bn[0][0]

yolo_conv5_2 (Conv2D) (None, 128, None, No 73728 yolo_conv5_1_lrelu[0][0]

yolo_conv5_2_bn (BatchNormaliza (None, 128, None, No 512 yolo_conv5_2[0][0]

yolo_conv5_2_lrelu (LeakyReLU) (None, 128, None, No 0 yolo_conv5_2_bn[0][0]

yolo_conv5_3 (Conv2D) (None, 64, None, Non 8192 yolo_conv5_2_lrelu[0][0]

yolo_conv5_3_bn (BatchNormaliza (None, 64, None, Non 256 yolo_conv5_3[0][0]

yolo_conv5_3_lrelu (LeakyReLU) (None, 64, None, Non 0 yolo_conv5_3_bn[0][0]

yolo_conv5_4 (Conv2D) (None, 128, None, No 73728 yolo_conv5_3_lrelu[0][0]

yolo_conv5_4_bn (BatchNormaliza (None, 128, None, No 512 yolo_conv5_4[0][0]

yolo_conv5_4_lrelu (LeakyReLU) (None, 128, None, No 0 yolo_conv5_4_bn[0][0]

yolo_conv5_5 (Conv2D) (None, 64, None, Non 8192 yolo_conv5_4_lrelu[0][0]

yolo_conv5_5_bn (BatchNormaliza (None, 64, None, Non 256 yolo_conv5_5[0][0]

yolo_conv5_5_lrelu (LeakyReLU) (None, 64, None, Non 0 yolo_conv5_5_bn[0][0]

yolo_conv1_6 (Conv2D) (None, 512, None, No 1179648 yolo_conv1_5_lrelu[0][0]

yolo_conv3_6 (Conv2D) (None, 256, None, No 294912 yolo_conv3_5_lrelu[0][0]

yolo_conv5_6 (Conv2D) (None, 128, None, No 73728 yolo_conv5_5_lrelu[0][0]

yolo_conv1_6_bn (BatchNormaliza (None, 512, None, No 2048 yolo_conv1_6[0][0]

yolo_conv3_6_bn (BatchNormaliza (None, 256, None, No 1024 yolo_conv3_6[0][0]

yolo_conv5_6_bn (BatchNormaliza (None, 128, None, No 512 yolo_conv5_6[0][0]

yolo_conv1_6_lrelu (LeakyReLU) (None, 512, None, No 0 yolo_conv1_6_bn[0][0]

yolo_conv3_6_lrelu (LeakyReLU) (None, 256, None, No 0 yolo_conv3_6_bn[0][0]

yolo_conv5_6_lrelu (LeakyReLU) (None, 128, None, No 0 yolo_conv5_6_bn[0][0]

conv_big_object (Conv2D) (None, 21, None, Non 10773 yolo_conv1_6_lrelu[0][0]

conv_mid_object (Conv2D) (None, 21, None, Non 5397 yolo_conv3_6_lrelu[0][0]

conv_sm_object (Conv2D) (None, 21, None, Non 2709 yolo_conv5_6_lrelu[0][0]

bg_permute (Permute) (None, None, None, 2 0 conv_big_object[0][0]

md_permute (Permute) (None, None, None, 2 0 conv_mid_object[0][0]

sm_permute (Permute) (None, None, None, 2 0 conv_sm_object[0][0]

bg_anchor (YOLOAnchorBox) (None, None, 6) 0 conv_big_object[0][0]

bg_reshape (Reshape) (None, None, 7) 0 bg_permute[0][0]

md_anchor (YOLOAnchorBox) (None, None, 6) 0 conv_mid_object[0][0]

md_reshape (Reshape) (None, None, 7) 0 md_permute[0][0]

sm_anchor (YOLOAnchorBox) (None, None, 6) 0 conv_sm_object[0][0]

sm_reshape (Reshape) (None, None, 7) 0 sm_permute[0][0]

encoded_bg (Concatenate) (None, None, 13) 0 bg_anchor[0][0]
bg_reshape[0][0]

encoded_md (Concatenate) (None, None, 13) 0 md_anchor[0][0]
md_reshape[0][0]

encoded_sm (Concatenate) (None, None, 13) 0 sm_anchor[0][0]
sm_reshape[0][0]

encoded_detections (Concatenate (None, None, 13) 0 encoded_bg[0][0]
encoded_md[0][0]
encoded_sm[0][0]

Total params: 19,161,983
Trainable params: 19,140,863
Non-trainable params: 21,120

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/yolo_v3/utils/tensor_utils.py:7: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/yolo_v3/utils/tensor_utils.py:8: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/yolo_v3/utils/tensor_utils.py:9: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

Epoch 1/20
Telemetry data couldn’t be sent, but the command ran successfully.
[WARNING]: ‘str’ object has no attribute ‘decode’
Execution status: FAIL

Morganh · May 15, 2024, 2:09am

Please double check the key is correct. You can set explicitly instead of $KEY.

Swapnadip_Moni · May 15, 2024, 4:50am

I am using the Google colab notebook provided by Nvidia. The $KEY is supposed to authenticate automatically.

Morganh · May 15, 2024, 6:19am

I am afraid you did not run this cell correctly. So, $KEY is none.

Please set it explicitly. That means, change to below in the command line.
-k nvidia_tlt

yingliu · June 11, 2024, 2:13am

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

system · June 25, 2024, 2:13am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Nvidia TAO Yolo_v3 training failure

Layer (type) Output Shape Param # Connected to

encoded_detections (Concatenate (None, None, 13) 0 encoded_bg[0][0] encoded_md[0][0] encoded_sm[0][0]

encoded_detections (Concatenate (None, None, 13) 0 encoded_bg[0][0]
encoded_md[0][0]
encoded_sm[0][0]