Training TAO on synthetic data images not producing A.P. score

I have been training some synthetic images generated with the Omniverse ISSAC sim and the images are not producing a A.P. score when it is training the model. Now that being said it does not seem to effect how well the synthetic images can be recognized when I run the images with the trained model.
Just wondering why the synthetic images are not generating a A.P. score.
I have attached 2 pic that may explain it better

Could you share training spec and some example label file of the evaluation dataset? Thanks.

here is spec file
and label file examples

forklift 0.00 3 0.00 503 49 746 368 0.00 0.00 0.00 0.00 0.00 0.00 0.00
forklift 0.00 3 0.00 700 158 833 319 0.00 0.00 0.00 0.00 0.00 0.00 0.00

random_seed: 42
yolov4_config {
big_anchor_shape: “[(158.00, 87.99), (223.20, 177.29), (322.00, 362.00)]”
mid_anchor_shape: “[(64.94, 30.71), (92.00, 54.45), (67.46, 147.00)]”
small_anchor_shape: “[(16.55, 15.87), (32.59, 23.51), (39.30, 46.92)]”
box_matching_iou: 0.25
matching_neutral_box_iou: 0.5
arch: “darknet”
nlayers: 53
arch_conv_blocks: 2
loss_loc_weight: 1.0
loss_neg_obj_weights: 1.0
loss_class_weights: 1.0
label_smoothing: 0.0
big_grid_xy_extend: 0.05
mid_grid_xy_extend: 0.1
small_grid_xy_extend: 0.2
freeze_bn: false
#freeze_blocks: 0
force_relu: false
}
training_config {
batch_size_per_gpu: 4
num_epochs: 80
enable_qat: false
checkpoint_interval: 10
learning_rate {
soft_start_cosine_annealing_schedule {
min_learning_rate: 1e-7
max_learning_rate: 1e-4
soft_start: 0.3
}
}
regularizer {
type: L1
weight: 3e-5
}
optimizer {
adam {
epsilon: 1e-7
beta1: 0.9
beta2: 0.999
amsgrad: false
}
}
pretrain_model_path: “/workspace/tao-experiments/yolo_v4/pretrained_resnet18/pretrained_object_detection_vdarknet53/darknet_53.hdf5”
}
eval_config {
average_precision_mode: SAMPLE
batch_size: 4
matching_iou_threshold: 0.5
}
nms_config {
confidence_threshold: 0.001
clustering_iou_threshold: 0.5
force_on_cpu: true
top_k: 200
}
augmentation_config {
hue: 0.1
saturation: 1.5
exposure:1.5
vertical_flip:0
horizontal_flip: 0.5
jitter: 0.3
output_width: 1248
output_height: 384
output_channel: 3
randomize_input_shape_period: 0
mosaic_prob: 0.5
mosaic_min_ratio:0.2
}
dataset_config {
data_sources: {
tfrecords_path: “/workspace/tao-experiments/data/training/tfrecords/train*”
image_directory_path: “/workspace/tao-experiments/data/training”
}
include_difficult_in_training: true
image_extension: “png”
target_class_mapping {
key: “car”
value: “car”
}
target_class_mapping {
key: “pedestrian”
value: “pedestrian”
}
target_class_mapping {
key: “forklift”
value: “forklift”
}
target_class_mapping {
key: “truck”
value: “truck”
}
validation_data_sources: {
tfrecords_path: “/workspace/tao-experiments/data/val/tfrecords/val*”
image_directory_path: “/workspace/tao-experiments/data/val”
}
}

How about change 503 49 746 368 to
503.0 49.0 746.0 368.0?

More, to narrow down,could you run “tao evaluate” against part of training tfrecords files?

Here is a snippet of the script from ominverse issac utility that generates the kitti label and annotations.
How do i modify it so the script generates the annotations with the X.00 format?

/home/joev/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.synthetic_utils/omni/isaac/synthetic_utils/writers/kitti.py

Check if bounding boxes are in the viewport
if (
x_min < 0
or y_min < 0
or x_max > viewport_width
or y_max > viewport_height
or x_min > viewport_width
or y_min > viewport_height
or y_max < 0
or x_max < 0
):
continue

        semantic_label = str(box[2])

        # Skip label if not in class list
        if self.classes != [] and semantic_label not in self.classes:
            continue

        # Adding Kitting Data,  NOTE: Only class and 2d bbox coordinates are filled in
        label.append(semantic_label)
        label.append(f"{0.00:.2f}")
        label.append(3)
        label.append(f"{0.00:.2f}")
        label.append(x_min)
        label.append(y_min)
        label.append(x_max)
        label.append(y_max)
        for _ in range(7):
            label.append(f"{0.00:.2f}")

        label_set.append(label)

You can add below before “# Adding Kitting Data, NOTE: Only class and 2d bbox coordinates are filled in”
xmin = float(xmin)
xmax = float(xmax)
ymin = float(ymin)
ymax = float(ymax)

So I took your advice and changed script to this.

Adding Kitting Data, NOTE: Only class and 2d bbox coordinates are filled in
x_min = float(x_min)
x_max = float(x_max)
y_min = float(y_min)
y_max = float(y_max)

        label.append(semantic_label)
        label.append(f"{0.00:.2f}")
        label.append(3)
        label.append(f"{0.00:.2f}")
        label.append("{:.2f}".format(x_min))
        label.append("{:.2f}".format(x_max))
        label.append("{:.2f}".format(y_min))
        label.append("{:.2f}".format(y_max))
        for _ in range(7):
            label.append(f"{0.00:.2f}")

and it produced this :

forklift 0.00 3 0.00 756.00 964.00 142.00 383.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
forklift 0.00 3 0.00 525.00 634.00 40.00 288.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
forklift 0.00 3 0.00 495.00 582.00 184.00 332.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Now its off to do some training.
Will keep you posted

So I have tried all the suggestions and still get an 000 AP for "forklift
I have some questions about “target_class_mapping”

When I set "target class_mapping to “key:forklift value:forklift” does it have to learn from scratch what a forklift is?

target_class_mapping {
key: “forklift”
value: “forklift”

When I set "target class_mapping to “key:forklift value:car” does is use what it knows as a car for transfer learning to learn what a forklift is?

target_class_mapping {
key: “forklift”
value: “car”

For target_class_mapping, please refer to the detailed in DetectNet_v2 — TAO Toolkit 3.21.11 documentation

I suggest you to double check the tfrecords.

The dataset_convert tool converts the class names in the KITTI-formatted data files to lowercase characters. Therefore, when configuring a training experiment, ensure that lowercase class names are used in the dataset_config section under target class mapping. Using incorrect class names in the dataset_config section can cause invalid training experiments with 0 mAP.

More, you can also run below experiments.
Only use forklift images/labels to generate tfrecords. Then try to train only one class-forklift.

A culprit is in the label file.
Please modify 3 to 0.

Morganh

It is working now.

Thanks for all your hard work tracking this down.

P.S. you may want to bring this issue up to the Omniverse Isaac Sim team

Thanks again

Thanks for the info.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.