spec_train1.txt (9.6 KB) spec_train2.txt (7.7 KB) result_train2.txt (1.3 KB) result_train1.txt (1.7 KB) result_dataset_convert.txt (1.4 KB)
Hey,
I am training detectnet_v2 with tlt-launcher using trafficCamNet pretrained model, on a dataset which you can find info about in the result_dataset_convert.txt attached file.
I’ve run several training specifications. You can find two spec_train files attached as well as the respective results they yield.
What I want to highlight is that everything work fine when using the spec_train1.txt file, meaning when using the original classes of the dataset. But when using the spec_train2.txt file, meaning using the classes twowheels, lightvehicle, intermediatevehicule, heavyvehicule and person.
- with twowheels mapping motorcycle and bicycle.
- with heavyvehicule mapping heavyvehicule and bus.
I get an AP of 0 for the twowheels class.
I’ve checked and compared the two spec files and I didn’t find any error in the spec_train2.txt file. Can you confirm me that the concerned spec file is correct ?
And if that’s the case, how can I train a model while grouping the original classes of my dataset together ?
Can you share the full log of result_train2.txt ?
According to your resut_train2.txt,
Matching predictions to ground truth, class 1/5.: 100% 14405/14405 [00:01<00:00, 8952.04it/s]
Matching predictions to ground truth, class 2/5.: 100% 1216/1216 [00:00<00:00, 9456.14it/s]
Matching predictions to ground truth, class 3/5.: 100% 1364/1364 [00:00<00:00, 9515.43it/s]
Matching predictions to ground truth, class 5/5.: 100% 20742/20742 [00:02<00:00, 9064.67it/s]
Epoch 80/80
=========================
Validation cost: 0.000047
Mean average_precision (in %): 75.9764
class name average precision (in %)
heavyvehicle 98.6643
intermediatevehicle 95.0015
lightvehicle 91.0542
person 95.1621
twowheels 0
Seems that one class does not run evaluation.
Does the motorcycle or bicycle has a much smaller size?
Please try to modify below
evaluation_box_config {
key: “twowheels”
value {
minimum_height: 4
maximum_height: 9999
minimum_width: 4
maximum_width: 9999
}
}
to
evaluation_box_config {
key: “twowheels”
value {
minimum_height: 1
maximum_height: 9999
minimum_width: 1
maximum_width: 9999
}
}
Then just run tlt detectnet_v2 evaluate ... to check the AP result.
Thanks for you quick reply !
No I can’t share the full log, but I’ve run the train command again, this time with the --log_file option.
Well I only have the .etlt file remaining and can’t seem to evaluate it (not a valid file format apparently).
Yes I think those two objects tend to be smaller than others, but I don’t think the evaluation_box_config is the issue, because if it were, then the AP results for motorcycle and bicycle in result_train1.txt would have an AP of 0 as well.
What could cause that the evaluation isn’t running on the twowheels class ?
Plus I’ve ran some tests with Deepstream, and indeed no twowheels are detected, but the model from result_train1.txt can detect bicycles and motorcycles on the same test samples. It doesn’t look like an evaluation issue, but a training issue.
I’ll come back with the full log in a few hours.
Thanks. We need to check if the AP is always 0 during the training.
Yes. Even though I don’t have the full log, everytime I checked my training, the AP was 0. from the first evaluation to the last.
So, I am afraid this class should be filtered.
Please modify
target_class_config {
key: “twowheels”
value: {
clustering_config {
coverage_threshold: 0.005
dbscan_eps: 0.15
dbscan_min_samples: 0.05
minimum_bounding_box_height: 20
}
}
}
to a lower value
target_class_config {
key: “twowheels”
value: {
clustering_config {
coverage_threshold: 0.005
dbscan_eps: 0.15
dbscan_min_samples: 0.05
minimum_bounding_box_height: 1
}
}
}
And also modify above-mentioned
minimum_height
Then just run tlt detectnet_v2 evaluate ... against one tlt model to check the AP result.
here are the results of evaluation with the newly modified config
Validation cost: 0.044589
Mean average_precision (in %): 0.0528
class name average precision (in %)
------------------- --------------------------
heavyvehicle 0.0173546
intermediatevehicle 0.0518693
lightvehicle 0.165507
person 0.0295068
twowheels 0
Median Inference Time: 0.006014
2021-03-25 11:09:36,714 [INFO] __main__: Evaluation complete.
I took a model from the epoch 5 which explains the low scores, but twowheels AP remains unchanged.
I don’t think the class is filtered out due to size constraint, if those parameters had an impact on twowheels (motorcycle, bicycle), then result_train1.txt would have an AP of 0 for the classes motorcycle and bicycle.
Indeed there might be some parameters filtering out my class, but it shouldn’t be related to the size of bounding boxes. Because the spec files are the same for both train, except for the name of the classes and the mapping
My training is still on-going, I’ll update you with the full log as soon as possible.
log_file.txt (1.9 MB)
Here is the full log file generated using the --log_file option.
As I thought the AP of twowheels stay at 0 during the whole training, even though I modified the spec as you mentioned and re-run a full training.
What ticks me is that the score of heavyvehicle is really great even though I mapped the bus class into it, which means that the mapping seems to work correctly for this class.
I think below is the root cause.
“twowheels” is a dummy class name. It does not exist in your label file.
So, please modify the class mapping to below
target_class_mapping {
key: “motorcycle”
value: “motorcycle”
}
target_class_mapping {
key: “bicycle”
value: “motorcycle”
}
Do not forget to modify other twowheels setting in your spec file too.
But I followed this example of the doc on detectnet_v2 :
dataset_config {
data_sources: {
tfrecords_path: "<path to the training tfrecords root/tfrecords train pattern>"
image_directory_path: "<path to the training data source>"
}
image_extension: "jpg"
target_class_mapping {
key: "car"
value: "car"
}
target_class_mapping {
key: "automobile"
value: "car"
}
target_class_mapping {
key: "heavy_truck"
value: "car"
}
target_class_mapping {
key: "person"
value: "pedestrian"
}
target_class_mapping {
key: "rider"
value: "cyclist"
}
validation_fold: 0
}
This example seems to use dummy class names as well, has it not been tested ?
Okay ! I’ll try that and get back to you !
Below is the specs in released jupyter notebooks. It has similar setting. It trained against the KITTI dataset. It has been tested.
target_class_mapping {
key: “car”
value: “car”
}
target_class_mapping {
key: “cyclist”
value: “cyclist”
}
target_class_mapping {
key: “pedestrian”
value: “pedestrian”
}
target_class_mapping {
key: “person_sitting”
value: “pedestrian”
}
target_class_mapping {
key: “van”
value: “car”
}
Great it works !
Oh okay so when we do the class mapping we have to use classes already existing in the dataset ! Good to know !
Thanks again for your quick reply ! Have a nice day !