Hi all
I am training peoplenet model with my custom dataset. while training i am getting same loss from last 3 epochs. So, how to implement early stop callback kind of thing so that if loss is not improving training will automatically stop.
For detectnet_v2 network, user can set “checkpoint interval”. It means the the interval (in epochs) at which tlt-train saves intermediate models.
End user can stop training by ctrl+c, and find the models in the output folder.
Thanks @Morganh for reply.
Hi @Morganh
I successfully completed training on my custom dataset and when i did evaluate i got the following average precision:
Matching predictions to ground truth, class 1/1.: 100%|█| 47067/47067 [00:02<00:
Validation cost: 0.001158
Mean average_precision (in %): 54.3145
class name average precision (in %)
------------ --------------------------
person 54.3145
Median Inference Time: 0.024809
2020-06-26 11:52:37,874 [INFO] iva.detectnet_v2.scripts.evaluate: Evaluation complete.
Time taken to run iva.detectnet_v2.scripts.evaluate:main: 0:05:20.913157.
I want to know what this 54.3145 average precison means? is this average precision with 0.5 iou or with something else.
See “minimum_detection_ground_truth_overlap” in your spec.
Minimum IOU between ground truth and predicted box after clustering to call a valid detection.
Thanks @Morganh.
But one more question is, I evaluate on hold out dataset which i did not involve for testing for that dataset i am getting 0 average precision and the resolution of images in hold out dataset are 19201080 and my training dataset images having 960960 resolution.
so i have to change the resolution of my hold out dataset before evaluation?
Yes.
But I think you can try tlt-infer firstly.