Detectnet_V2 inference issue, no bounding boxes drawn on annotated_images

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Hello,

I have trained a detectnet_v2 model, however when I run my inference command I see no bounding boxes being drawn on my inference data set. As a sanity check, I ran the inference over the training image set and still see no drawn bounding boxes in the images. This model is still currently training, but I used some of the output file steps. I’ve tried step 0 (no bounding box output), I’ve tried step 52680 (which actually threw a decryption key error and wouldn’t inference anything), then I tried step 105360 which also has no output (like step 0).

Command:
tlt-infer detectnet_v2 -m /workspace/tlt-experiments/testing/model.step-105360.tlt -i /workspace/tlt-experiments/KITTIRoot/images -o /workspace/tlt-experiments/testing -bs 16 -cp /workspace/tlt-experiments/testing/test.txt -lw 3 -k <KEY_OMITTED_FOR_POST> --kitti_dump

All of the label files are empty and no bounding boxes are drawn on any of the output from the training image set data. What could be the issue?

Sincerely,
kwindham
inferencepost.txt (1.31 KB)

2

I realized that my inference cut off values were too high. After some testing, reducing the confidence_th to 0.1 for a sanity check resolved my issues.

I do have one more question though, as the model is training and I check the models output via the steps, what is a good way to figure out the mAP of the individual classes at that current step in the model creation process? I’ve only known the output of the mAP for the model to be displayed once the training has finished its last epoch, is there any way to view it as you’re training? Say, in a log file somewhere that gets updated with each step that is output?

Sincerely,
kwindham

3

Hi kwindham,
Actually you can find the mAP result when the training is ongoing instead of the last epoch.
See below lines of the training spec.

evaluation_config {
  validation_period_during_training: 10
  first_validation_epoch: 1

It will calculate mAP in the first epoch, then show mAP again in 11st epoch, 21st epoch, etc. And in the last epoch, mAP is also shown.