Unable to detect object after training

Hi pritam,
I do not understand why you said you can get good result but runs into error?
If tlt-infer runs into error, how can you know tlt-infer can get good result?

Actually I am getting good result using tlt-infer for some images but then get error like

More, I mentioned previously you can change model for quickly test.

Change “-m” to the unpruned tlt model. Because you train for the first time, the validation during training can get 82% mAP. So you can use that tlt model directly to do tlt-infer.

-m experiment_dir_unpruned/weights/resnet18_detector.tlt

Result of

# Running inference for detection on n images
!tlt-infer detectnet_v2 -i $USER_EXPERIMENT_DIR/data/training/image_2 \
                        -o $USER_EXPERIMENT_DIR/tlt_infer_testing \
                        -m $USER_EXPERIMENT_DIR/experiment_dir_retrain/weights/resnet18_detector_pruned.tlt \
                        -cp $SPECS_DIR/detectnet_v2_clusterfile_kitti.json \
                        -k $KEY \
                        --kitti_dump \
                        -lw 3 \
                        -g 0 \
                        -bs 16

is

_________________________________________________________________
2020-02-04 06:23:00,339 [INFO] iva.detectnet_v2.scripts.inference: Initialized model
2020-02-04 06:23:00,341 [INFO] iva.detectnet_v2.scripts.inference: Commencing inference
0it [00:00, ?it/s]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 47.94it/s]
 62%|######2   | 10/16 [00:00<00:00, 47.95it/s]
100%|##########| 16/16 [00:00<00:00, 47.63it/s]
1it [00:03,  3.27s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 48.80it/s]
 62%|######2   | 10/16 [00:00<00:00, 48.55it/s]
100%|##########| 16/16 [00:00<00:00, 48.35it/s]
2it [00:04,  2.65s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 48.10it/s]
 62%|######2   | 10/16 [00:00<00:00, 47.66it/s]
100%|##########| 16/16 [00:00<00:00, 47.12it/s]
3it [00:05,  2.21s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 45.41it/s]
 62%|######2   | 10/16 [00:00<00:00, 46.02it/s]
100%|##########| 16/16 [00:00<00:00, 47.01it/s]
4it [00:06,  1.92s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 47.12it/s]
 62%|######2   | 10/16 [00:00<00:00, 47.04it/s]
100%|##########| 16/16 [00:00<00:00, 46.86it/s]
5it [00:08,  1.73s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 43.39it/s]
 62%|######2   | 10/16 [00:00<00:00, 43.32it/s]
100%|##########| 16/16 [00:00<00:00, 44.30it/s]
6it [00:09,  1.59s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 47.93it/s]
 62%|######2   | 10/16 [00:00<00:00, 48.20it/s]
100%|##########| 16/16 [00:00<00:00, 47.93it/s]
7it [00:10,  1.50s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 45.96it/s]
 62%|######2   | 10/16 [00:00<00:00, 45.43it/s]
100%|##########| 16/16 [00:00<00:00, 44.92it/s]
8it [00:12,  1.44s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 43.65it/s]
 62%|######2   | 10/16 [00:00<00:00, 43.05it/s]
100%|##########| 16/16 [00:00<00:00, 42.62it/s]
9it [00:13,  1.41s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 45.69it/s]
 62%|######2   | 10/16 [00:00<00:00, 45.89it/s]
100%|##########| 16/16 [00:00<00:00, 45.76it/s]
10it [00:14,  1.39s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 46.78it/s]
 62%|######2   | 10/16 [00:00<00:00, 46.59it/s]
100%|##########| 16/16 [00:00<00:00, 46.47it/s]
11it [00:16,  1.37s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 48.42it/s]
 62%|######2   | 10/16 [00:00<00:00, 48.45it/s]
100%|##########| 16/16 [00:00<00:00, 47.90it/s]
12it [00:17,  1.34s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 47.56it/s]
 62%|######2   | 10/16 [00:00<00:00, 47.28it/s]
100%|##########| 16/16 [00:00<00:00, 47.31it/s]
13it [00:18,  1.33s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 47.07it/s]
 62%|######2   | 10/16 [00:00<00:00, 47.04it/s]
100%|##########| 16/16 [00:00<00:00, 47.39it/s]
14it [00:19,  1.30s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 45.64it/s]
 62%|######2   | 10/16 [00:00<00:00, 45.51it/s]
100%|##########| 16/16 [00:00<00:00, 45.42it/s]
15it [00:21,  1.31s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 48.60it/s]
 62%|######2   | 10/16 [00:00<00:00, 48.23it/s]
100%|##########| 16/16 [00:00<00:00, 47.62it/s]
16it [00:22,  1.34s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 47.05it/s]
 62%|######2   | 10/16 [00:00<00:00, 46.88it/s]
100%|##########| 16/16 [00:00<00:00, 46.71it/s]
17it [00:23,  1.35s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 47.54it/s]
 62%|######2   | 10/16 [00:00<00:00, 47.02it/s]
100%|##########| 16/16 [00:00<00:00, 46.72it/s]
18it [00:25,  1.32s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 43.74it/s]
 62%|######2   | 10/16 [00:00<00:00, 45.29it/s]
100%|##########| 16/16 [00:00<00:00, 46.51it/s]
19it [00:26,  1.30s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 46.88it/s]
 62%|######2   | 10/16 [00:00<00:00, 46.72it/s]
100%|##########| 16/16 [00:00<00:00, 47.40it/s]
20it [00:27,  1.28s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 46.74it/s]
 62%|######2   | 10/16 [00:00<00:00, 46.80it/s]
100%|##########| 16/16 [00:00<00:00, 46.43it/s]
21it [00:28,  1.27s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 46.32it/s]
 62%|######2   | 10/16 [00:00<00:00, 47.36it/s]
100%|##########| 16/16 [00:00<00:00, 47.66it/s]
22it [00:30,  1.27s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 48.89it/s]
 62%|######2   | 10/16 [00:00<00:00, 48.08it/s]
100%|##########| 16/16 [00:00<00:00, 46.89it/s]
23it [00:31,  1.27s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 47.09it/s]
 62%|######2   | 10/16 [00:00<00:00, 47.51it/s]
100%|##########| 16/16 [00:00<00:00, 47.40it/s]
24it [00:32,  1.31s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 49.55it/s]
 62%|######2   | 10/16 [00:00<00:00, 49.26it/s]
100%|##########| 16/16 [00:00<00:00, 49.07it/s]
25it [00:34,  1.30s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 49.33it/s]
 62%|######2   | 10/16 [00:00<00:00, 48.81it/s]
100%|##########| 16/16 [00:00<00:00, 48.44it/s]
26it [00:35,  1.29s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 47.35it/s]
 62%|######2   | 10/16 [00:00<00:00, 47.55it/s]
100%|##########| 16/16 [00:00<00:00, 47.80it/s]
27it [00:36,  1.27s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 48.30it/s]
 62%|######2   | 10/16 [00:00<00:00, 48.17it/s]
100%|##########| 16/16 [00:00<00:00, 48.12it/s]
28it [00:37,  1.27s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 47.30it/s]
 62%|######2   | 10/16 [00:00<00:00, 46.73it/s]
100%|##########| 16/16 [00:00<00:00, 46.53it/s]
29it [00:39,  1.27s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 47.69it/s]
 62%|######2   | 10/16 [00:00<00:00, 47.33it/s]
100%|##########| 16/16 [00:00<00:00, 47.31it/s]
30it [00:40,  1.27s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 46.66it/s]
 62%|######2   | 10/16 [00:00<00:00, 46.92it/s]
100%|##########| 16/16 [00:00<00:00, 47.51it/s]
31it [00:41,  1.27s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 49.42it/s]
 62%|######2   | 10/16 [00:00<00:00, 48.42it/s]
100%|##########| 16/16 [00:00<00:00, 47.83it/s]
32it [00:42,  1.26s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 47.70it/s]
 62%|######2   | 10/16 [00:00<00:00, 47.62it/s]
100%|##########| 16/16 [00:00<00:00, 47.89it/s]
33it [00:44,  1.25s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 48.63it/s]
 62%|######2   | 10/16 [00:00<00:00, 48.01it/s]
100%|##########| 16/16 [00:00<00:00, 47.36it/s]
34it [00:45,  1.25s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 46.47it/s]
 62%|######2   | 10/16 [00:00<00:00, 46.93it/s]
100%|##########| 16/16 [00:00<00:00, 47.44it/s]
35it [00:46,  1.25s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 47.32it/s]
 62%|######2   | 10/16 [00:00<00:00, 47.12it/s]
100%|##########| 16/16 [00:00<00:00, 48.57it/s]
36it [00:47,  1.24s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 55.01it/s]
100%|##########| 16/16 [00:00<00:00, 54.75it/s]
37it [00:49,  1.22s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 50.40it/s]
100%|##########| 16/16 [00:00<00:00, 48.28it/s]
38it [00:50,  1.25s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 51.44it/s]
 69%|######8   | 11/16 [00:00<00:00, 50.91it/s]
100%|##########| 16/16 [00:00<00:00, 50.14it/s]
39it [00:51,  1.27s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 53.75it/s]
100%|##########| 16/16 [00:00<00:00, 53.64it/s]
40it [00:52,  1.27s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 49.78it/s]
 69%|######8   | 11/16 [00:00<00:00, 49.39it/s]
100%|##########| 16/16 [00:00<00:00, 48.55it/s]
41it [00:54,  1.29s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 49.13it/s]
 62%|######2   | 10/16 [00:00<00:00, 48.29it/s]
100%|##########| 16/16 [00:00<00:00, 47.82it/s]
42it [00:55,  1.32s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 53.32it/s]
100%|##########| 16/16 [00:00<00:00, 53.12it/s]
43it [00:57,  1.34s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 55.78it/s]
100%|##########| 16/16 [00:00<00:00, 55.21it/s]
44it [00:58,  1.29s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 56.34it/s]
100%|##########| 16/16 [00:00<00:00, 55.44it/s]
45it [00:59,  1.25s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 55.54it/s]
100%|##########| 16/16 [00:00<00:00, 54.55it/s]
46it [01:00,  1.22s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 55.56it/s]
100%|##########| 16/16 [00:00<00:00, 53.32it/s]
47it [01:01,  1.22s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 52.32it/s]
100%|##########| 16/16 [00:00<00:00, 52.97it/s]
48it [01:03,  1.28s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 48.52it/s]
 62%|######2   | 10/16 [00:00<00:00, 46.68it/s]
100%|##########| 16/16 [00:00<00:00, 45.47it/s]
49it [01:04,  1.27s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 53.06it/s]
100%|##########| 16/16 [00:00<00:00, 46.16it/s]
50it [01:05,  1.29s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 54.61it/s]
100%|##########| 16/16 [00:00<00:00, 54.84it/s]
51it [01:07,  1.30s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 54.80it/s]
100%|##########| 16/16 [00:00<00:00, 54.50it/s]
52it [01:08,  1.29s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 55.22it/s]
100%|##########| 16/16 [00:00<00:00, 52.92it/s]
53it [01:09,  1.29s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 54.40it/s]
100%|##########| 16/16 [00:00<00:00, 53.76it/s]
54it [01:10,  1.28s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 53.76it/s]
100%|##########| 16/16 [00:00<00:00, 53.45it/s]
55it [01:12,  1.28s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 54.25it/s]
100%|##########| 16/16 [00:00<00:00, 52.88it/s]
56it [01:13,  1.26s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 55.32it/s]
100%|##########| 16/16 [00:00<00:00, 54.50it/s]
57it [01:14,  1.26s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 54.29it/s]
100%|##########| 16/16 [00:00<00:00, 53.89it/s]
58it [01:15,  1.24s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 55.98it/s]
100%|##########| 16/16 [00:00<00:00, 54.12it/s]
59it [01:17,  1.23s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 54.86it/s]
100%|##########| 16/16 [00:00<00:00, 53.86it/s]
60it [01:18,  1.23s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 54.87it/s]
100%|##########| 16/16 [00:00<00:00, 54.67it/s]
61it [01:19,  1.22s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 55.27it/s]
100%|##########| 16/16 [00:00<00:00, 55.37it/s]
62it [01:20,  1.20s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 53.81it/s]
100%|##########| 16/16 [00:00<00:00, 53.82it/s]
63it [01:21,  1.19s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 55.26it/s]
100%|##########| 16/16 [00:00<00:00, 54.68it/s]
64it [01:22,  1.19s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 56.29it/s]
100%|##########| 16/16 [00:00<00:00, 55.06it/s]
65it [01:24,  1.18s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 56.19it/s]
100%|##########| 16/16 [00:00<00:00, 55.13it/s]
66it [01:25,  1.18s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 55.26it/s]
100%|##########| 16/16 [00:00<00:00, 54.87it/s]
67it [01:26,  1.18s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 54.24it/s]
100%|##########| 16/16 [00:00<00:00, 54.07it/s]
68it [01:27,  1.17s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 46.64it/s]
 62%|######2   | 10/16 [00:00<00:00, 47.03it/s]
100%|##########| 16/16 [00:00<00:00, 47.43it/s]
69it [01:28,  1.20s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 52.65it/s]
100%|##########| 16/16 [00:00<00:00, 50.45it/s]
70it [01:30,  1.22s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 49.89it/s]
 62%|######2   | 10/16 [00:00<00:00, 49.63it/s]
100%|##########| 16/16 [00:00<00:00, 48.73it/s]
71it [01:31,  1.24s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 47.56it/s]
 62%|######2   | 10/16 [00:00<00:00, 46.86it/s]
100%|##########| 16/16 [00:00<00:00, 47.39it/s]
72it [01:32,  1.25s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 49.19it/s]
 62%|######2   | 10/16 [00:00<00:00, 49.18it/s]
100%|##########| 16/16 [00:00<00:00, 49.70it/s]
73it [01:34,  1.25s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 49.21it/s]
100%|##########| 16/16 [00:00<00:00, 51.72it/s]
74it [01:35,  1.25s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 51.76it/s]
 69%|######8   | 11/16 [00:00<00:00, 50.45it/s]
100%|##########| 16/16 [00:00<00:00, 48.20it/s]
75it [01:36,  1.25s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 38%|###7      | 6/16 [00:00<00:00, 50.80it/s]
 69%|######8   | 11/16 [00:00<00:00, 49.56it/s]
100%|##########| 16/16 [00:00<00:00, 49.24it/s]
76it [01:37,  1.25s/it]
  0%|          | 0/16 [00:00<?, ?it/s]
 31%|###1      | 5/16 [00:00<00:00, 47.05it/s]
 69%|######8   | 11/16 [00:00<00:00, 47.92it/s]
100%|##########| 16/16 [00:00<00:00, 48.10it/s]
77it [01:39,  1.26s/it]Traceback (most recent call last):
  File "/usr/local/bin/tlt-infer", line 8, in <module>
    sys.exit(main())
  File "./common/magnet_infer.py", line 35, in main
  File "./detectnet_v2/scripts/inference.py", line 222, in main
  File "./detectnet_v2/scripts/inference.py", line 180, in inference_wrapper_batch
  File "./detectnet_v2/inferencer/tlt_inferencer.py", line 123, in infer_batch
  File "./detectnet_v2/inferencer/base_inferencer.py", line 107, in input_preprocessing
ValueError: axes don't match array
77it [01:39,  1.29s/it]

more then 50% of images were saved into tlt_infer_testing but not completely and result was good on that 50% images.
But no problem because weights are working on Deep stream and giving good results.

Hi pritam,
Could you do some triage to the failed images?
You can select several images which have no bboxes, run tlt-infer against them.

If you meet the issue you mentioned, plesae check the difference between the failed images and workable images.
Especially check if some are BGR but others are not.

Thanks morganh I will try it.