TLT different results inference

[nikolai0792]

Description

After exporting and converting the trained model to an engine in fp16 I’m getting bad results when I try to run inference. If I run inference .tlt then the results are perfect, but if I run inference .engine then the results are very bad.

Environment

I’m using this container: nvcr.io/nvidia/tlt-streamanalytics:v3.0-dp-py3.
FasterRCNN/efficientnet_b1.
Pretrained model: efficientnet_b1_relu.hdf5;

Command:
faster_rcnn export -m $USER_EXPERIMENT_DIR/data/faster_rcnn/frcnn_kitti_efficientnet_b1.epoch41.tlt
-o $USER_EXPERIMENT_DIR/data/faster_rcnn/frcnn_kitti_efficientnet_b1_fp16.etlt
-e $SPECS_DIR/default_spec_efficientnet_b1.txt
-k $KEY
–data_type fp16
–batch_size 4
–max_workspace_size 2000000000

CUDA_VISIBLE_DEVICES=0 tlt-converter -k $KEY
-d 3,512,512
-o NMS
-e $USER_EXPERIMENT_DIR/data/faster_rcnn/trt.fp16.engine
-m 4
-w 2000000000
-t fp16
-i nchw
$USER_EXPERIMENT_DIR/data/faster_rcnn/frcnn_kitti_efficientnet_b1_fp16.etlt

!faster_rcnn inference --gpu_index 0 -e $SPECS_DIR/default_spec_efficientnet_b1.txt!
default_spec_efficientnet_b1.txt (5.1 KB)

[Morganh]

To narrow down, can you update the tlt docker to v3.0-py3?

pip3 install --upgrade nvidia-tlt

[nikolai0792]

I did it. nothing changed

[Morganh]

I just try to train a FasterRCNN (backbone: efficientnet_b1) tlt model with public KITTI dataset.
Train only for 3 epochs. And export the tlt model to etlt model. Then generate the trt_fp16 engine.
But I cannot see much difference between inference with tlt model and inference with trt_fp16 engine.
Can you double check or try KITTI dataset?
More, you can also try to set a lower bbox_visualize_threshold when run inference with trt_fp16 engine.