Same ETLT file showing different results on other system (with same configuration)

I trained model on 120 epochs for single class using transfer learning toolkit using (detect net_v2 + resnet_18).

System-1:
GPU : Quadro P2200

System-2:
GPU : Quadro P2200

I am using deep-stream-test3-app for detection.

After training, I got weights file (TLT), and I then converted that TLT file into etlt file. Now, etlt/tlt file is platform independent, so I moved etlt file to System-2. I also tested same file on System-1, but when i converted that etlt file (exists on other system) to trt file on System-2, and compare detection, on both systems, I found that detection is very different?

I am bit confused, why this happens?

Can you check if the TRT version are the same in these two systems?

Yes, the TRT version on both systems is exactly matching. (TRT version 7.2.1)

It really does not make sense. Could you please share how did you generate the trt engine in these two systems?

After training and pruning of detect-net v2, I have tll file, I converted tlt file to etlt file.

Finally I download TRT Converter from link

I then converted etlt to trt file using below command.
tlt-converter /workspace/tlt-experiments/detectnet_v2/experiment_dir_final/resnet18_detector.etlt -k NXY2MDJjaWtjOW5kZGx1bms3MWJjb2k4NXQ6YTY0NGRmNWEtMGYyNS00MTU2LTk4NWMtN2Y2MGQ1ZjU0NDBm-c /workspace/tlt-experiments/detectnet_v2/experiment_dir_final/calibration.bin -o output_cov/Sigmoid,output_bbox/BiasAdd -d 3,480,640 -i nchw -m 8 -t int8 -e /workspace/tlt-experiments/detectnet_v2/experiment_dir_final/resnet18_detector.trt -b 4

To narrow down, could you please generate fp32 trt engine in both systems to check if they have similar performance on detection?

I have jetson nano, that does not support fp32 mode, is there any-other way to handle this?

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

Why did you use jetson nano?
As you mentioned above, you were using below two systems , right?

System-1:
GPU : Quadro P2200

System-2:
GPU : Quadro P2200

BTW, Nano can support fp32 mode.

1 Like