Actually I cannot reproduce the issue you mention.
I trained the fire dataset previously. See topic Problems encountered in training unet and inference unet - #27 by Morganh
I train the model of 960x544.
And today I use it to run evaluation against tlt model or trt engine. There is not accuracy drop.
$ cat evaluation_result_trt/results_trt.json
“{‘fire’: {‘precision’: 0.9988102, ‘Recall’: 0.99939096, ‘F1 Score’: 0.999100481505847, ‘iou’: 0.9982026}, ‘background’: {‘precision’: 0.88543546, ‘Recall’: 0.79814464, ‘F1 Score’: 0.8395270786586672, ‘iou’: 0.72343534}}”
$ cat evaluation_result_json/results_tlt.json
“{‘fire’: {‘precision’: 0.9988102, ‘Recall’: 0.9993909, ‘F1 Score’: 0.9991004219185265, ‘iou’: 0.9982025}, ‘background’: {‘precision’: 0.885423, ‘Recall’: 0.7981348, ‘F1 Score’: 0.839516098048122, ‘iou’: 0.72341895}}”
Are your test images 1280x704? Is it .png file?