I suggest you to check if the preprocessing is the same between tao-deploy and deepstream.
In tao-deploy, you can add debug code in tao_deploy/nvidia_tao_deploy/cv/classification_tf2/scripts/inference.py at 31c7e0ed3fe48942c254b3b85517e7418eea17b3 · NVIDIA/tao_deploy · GitHub and tao_deploy/nvidia_tao_deploy/cv/classification_tf1/dataloader.py at 31c7e0ed3fe48942c254b3b85517e7418eea17b3 · NVIDIA/tao_deploy · GitHub to save images after preprocessing. Similar to Fine-tuned TAO ClassificationTF2 Accuracy Drop after Compiling to TensorRT - #35 by Morganh. This will help you understand the preprocessing based on your training spec file.
Also, you can leverage the tao-deploy code to generate a standalone inference code to run inference against the tensorrt engine.
Check if your standalone inference can get the expected result.