Classification accuracy mismatch between running same resnet18 engine on tensorrt and deepstream

Please provide complete information as applicable to your setup.

**• Hardware Platform (Jetson Nano 4GB/ GPU)
• DeepStream Version 6.0
• JetPack Version (valid for Jetson only) 4.6.4-b39
** Tensorrt Version

I am facing problem with classification accuracy mismatch in deepstream. I have two pipelines one with tensorrt where i run inference directly using tensorrt and other one where i am using deep-stream. I am using yolov5n for face detection and then i use resnet18 classifier to classify detected faces into gender (male, female). The detection engine works similarly in both pipelines but difference is in classification accuracy for resnet18. In deepstream yolov5n is used as primary detector and resnet18 is used as secondary inference engine in sgie. I have changed parameters in config file for sgie as needed for inference but nothing changes. I have trained resnet model on custom dataset using pytorch and i am building its engine using same code for both pipelines.
I was thinking may be the problem is with pre-processing before feeding the detected faces to sgie for inference, I have set parameters of normalization and scaling as required but still problem persists. I have also removed normalization from training and used that model for inference but that also does not work. I am using same video to compare accuracy of both pipelines. Accuracy for deepstream pipeline is much worse than in tensorrt pipeline. Need help to resolve this problem.