You may check the nvinfer configurations as DeepStream SDK FAQ - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums
If the dataset is PNG/JPEG files, the extra RGB to NV12 format conversion will introduce difficiation into the pipeline.
Pytorch use OpenCV and Pytorch functions as decoding and processing. It is not guranteed the DeepStream is bit aligned as Pytorch and OpenCV.