What is your final purpose with checking latency by Nsight?
The gst-nvinfer plugin is open source. The source code is in /opt/nvidia/deepstream/deepstream/sources/libs/nvdsinfer and /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvinfer. There is also source code diagram for your reference. DeepStream SDK FAQ - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums
As to the source code, you may know the “queueInputBatch” includes the preprocess(convert the image data to Tensor data), TensorRT inferencing and postprocess(parsing the output Tensor data and calculating the output bboxes). As your inputs, you are using a batch size 1 TensorRT engine for inferencing, the performance is around 120FPS. The preprocess is accelerated by GPU and VIC. The postprocessing is customized by the third party and run on CPU. The total performance of preprocess+inferencing+postprocess is less than 120 FPS and we don’t know the exact data.