From your graph, it seems that the the model was processed by the nvinfer before 3:05, which led to a rapid increase in memory usage. After that, the usage remained relatively stable.
Could you please give it a try as follows?
- Remove the nvinfer from the pipeline and monitor the memory status
- Please refer to our FAQ 19 and do a preliminary analysis using Valgrand.