After upgrading tensorrt to 10.8, the model accuracy decreased.
After setting all nodes of the model to output, the model accuracy was aligned. It was suspected that the fusion strategy introduced by the upgrade caused the accuracy problem.
Finally, we used the polygraphy tool to find an onnx subgraph that could reproduce the problem.
By the way, it needs to be reminded that the model only uses fp32 precision, not even fp16 or int8
I post onnx file here: issues/trt_inconsistent/mini_graph.onnx at main · 2730gf/issues · GitHub
command: polygraphy run mini_graph.onnx -v -v -v -v -v --pool-limit workspace:20G --onnxrt --trt --validate --atol 1e-4 --rtol 1e-3 --onnx-outputs p2o.Concat.125 --trt-outputs p2o.Concat.125
