I am currently working on deploying some neural networks (DETR variants) to the edge using Jetson Nanos for my company and am having trouble translating these networks from PyTorch to TensorRT. I am able to run them just fine as native PyTorch models on the Nano, but they are slow. I have tried to convert the model into TensorRT but am getting some cryptic errors for which I cannot find any solutions online. See below for the error text.
[TensorRT] ERROR: ../builder/myelin/codeGenerator.cpp (396) - Myelin Error in compileGraph: 64 (myelinProgramAnalysisError : Mismatched shape for tensor (Unnamed Layer_ 1837) [Identity]_output', current shape:b,[1,1,1,256], vs. expected shape:b,[].) [TensorRT] ERROR: ../builder/myelin/codeGenerator.cpp (396) - Myelin Error in compileGraph: 64 (myelinProgramAnalysisError : Mismatched shape for tensor (Unnamed Layer_ 1837) [Identity]_output', current shape:b,[1,1,1,256], vs. expected shape:b,[].)
This is caused when calling builder.build_cuda_engine(network) in the TensorRT Python API. The network is parsed successfully from an ONNX file, and then passed to the builder, which raises this error. When I plot the network graph from the ONNX file, I don’t see any undefined expected shapes, which seems to be the source of the problem. The specified node number is just an addition node, so I’m not even sure that node number is correct.
I have found a repo on GitHub that claims to have successfully compiled DETR to TensorRT, and even shows a screenshot of this error in its discussion, but much of the documentation is in Chinese and I’m having trouble figuring out what they did to get around this error using Google Translate.