Description
I’m trying to build a TensorRT engine by parsing the network from an .onnx model file via TensorRT’s C++ API. The model is successfully turned into a TensorRT engine when trtexec.exe is used (I’ve used the engine to test inference time) so I don’t think it’s the .onnx file’s problem.
Creating network and parsing seem to be working, but the program dies when builder->buildSerializedNetwork(*network, *config) is called. Sometimes the “2: Assertion gemmShaders != nullptr failed.
2: [caskTacticEncoder.cpp::nvinfer1::builder::CaskBaseTacticEncoder::CaskBaseTacticEncoder::28] Error Code 2: Internal Error (Assertion gemmShaders != nullptr failed. )” error message is shown as an output and sometime it dies silently (The last line of output is “Graph optimization time: xxxxx seconds.”).
Environment
TensorRT Version: 10.0.1.6
GPU Type: RTX 4090
Nvidia Driver Version: 552.41
CUDA Version: 11.8
CUDNN Version: 11
Operating System + Version: Windows 11 Pro
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):
Relevant Files
Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)
I use the model specified in this github repo to turn it into an onnx file without precision set to fp32:
Steps To Reproduce
I parse the onnx file using parseFromFile function of nvonnxparser’s IParser, then with default builderconfig with profileStream call buildSerializedNetwork.
Things I include are:
include “NvInfer.h”
include “NvOnnxParser.h”
include <cuda_runtime.h>
I link nvinfer_10.dll, nvinfer_builder_resource_10.dll, nvinfer_plugin_10.dll, nvonnxparser_10.dll, and cudart for cuda_runtime.h.
I tried including cublas.lib since gemm related functions seemed to be part of that, but with no difference in the results.
When I use trtexec, the command I use is:
trtexec --precisionConstraints=obey --layerPrecisions=*:fp32 --onnx=alveolar_seg_model.onnx --saveEngine=directory