I have created TAO model yolov4.
It is onnx format but i am not able to run it using onnxruntime-gpu so i found that it contains custome layers, i reffered following links:-
BatchedNMS_TRT: could not find any supported formats consistent with input/output data types · Issue #954 · NVIDIA/TensorRT · GitHub
so i wrote the following code snippet
import onnx_graphsurgeon as gs
import numpy as np
import onnx
graph = gs.import_onnx(onnx.load(“yolov4.onnx”))
@gs.Graph.register()
def trt_batched_nms(self, boxes_input, scores_input, nms_output,
shareLocation, numClasses): # and other attrs
boxes_input.outputs.clear()
scores_input.outputs.clear()
nms_output.inputs.clear()
attrs = {
“shareLocation”: shareLocation,
“numClasses” : numClasses,
“backgroundLabelId”: -1,
“topK”:400,
“keepTopK”:200,
“scoreBits”:0,
“scoreThreshold”:0.001,
“iouThreshold”: 0.5,
“isNormalized”:True,
“clipBoxes”:True
}
return self.layer(op=“BatchedNMSDynamic_TRT”, attrs=attrs,
inputs=[boxes_input, scores_input],
outputs=[nms_output])
tmap = graph.tensors()
graph.trt_batched_nms(tmap[“box”], tmap[“cls”],
tmap[“BatchedNMS”], shareLocation = True,
numClasses = 2)
graph.trt_batched_nms(tmap[“box”], tmap[“cls”],
tmap[“BatchedNMS_1”], shareLocation = True,
numClasses = 2)
graph.trt_batched_nms(tmap[“box”], tmap[“cls”],
tmap[“BatchedNMS_2”], shareLocation = True,
numClasses = 2)
graph.trt_batched_nms(tmap[“box”], tmap[“cls”],
tmap[“BatchedNMS_3”], shareLocation = True,
numClasses = 2)
graph.cleanup().toposort()
graph.fold_constants().cleanup()
onnx.checker.check_model(gs.export_onnx(graph))
but i get following error
[W] Inference failed. You may want to try enabling partitioning to see better results. Note: Error was:
[ONNXRuntimeError] : 10 : INVALID_GRAPH : This is an invalid model. In Node, (“onnx_graphsurgeon_node_0”, BatchedNMSDynamic_TRT, “”, -1) : () → (“BatchedNMS”: tensor(int32),) , Error No Op registered for BatchedNMSDynamic_TRT with domain_version of 12
ValidationError Traceback (most recent call last)
Cell In[14], line 49
47 graph.fold_constants().cleanup()
48 # print(tmap)
—> 49 onnx.checker.check_model(gs.export_onnx(graph))
File ~\AppData\Local\anaconda3\envs\gpu_env_demo\lib\site-packages\onnx\checker.py:179, in check_model(model, full_check, skip_opset_compatibility_check, check_custom_domain)
175 if sys.getsizeof(protobuf_string) > MAXIMUM_PROTOBUF:
176 raise ValueError(
177 “This protobuf of onnx model is too large (>2GB). Call check_model with model path instead.”
178 )
→ 179 C.check_model(
180 protobuf_string,
181 full_check,
182 skip_opset_compatibility_check,
183 check_custom_domain,
184 )
ValidationError: Field ‘shape’ of ‘type’ is required but missing.
the shapes are dynamic so none
out_tensors = graph.outputs
print(out_tensors)
[Variable (BatchedNMS): (shape=None, dtype=int32), Variable (BatchedNMS_1): (shape=None, dtype=float32), Variable (BatchedNMS_2): (shape=None, dtype=float32), Variable (BatchedNMS_3): (shape=None, dtype=float32)]
I feel like i am missing something can anyone please help me out.
i have alredy created engine for triton on jetson using trtexec.
Now i need it to run with gpu onnx runtime.