Can we convert Detectron2 to TensorRT engine?

[amoolya.shetty]

Hi there!

I was wondering if it is possible to convert detetectron2 to the TensortRT engine?

According to this 2020 discussion, what I learn is that a solution is still being worked on - About the GPU-Accelerated Libraries category.

Is there any new information on this?

Thank you,
Amoolya

[AastaLLL]

Hi,

There are several newer TensorRT releases from the 2020 discussion.

Do you have the ONNX format Detectron2 model?
If yes, it’s recommended to give it a try.

$ /usr/src/tensorrt/bin/trtexec --onnx=[model]

Thanks.

[amoolya.shetty]

@AastaLLL

$ /usr/src/tensorrt/bin/trtexec --onnx=[model]

Hi, sorry for the delayed response.

So, I first used detectron2.export ( Caffe2Tracer().export onnx() ) to convert the detectron2 model to ONNX. I then used the command you recommended to convert ONNX to TensorRT, but I ran into the following error.

[04/25/2022-11:29:58] [I] === Model Options ===
[04/25/2022-11:29:58] [I] Format: ONNX
[04/25/2022-11:29:58] [I] Model: deploy.onnx
[04/25/2022-11:29:58] [I] Output:
[04/25/2022-11:29:58] [I] === Build Options ===
[04/25/2022-11:29:58] [I] Max batch: explicit batch
[04/25/2022-11:29:58] [I] Workspace: 16 MiB
[04/25/2022-11:29:58] [I] minTiming: 1
[04/25/2022-11:29:58] [I] avgTiming: 8
[04/25/2022-11:29:58] [I] Precision: FP32
[04/25/2022-11:29:58] [I] Calibration:
[04/25/2022-11:29:58] [I] Refit: Disabled
[04/25/2022-11:29:58] [I] Sparsity: Disabled
[04/25/2022-11:29:58] [I] Safe mode: Disabled
[04/25/2022-11:29:58] [I] DirectIO mode: Disabled
[04/25/2022-11:29:58] [I] Restricted mode: Disabled
[04/25/2022-11:29:58] [I] Save engine:
[04/25/2022-11:29:58] [I] Load engine:
[04/25/2022-11:29:58] [I] Profiling verbosity: 0
[04/25/2022-11:29:58] [I] Tactic sources: Using default tactic sources
[04/25/2022-11:29:58] [I] timingCacheMode: local
[04/25/2022-11:29:58] [I] timingCacheFile:
[04/25/2022-11:29:58] [I] Input(s)s format: fp32:CHW
[04/25/2022-11:29:58] [I] Output(s)s format: fp32:CHW
[04/25/2022-11:29:58] [I] Input build shapes: model
[04/25/2022-11:29:58] [I] Input calibration shapes: model
[04/25/2022-11:29:58] [I] === System Options ===
[04/25/2022-11:29:58] [I] Device: 0
[04/25/2022-11:29:58] [I] DLACore:
[04/25/2022-11:29:58] [I] Plugins:
[04/25/2022-11:29:58] [I] === Inference Options ===
[04/25/2022-11:29:58] [I] Batch: Explicit
[04/25/2022-11:29:58] [I] Input inference shapes: model
[04/25/2022-11:29:58] [I] Iterations: 10
[04/25/2022-11:29:58] [I] Duration: 3s (+ 200ms warm up)
[04/25/2022-11:29:58] [I] Sleep time: 0ms
[04/25/2022-11:29:58] [I] Idle time: 0ms
[04/25/2022-11:29:58] [I] Streams: 1
[04/25/2022-11:29:58] [I] ExposeDMA: Disabled
[04/25/2022-11:29:58] [I] Data transfers: Enabled
[04/25/2022-11:29:58] [I] Spin-wait: Disabled
[04/25/2022-11:29:58] [I] Multithreading: Disabled
[04/25/2022-11:29:58] [I] CUDA Graph: Disabled
[04/25/2022-11:29:58] [I] Separate profiling: Disabled
[04/25/2022-11:29:58] [I] Time Deserialize: Disabled
[04/25/2022-11:29:58] [I] Time Refit: Disabled
[04/25/2022-11:29:58] [I] Skip inference: Disabled
[04/25/2022-11:29:58] [I] Inputs:
[04/25/2022-11:29:58] [I] === Reporting Options ===
[04/25/2022-11:29:58] [I] Verbose: Disabled
[04/25/2022-11:29:58] [I] Averages: 10 inferences
[04/25/2022-11:29:58] [I] Percentile: 99
[04/25/2022-11:29:58] [I] Dump refittable layers:Disabled
[04/25/2022-11:29:58] [I] Dump output: Disabled
[04/25/2022-11:29:58] [I] Profile: Disabled
[04/25/2022-11:29:58] [I] Export timing to JSON file:
[04/25/2022-11:29:58] [I] Export output to JSON file:
[04/25/2022-11:29:58] [I] Export profile to JSON file:
[04/25/2022-11:29:58] [I]
[04/25/2022-11:29:58] [I] === Device Information ===
[04/25/2022-11:29:58] [I] Selected Device: NVIDIA Tegra X1
[04/25/2022-11:29:58] [I] Compute Capability: 5.3
[04/25/2022-11:29:58] [I] SMs: 1
[04/25/2022-11:29:58] [I] Compute Clock Rate: 0.9216 GHz
[04/25/2022-11:29:58] [I] Device Global Memory: 3956 MiB
[04/25/2022-11:29:58] [I] Shared Memory per SM: 64 KiB
[04/25/2022-11:29:58] [I] Memory Bus Width: 64 bits (ECC disabled)
[04/25/2022-11:29:58] [I] Memory Clock Rate: 0.01275 GHz
[04/25/2022-11:29:58] [I]
[04/25/2022-11:29:58] [I] TensorRT version: 8.2.1
[04/25/2022-11:30:00] [I] [TRT] [MemUsageChange] Init CUDA: CPU +229, GPU +0, now: CPU 248, GPU 3423 (MiB)
[04/25/2022-11:30:01] [I] [TRT] [MemUsageSnapshot] Begin constructing builder kernel library: CPU 248 MiB, GPU 3456 MiB
[04/25/2022-11:30:01] [I] [TRT] [MemUsageSnapshot] End constructing builder kernel library: CPU 278 MiB, GPU 3486 MiB
[04/25/2022-11:30:01] [I] Start parsing network model
[04/25/2022-11:30:01] [I] [TRT] ----------------------------------------------------------------
[04/25/2022-11:30:01] [I] [TRT] Input filename: deploy.onnx
[04/25/2022-11:30:01] [I] [TRT] ONNX IR version: 0.0.6
[04/25/2022-11:30:01] [I] [TRT] Opset version: 9
[04/25/2022-11:30:01] [I] [TRT] Producer name: pytorch
[04/25/2022-11:30:01] [I] [TRT] Producer version: 1.8
[04/25/2022-11:30:01] [I] [TRT] Domain:
[04/25/2022-11:30:01] [I] [TRT] Model version: 0
[04/25/2022-11:30:01] [I] [TRT] Doc string:
[04/25/2022-11:30:01] [I] [TRT] ----------------------------------------------------------------
[04/25/2022-11:30:04] [I] [TRT] No importer registered for op: AliasWithName. Attempting to import as plugin.
[04/25/2022-11:30:04] [I] [TRT] Searching for plugin: AliasWithName, plugin_version: 1, plugin_namespace:
[04/25/2022-11:30:04] [E] [TRT] ModelImporter.cpp:773: While parsing node number 0 [AliasWithName → “315”]:
[04/25/2022-11:30:04] [E] [TRT] ModelImporter.cpp:774: — Begin node —
[04/25/2022-11:30:04] [E] [TRT] ModelImporter.cpp:775: input: “data”
output: “315”
op_type: “AliasWithName”
attribute {
name: “name”
s: “data”
type: STRING
}
attribute {
name: “is_backward”
i: 0
type: INT
}
domain: “org.pytorch._caffe2”

[04/25/2022-11:30:04] [E] [TRT] ModelImporter.cpp:776: — End node —
[04/25/2022-11:30:04] [E] [TRT] ModelImporter.cpp:779: ERROR: builtin_op_importers.cpp:4870 In function importFallbackPluginImporter:
[8] Assertion failed: creator && “Plugin not found, are the plugin name, version, and namespace correct?”
[04/25/2022-11:30:05] [E] Failed to parse onnx file
[04/25/2022-11:30:05] [I] Finish parsing network model
[04/25/2022-11:30:05] [E] Parsing model failed
[04/25/2022-11:30:05] [E] Failed to create engine from model.
[04/25/2022-11:30:05] [E] Engine set up failed

Any advice on how to solve this problem would be greatly appreciated.

Thanks!

[amoolya.shetty]

I see that someone else has had the same problem -

Can't convert from torch-onnx to tensorrt

Hi, Sorry for the late update. We are trying to deploy the Detectron2 model with TensorRT. For the following progress, please find this topic: Thanks.

Detectron2 Jetson NX

Hi, This may work but not guarantee. Convert your detectron2 model into ONNX format first. You can find some information below: detectron2.export — detectron2 0.6 documentation Try to modify TensorRT FasterRCNN (caffe-based) example. Although the file format is different, you should get the required plugin layer from the example: https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/sampleFasterRCNN Thanks.

And, you recommended modifying TensorRT FasterRCNN (caffe-based) example. (However, the GitHub link you provided is currently not working)

I also have a few other questions -

1. Are there any other ways to run the ONNX format on the TensorRT engine?
2. What other formats do the TensorRT engine support?

[amoolya.shetty]

Do we have any updates on this, please?

Thank you!

[amoolya.shetty]

@AastaLLL, Hey, do we have any updates on this?

[AastaLLL]

Hi,

Sorry for the late update.

Based on the error log, there is a ‘string’ data type used in the ONNX model.
Unfortunately, TensorRT doesn’t support the string data type yet.

You can find the detailed support matrix below:

Support Matrix :: NVIDIA Deep Learning TensorRT Documentation

Is the string data type nessary for your model?
Or you can replace it with a number or one-hot representation?

Thanks.