Trt-engine-explorer fails to parse trtexec output

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

I am trying to use the TRT Engine Explorer to get insights about my model.

The neccessary JSON file were generated using utils/process_engine.py:
<MODEL_NAME>.(graph.json, metadata.json, profile.json, timing.json)

However, trying to instantiate the EnginePlan object, I am getting the following error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [2], in <cell line: 2>()
      1 assert engine_name is not None
----> 2 plan = EnginePlan(f'{engine_name}.graph.json', f'{engine_name}.profile.json', f'{engine_name}.metadata.json')

File ~/devel/TensorRT/tools/experimental/trt-engine-explorer/trex/engine_plan.py:103, in EnginePlan.__init__(self, graph_file, profiling_file, profiling_metadata_file, build_metadata_file, name)
     99     self.total_runtime = sum(
    100         [avg_time for avg_time in self._df["latency.avg_time"]])
    102 self.name = name or path_leaf(graph_file)
--> 103 raw_layers, self.bindings = import_graph_file(graph_file)
    104 raw_layers = create_layers(self, raw_layers)
    106 self._df = None

File ~/devel/TensorRT/tools/experimental/trt-engine-explorer/trex/parser.py:134, in import_graph_file(graph_file)
    131             pass
    132     return raw_layers
--> 134 raw_layers, bindings = read_graph_file(graph_file)
    135 raw_layers = convert_deconv(raw_layers)
    136 raw_layers = disambiguate_layer_names(raw_layers)

File ~/devel/TensorRT/tools/experimental/trt-engine-explorer/trex/parser.py:52, in read_graph_file(graph_file)
     50     details_msg = "\nMake sure to enable detailed ProfilingVerbosity."
     51     details_msg += "\nSee https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#engine-inspector"
---> 52     raise ValueError(err_msg + details_msg)
     53 return layers, bindings

ValueError: File engine.trt.graph.json does not conform to the expected JSON format.
Make sure to enable detailed ProfilingVerbosity.
See https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#engine-inspector

Quick look at the parser code tells me that parser expect the list of layers in graph.json to be a list of dictionaries. The file generated by trtexec contains only plain layer names though:

{"Layers": ["Range_1363"
,"Range_1367"
,"Squeeze_454"
,"Conv_455 + Relu_456"
,"MaxPool_457"

...

Profiling files were generated like this:

$ python3 utils/process_engine.py --profile_engine engine.trt outputs

Profiling the engine:
trtexec --verbose --noDataTransfers --useCudaGraph --separateProfileRun --nvtxMode=verbose --loadEngine=engine.trt --exportTimes=outputs/engine.trt.timing.json --exportProfile=outputs/engine.trt.profile.json --exportLayerInfo=outputs/engine.trt.graph.json --timingCacheFile=./timing.cache 
Successfully profiled the engine.

Device info: 1 device(s) found
Device info: generated output file outputs/engine.trt.metadata.json
Artifcats directory: outputs

I would be grateful about any hint here.

Environment

TensorRT Version: 8.2.5
NVIDIA GPU: RTX A6000
NVIDIA Driver Version: 515.65.01
CUDA Version: 11.7
CUDNN Version:
Operating System: Ubuntu 20.04
Python Version (if applicable): 3.8.10
Tensorflow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if so, version): nvcr.io/nvidia/pytorch:22.06-py3

2

Hi,
Please refer to below links related custom plugin implementation and sample:

While IPluginV2 and IPluginV2Ext interfaces are still supported for backward compatibility with TensorRT 5.1 and 6.0.x respectively, however, we recommend that you write new plugins or refactor existing ones to target the IPluginV2DynamicExt or IPluginV2IOExt interfaces instead.

Thanks!

4

Do you mean that trtexec does not output the graph properly because my models include plugins implementing the obsolete interfaces?

There are indeed two plugins for the model, but both are inherited from IPluginV2DynamicExt.

My problem is that XXX.graph.json contains just a plain list of layers’ names instead of a graph.

5

The error message says “Make sure to enable detailed ProfilingVerbosity.”
It might have been clearer if it said “Make sure to enable detailed ProfilingVerbosity when building the engine .”
You are using process_engine.py to profile an engine that you’ve created somehow. This is fine, but TRT will not create the JSON containing the graph information (what you refer to as List of Dictionaries above: for each layer there is information about inputs, outputs and layer specific attributes - in the form of a dictionary) if the engine was built with ProfilingVerbosity != kDETAILED .
Since your JSON file has a list of layer names only, we can safely assume the engine was created with ProfilingVerbosity == kLAYER_NAMES_ONLY .
Therefore, please rebuild your engine with the elevated ProfilingVerbosity and try again.

The file process_engine.py uses the old --nvtxMode=verbose argument (equivalent to profilingVerbosity=detailed but backward compatible with 8.2) - see
here

1 Like
6

Thanks, I added this to my engine build code:

self.config.profiling_verbosity = trt.ProfilingVerbosity.DETAILED

Now it works.

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.