Running EES with TensorRT in C++

[ruben_v]

Hi,

I already used ESS DNN Stereo Disparity
with Isaac ROS2 docker environment and now want to create a simple C++ example for stand alone inference of two example images (left & right). I followed the TensorRT documentation to convert the .etlt file to an .engine file but now I got stuck on how to continue.

I got this from the documentations:

nvinfer1::IRuntime* runtime(nvinfer1::createInferRuntime(logger));
nvinfer1::ICudaEngine* engine(runtime->deserializeCudaEngine(engineData.data(), engineData.size()));
nvinfer1::IExecutionContext* context = engine->createExecutionContext();

but how to continue?

• after reading the images do I need to copy them to GPU memory?
• how to perform inference using both images?
• how to get the output images for disparity and confidence?
• Is there an example somewhere around for EES in C++? Unfortunately I couldn’t find what I am looking for in the ROS2 node.

Thanks in advance

Environment

TensorRT Version: 10.2.0.19
GPU Type: RTX A4500
CUDA Version: 12.2.2
Operating System + Version: Windows 10 Enterprise

[ruben_v]

I will answer my question myself. Feels like level up TensorRT Dev to get this thing run :)

External plugin:
ESS uses an external plugin, that needs to be added before deserializing the engine.

In cmake make sure to add libnvinfer_plugin.so:

target_link_libraries(${PROJECT_NAME}
				  libnvinfer.so
				  libnvinfer_plugin.so)

Load external plugin:

#include <NvInferPlugin.h>
#include <NvInferPluginUtils.h>

....
initLibNvInferPlugins(&logger, "");
getPluginRegistry()->loadLibrary("path/to/ess_plugin.so");

Create execution context:

auto runtime = std::shared_ptr<nvinfer1::IRuntime>{ nvinfer1::createInferRuntime(logger) };
auto engine = std::shared_ptr<nvinfer1::ICudaEngine>(runtime->deserializeCudaEngine(engineData.data(), engineData.size()));
auto context = std::shared_ptr<nvinfer1::IExecutionContext>(engine->createExecutionContext());

Normalize input data

for left and right images apply transformation

pixel_new = (float(pixel) / 255.0 - 0.5) / 0.5

run inference

void* bindings[] = { input_left_mem, input_right_mem, confidence_mem , disparity_mem};

std::cout << "async: enqueue inference" << std::endl;
bool status = context->enqueueV2(bindings, stream, nullptr);

input_left_mem, input_right_mem, disparity_mem, confidence_mem is device memory of needed size. (sizeof(float) * net_dimensions)

After inference copy memory from device to host

Result: