Deployment methods of TensorRT-based applications using UFF

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
,
1

Hi,

I am trying to optimise my custom model for MaskRCNN. For this, I successfully converted the h5 model to uff and inferred using

TensorRT 8.0.0.1
Cuda 11.3
cudnn 8.2.1.32-1
TensorFlow 1.15.5
Onyx 1.8

For the inference part, I recompile the sample_uff_maskRCNN after changing its config file. This requires me to run a built executable .sample_uff_maskRCNN each time.

I need to know what options do I have for deploying this optimised Uff model to my Jetson Xavier system ? I am aware of DeepStream and ‘.plan’ inference engine based on onnx workflow. The latter nowhere describes how to use the Uff file. I would be grateful if someone could help me understand all the options I currently have.

Thanks.

3

Hi,

You can try the uff model together with the customized sample_uff_maskRCNN on Jetson.
It may need some update since we now only have TensorRT v7.1.3 on Jetson.

But we are going to release TensorRT 8.0 (JetPack4.6) soon.
You can also wait for the new JetPack.

Thanks.

4

I believe some sort of serialization will help me better. How is it different from the approach you suggest ?

5

Hi,

The sample also serializes and de-serializes the TensorRT engine file:
https://github.com/NVIDIA/TensorRT/blob/master/samples/opensource/sampleUffMaskRCNN/sampleUffMaskRCNN.cpp#L437

But please noted that serialization is hardware-dependent.
You will need to do it on the Jetson platform directly.

Thanks.

6

Oh that’s interesting.

This is my understanding till now:

the above code checks for an serialised engine available and if not found, creates one, on the system its compiled/run on.

Please correct me if wrong.

7

Hi,

Sorry, it doesn’t.
In the main function, it generates an engine whatever there is an engine or not.

In build() function, it will serialize the engine and reload it again to demonstrate the usage.

# serialize
SampleUniquePtr<IHostMemory> plan{builder->buildSerializedNetwork(*network, *config)};
...

# de-serialize
mEngine = std::shared_ptr<nvinfer1::ICudaEngine>(runtime->deserializeCudaEngine(plan->data(), plan->size()), samplesCommon::InferDeleter());

Thanks.

8

As the UFF parser is not supported now, I am trying the ONNX parser. Thanks for this, might need it sometime later.

9

Hi,

Let us close this issue since it is for the uff model.
If you meet any error with the ONNX, please file a new topic specified for it.

Thanks.

11

Hi AstaLLL, with the release of 4.6, can you please guide me on how this new JetPack can help me deploy my model now ?