Calibrate Model with INT8 Precision and Generate Engine Plan file for DeepStream Consumption

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hi,

Hardware Platform: Nvidia AGX Xavier 8 cores 32GB
DeepStream version: 5.0
JetPack Version: R32 (release), REVISION: 4.3, GCID: 21589087, BOARD: t186ref, EABI: aarch64, DATE: Fri Jun 26 04:34:27 UTC 2020
TensorRT Version: 7.1.3-1+cuda10.2

Issue:
I have tried to calibrate the SSD Mobilenet V1 model to INT8 precision with code snippet as below:

import numpy as np
from PIL import Image

img = Image.open("cat.jpg")
img_data = np.array(img.getdata()).reshape(img.size[1], img.size[0], 3)
img_data = np.expand_dims(img_data.astype(np.uint8), axis=0)

 # Convert and calibrate model
from tensorflow.python.compiler.tensorrt import trt_convert as trt

converter = trt.TrtGraphConverter(input_saved_model_dir="ssd_mobilenet_v1_coco_2018_01_28/saved_model", precision_mode='INT8')
converted_graph_def = converter.convert()
calibrated_graph_def = converter.calibrate(
    fetch_names=['num_detections:0','detection_boxes:0','detection_scores:0','detection_classes:0'],
    num_runs=1,
    feed_dict_fn=lambda: {'image_tensor:0': img_data}
)
converter.save("calibrated_int8_ssdmobilenetv1")

But I am unsure of how to export an engine plan file for DeepStream(Gst) consumption after the calibration?

Thanks,
Vincent

3

I think you need to convert the model to onnx or uff format so that DS can consume that model