How do we write business logic with python on top of the model trained with TLT?

•Hardware Platform (Jetson / GPU): Deployment on Jetson Xavier NX and development on NVIDIA 2080Ti based Ubuntu machine
• NVIDIA GPU Driver Version (valid for GPU only) 2080 Ti on the development machine and the native GPU on NX
• Issue Type Question
• Requirement details

  1. The development environment is in Ubuntu 20.0.4. In the Jupyter Notebook, I’m able to infer the images using the following script,

    !ssd inference --gpu_index=$GPU_INDEX
    -m $USER_EXPERIMENT_DIR/export/trt.engine
    -e $SPECS_DIR/ssd_retrain_resnet18_kitti.txt
    -i /workspace/examples/ssd/test_samples
    -o $USER_EXPERIMENT_DIR/ssd_infer_images
    -t 0.4
    

    Now, How do I write business logic on top of it by feeding images from the OpenCV? Where is this file that has got the inference code?

  2. In Jetson, I have used Deepstream to deploy. But I couldn’t find a simple implementation that shows how I could deploy the custom model on the Jetson using Python API? I have both .tlt,.etlt,.engine files . And how do I see the output as we would with OpenCV using imshow ?

What do you mean by " business logic" ?
Do you mean you want to run inference without deepstream or “tlt ssd inference”?

Yes… I want to be able to run it without Deepstream… I have an incoming stream from OpenCV. I would like to know how I can feed it to the model and perform the detection.

Refer to Why we need image pre and post processing when deploying a tlt model by using TensorRT? - #5 by Morganh and How to use tlt trained model on Jetson Nano - #3 by Morganh

But, I’m developing the application on Ubuntu 20.0.4. NVIDIA doesn’t support deep stream for Ubuntu 20.0.4. I have trained the models with Ubuntu 20.0.4. I know that the following command performs inference and displays the image output,

!ssd inference --gpu_index=$GPU_INDEX
-m $USER_EXPERIMENT_DIR/export/trt.engine
-e $SPECS_DIR/ssd_retrain_resnet18_kitti.txt
-i /workspace/examples/ssd/test_samples
-o $USER_EXPERIMENT_DIR/ssd_infer_images
-t 0.4

But, I’m not interested in seeing some visual output. I would like to obtain coordinates so I can write other logic on top of the detected objects. How can I even do it? I have been striving for nearly a month.

Hi,
In the output folder $USER_EXPERIMENT_DIR/ssd_infer_images, there are coordinates.

I mean I want to receive it via code. For example, something like this.

bounding_boxes = model.predict(frame)

How do I do that in my Ubuntu 20.0.4 docker that I have used to train the model via TLT?

The ssd inference is not open source. It can only provide bboxes in the output folder.
For your case, suggest you to try to run inference with one of below ways:

Thanks! It would have saved my time and efforts if I knew that it was not open source previously since I wouldn’t have tried training the model with TLT. Finally, we have used the onnxruntime-gpu option. It was simple, easily works with OpenCV and most essentially it is very well documented. The performance was better than Deepstream. Considering the pain in configuring Deepstream + the closed source TLT + conversion of TRT etc…, onnxruntime is way better and straightforward.