•Hardware Platform (Jetson / GPU): Deployment on Jetson Xavier NX and development on NVIDIA 2080Ti based Ubuntu machine • NVIDIA GPU Driver Version (valid for GPU only) 2080 Ti on the development machine and the native GPU on NX • Issue Type Question • Requirement details The development environment is in Ubuntu 20.0.4. In the Jupyter Notebook, I’m able to infer the images using the following script, !ssd inference --gpu_index=$GPU_INDEX -m $USER_EXPERIMENT_DIR/export/trt.engine -e $SPECS_DIR/ssd_retrain_resnet18_kitti.txt -i /workspace/examples/ssd/test_samples -o $USER_EXPERIMENT_DIR/ssd_infer_images -t 0.4 Now, How do I write business logic on top of it by feeding images from the OpenCV? Where is this file that has got the inference code? In Jetson, I have used Deepstream to deploy. But I couldn’t find a simple implementation that shows how I could deploy the custom model on the Jetson using Python API? I have both .tlt,.etlt,.engine files . And how do I see the output as we would with OpenCV using imshow ?

Actually end users can generate trt engine based on etlt model. If they want to run inference with deepstream, they can deploy etlt model or trt engine in it. If they want to run inference with standalone inference way with the trt engine, end user can do some coding with correct preprocesing, postprocessing,etc. Some examples, ( TLT different results - #11 ) ( Run PeopleNet with tensorrt ) (Python run LPRNet with TensorRT show pycuda._driver.MemoryError: cuMemHostAlloc failed: out of memory - #8 by Morganh)

What do you mean by " business logic" ? Do you mean you want to run inference without deepstream or “tlt ssd inference”?

Yes… I want to be able to run it without Deepstream… I have an incoming stream from OpenCV. I would like to know how I can feed it to the model and perform the detection.

But, I’m developing the application on Ubuntu 20.0.4. NVIDIA doesn’t support deep stream for Ubuntu 20.0.4. I have trained the models with Ubuntu 20.0.4. I know that the following command performs inference and displays the image output, !ssd inference --gpu_index=$GPU_INDEX -m $USER_EXPERIMENT_DIR/export/trt.engine -e $SPECS_DIR/ssd_retrain_resnet18_kitti.txt -i /workspace/examples/ssd/test_samples -o $USER_EXPERIMENT_DIR/ssd_infer_images -t 0.4 But, I’m not interested in seeing some visual output. I would like to obtain coordinates so I can write other logic on top of the detected objects. How can I even do it? I have been striving for nearly a month.

[image] linuxfs: I would like to obtain coordinates Hi, In the output folder $USER_EXPERIMENT_DIR/ssd_infer_images, there are coordinates.

I mean I want to receive it via code. For example, something like this. bounding_boxes = model.predict(frame) How do I do that in my Ubuntu 20.0.4 docker that I have used to train the model via TLT?

The ssd inference is not open source. It can only provide bboxes in the output folder. For your case, suggest you to try to run inference with one of below ways: Run GitHub - NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream Run inference against the trt engine

Thanks! It would have saved my time and efforts if I knew that it was not open source previously since I wouldn’t have tried training the model with TLT. Finally, we have used the onnxruntime-gpu option. It was simple, easily works with OpenCV and most essentially it is very well documented. The performance was better than Deepstream. Considering the pain in configuring Deepstream + the closed source TLT + conversion of TRT etc…, onnxruntime is way better and straightforward.

I was having the same issue. The fact TLT is so hard to integrate (Due to closed source) in proyects makes it impossible to use, same as the bare answers you get from Nvidia, you need to beg for better and more developed answers… Might as well give ONNX runtime a try.

Maybe this should be a new topic, but i was able to create a trt engine with tlt, but i’m getting wrong inferences, i saw another issue related to this but had no success, the pre processing that tlt does looks like a black box, i tried scaling the image with padding, normalizing and changing BGR to RGB but no success. I’m sorry, tensorrt is a great tool but i’m still stuck in this matter for a month and got no progress.

How do we write business logic with python on top of the model trained with TLT?

Accelerated Computing Intelligent Video Analytics TAO Toolkit

Morganh May 27, 2021, 2:48pm 5

Refer to Why we need image pre and post processing when deploying a tlt model by using TensorRT? - #5 by Morganh and How to use tlt trained model on Jetson Nano - #3 by Morganh

Topic		Replies	Views
How do we write business logic on top of the model trained with TLT? DeepStream SDK tensorrt , opencv , ubuntu , jetson-inference , python	2	459	May 20, 2021
How to use tlt trained model on Jetson Nano TAO Toolkit tensorrt , jetson-inference	6	2295	November 12, 2020
How preform inference retinanet using a TLT export .engine file by python TAO Toolkit tensorrt	3	1024	February 23, 2021
Using TLT trained models in TF (or Keras) to run inference TAO Toolkit	8	1621	June 22, 2021
trt engine inference in python without deepstream TAO Toolkit	8	1718	October 4, 2020
How to do inference in own code? TAO Toolkit	7	1153	May 26, 2020
Apart from Deepstream where else I can deploy tlt-converted models or .trt engine files TAO Toolkit	4	1630	December 2, 2020
Transfer Learning Toolkit for Jetson Nano TAO Toolkit	13	1634	May 21, 2020
Running .engine file outside deepstream SDK TAO Toolkit	1	559	October 15, 2019
Python App Cutom Model on the Jetson Nano TAO Toolkit	9	1435	October 1, 2020

How do we write business logic with python on top of the model trained with TLT?

Related topics