Run peoplenet on python virtual environment without jetson

shima.khosravani · December 21, 2023, 3:55pm

Hello all
Is there a way to run the peoplenet model on a python virtual environment on Ubuntu? I’m a bit confused. I have downloaded the .onnx model from Nvidia model card page. But I do not know the following steps. From here٫please tell me step by step what to do to run the model on an input image,
I have download’ed the model from here:

Morganh · December 21, 2023, 4:05pm

There are three different kinds of peoplenet now.
One is trained from detectnet_v2 network.
2nd is trained from D-DETR network.
3rd is trained from DINO network.

A simple way to run peoplenet inference is to use a deepstream docker.
See below steps.

$ docker run --runtime=nvidia -it --rm nvcr.io/nvidia/deepstream:6.3-triton-multiarch  /bin/bash
# apt install libeigen3-dev && cd /usr/include && ln -sf eigen3/Eigen Eigen
# cd -
# git clone https://github.com/NVIDIA-AI-IOT/deepstream_tao_apps.git
# cd deepstream_tao_apps
# export CUDA_VER=12.1
# make

# mkdir -p ./models/peoplenet && cd ./models/peoplenet && wget https://api.ngc.nvidia.com/v2/models/nvidia/tao/peoplenet/versions/deployable_quantized_v2.6.1/files/resnet34_peoplenet_int8.etlt && wget https://api.ngc.nvidia.com/v2/models/nvidia/tao/peoplenet/versions/deployable_quantized_v2.6.1/files/resnet34_peoplenet_int8.txt && wget https://api.ngc.nvidia.com/v2/models/nvidia/tao/peoplenet/versions/deployable_quantized_v2.6.1/files/labels.txt

# cd - && mkdir -p ./models/peoplenet_transformer  && cd ./models/peoplenet_transformer && wget --content-disposition 'https://api.ngc.nvidia.com/v2/models/org/nvidia/team/tao/peoplenet_transformer/deployable_v1.1/files?redirect=true&path=resnet50_peoplenet_transformer_op12.onnx' -O resnet50_peoplenet_transformer_op12.onnx && wget --content-disposition 'https://api.ngc.nvidia.com/v2/models/org/nvidia/team/tao/peoplenet_transformer/deployable_v1.0/files?redirect=true&path=labels.txt' -O labels.txt && wget --content-disposition 'https://api.ngc.nvidia.com/v2/models/org/nvidia/team/tao/peoplenet_transformer_v2/deployable_v1.0/files?redirect=true&path=dino_fan_small_astro_delta.onnx' -O dino_fan_small_astro_delta.onnx

# cd -
# ./apps/tao_detection/ds-tao-detection -c configs/nvinfer/peoplenet_tao/config_infer_primary_peoplenet.txt -i file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4

For 2nd peoplenet,

$ vim configs/nvinfer/peoplenet_transformer_tao/pgie_peoplenet_transformer_tao_config.txt
onnx-file = ../../../models/peoplenet_transformer/resnet50_peoplenet_transformer_op12.onnx
model-engine-file=../../../models/peoplenet_transformer/resnet50_peoplenet_transformer_op12.onnx_b1_gpu0_fp16.engine
#model-engine-file=../../../models/peoplenet_transformer/resnet50_peoplenet_transformer.etlt_b1_gpu0_fp16.engine
#tlt-encoded-model=../../../models/peoplenet_transformer/resnet50_peoplenet_transformer.etlt
#tlt-model-key=nvidia_tao

$./apps/tao_detection/ds-tao-detection -c configs/nvinfer/peoplenet_transformer_tao/pgie_peoplenet_transformer_tao_config.txt -i file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4

For 3rd peoplenet,

update TensorRT8.5 to TensorRT 8.6
$ apt-get install sudo
$ wget https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/secure/8.6.1/local_repos/nv-tensorrt-local-repo-ubuntu2004-8.6.1-cuda-11.8_1.0-1_amd64.deb
$ sudo dpkg -i nv-tensorrt-local-repo-ubuntu2004-8.6.1-cuda-11.8_1.0-1_amd64.deb
$ sudo cp /var/nv-tensorrt-local-repo-ubuntu2004-8.6.1-cuda-11.8/nv-tensorrt-local-D7BB1B18-keyring.gpg /usr/share/keyrings/
$ sudo apt-get update
$ sudo apt-get install tensorrt

$ vim configs/nvinfer/peoplenet_transformer_tao/pgie_peoplenet_transformer_tao_config.txt
onnx-file = ../../../models/peoplenet_transformer/dino_fan_small_astro_delta.onnx
model-engine-file=../../../models/peoplenet_transformer/dino_fan_small_astro_delta.onnx_b1_gpu0_fp16.engine
#model-engine-file=../../../models/peoplenet_transformer/resnet50_peoplenet_transformer.etlt_b1_gpu0_fp16.engine
#tlt-encoded-model=../../../models/peoplenet_transformer/resnet50_peoplenet_transformer.etlt
#tlt-model-key=nvidia_tao

$./apps/tao_detection/ds-tao-detection -c configs/nvinfer/peoplenet_transformer_tao/pgie_peoplenet_transformer_tao_config.txt -i file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4

shima.khosravani · December 21, 2023, 4:13pm

thanks
what are the requirements for the first approach ? python version, cuda version and … ?

Morganh · December 21, 2023, 4:42pm

Above steps are running inside the docker nvcr.io/nvidia/deepstream:6.3-triton-multiarch. So, you can directly docker run into it.

shima.khosravani · December 24, 2023, 12:05pm

So, If I want to run the first model on webrtc/rtsp , there is no need to use GitHub - NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream

just installing docker is enough?

Morganh · December 25, 2023, 1:43am

It is needed. Above-mentioned steps are running via GitHub - NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream.

shima.khosravani · December 25, 2023, 10:12am

Thanks, I got.

shima.khosravani · January 1, 2024, 7:37am

Can I run the model which detects bags ( resnet34 ,v2.0 , unpruned) with deepstreem ?

Morganh · January 1, 2024, 3:09pm

Yes, you can run with deepstream.

shima.khosravani · January 2, 2024, 9:45am

I firstly have chosen this docker
nvcr.io/nvidia/tlt-streamanalytics:v2.0_py3

to run nvidia engine model on camera stream, however it is unable to get frames from the camera.

[h264 @ 0x28e7180] error while decoding MB 16 17, bytestream -8
Can’t receive frame (stream end?). Exiting …

(the code is correct because it can be run outside the container)
I have tested ordinary solutions in the net, but were not effective. I s there any special settings for nvidia containers ?

Morganh · January 2, 2024, 4:40pm

Could you please follow the above-mentioned steps mentioned in Run peoplenet on python virtual environment without jetson - #2 by Morganh to run inference? It will run inside a deepstream docker.

shima.khosravani · January 3, 2024, 9:14am

Thanks Morganh
I have run the docker with this command

sudo docker run --gpus all --net=host -it -v /home/My_folder/p1:/workspace/tlt-experiments --entrypo
int “” -e DISPLAY=$DISPLAY --env=“QT_X11_NO_MITSHM=1” nvcr.io/nvidia/tlt-streamanalytics:v2.0_py3 /bin/bash

but now, there is another problem
the model does not work correctly , I mean the model output on a single image is different from tao-deploy output.

is there any problem with the .tlt moldel , or this is because , in the tao deploy , I used the tensorrt version, and here , I use .tlt version ? or maybe there is something wrong with my config file ?
this is a sample output of the .tlt model

and this is the tensorrt output with tao deploy

Morganh · January 3, 2024, 9:21am

May I know which one does not work correctly? Run with .tlt model or tensorrt engine?
From above screenshot, it works well.

Also nvcr.io/nvidia/tlt-streamanalytics:v2.0_py3 is an old version, you can use the same version of tao-py docker and tao-deploy docker.
For example, you can run with nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf1.15.5 and nvcr.io/nvidia/tao/tao-toolkit:5.0.0-deploy.

shima.khosravani · January 3, 2024, 9:35am

the tlt version does not work correctly. false positive is high. the output of tensorrt is more precise
I want to detect bags and use the model on camera stream. So , I must be able to feed a single image to the model, but in tao-deploy, the input is a directory. So, when I add frame images to the directory, the model will process all the directory (including previous frames ) as well, which reduces the performance.
So, due to this thread :

I have used the tlt version.

with the inference command
tlt-infer detectnet_v2 -e detectnet_v2_inference_kitti_tlt.txt -k tlt_encode -o infer_result -i frame_238.jpg -v
but the results are not as good as tao version

please let me know how to use the tensorrt version without
reducing the performance

Morganh · January 3, 2024, 9:43am

Please run with nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf1.15.5 to double check.

$ docker run --runtime=nvidia -it --rm nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf1.15.5 /bin/bash

Then inside the docker, please run
#detectnet_v2 inference -e $SPECS_DIR/detectnet_v2_inference_kitti_tlt.txt
-r $USER_EXPERIMENT_DIR/tlt_infer_testing
-i $DATA_DOWNLOAD_DIR/test_samples

Refer to tao_tutorials/notebooks/tao_launcher_starter_kit/detectnet_v2/detectnet_v2.ipynb at main · NVIDIA/tao_tutorials · GitHub and tao_tutorials/notebooks/tao_launcher_starter_kit/detectnet_v2/specs/detectnet_v2_inference_kitti_tlt.txt at main · NVIDIA/tao_tutorials · GitHub.
Note: need to change to 960x544 in the spec file.

shima.khosravani · January 3, 2024, 9:46am

I have done this
but , how can I feed an image to the model ? rather that a directory ?
here: -i $DATA_DOWNLOAD_DIR/test_samples

Morganh · January 3, 2024, 9:49am

As a workaround, you can copy only one image to the folder test_samples.

shima.khosravani · January 3, 2024, 9:50am

So, how can I handle it in a camera stream?
when I copy the new frame to the directory, the model will process all the content again.

Morganh · January 3, 2024, 9:58am

There are several ways to run inference for peoplenet.

You can run with above-mentioned. This way run run with etlt(or onnx) model or tensorrt engine.

# ./apps/tao_detection/ds-tao-detection -c configs/nvinfer/peoplenet_tao/config_infer_primary_peoplenet.txt -i file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4

The input file can be a stream.

You can run with tao-tf docker(you are running), according to tao_tensorflow1_backend/nvidia_tao_tf1/cv/detectnet_v2/scripts/inference.py at main · NVIDIA/tao_tensorflow1_backend · GitHub, it can run inference against the folder or single image.
You can also run with GitHub - NVIDIA-AI-IOT/tao-toolkit-triton-apps: Sample app code for deploying TAO Toolkit trained models to Triton, using detectnet_v2 section since peoplenet is actually trained with detectnet_v2 network.
You can also run with standalone way. For example, Run PeopleNet with tensorrt - #22 by gharbv.

shima.khosravani · January 3, 2024, 10:36am

Is the second solution specific to version 5.0 ?

I have ran this command:

detectnet_v2 inference
-e /home/shima/p1/nvidia-tao/tao_deploy/specs/PeopleNet/PeopleNet_infer.txt
-m /home/shima/p1/out_dir3/PeopleNet.trt.fp32
-r /home/shima/p1/compare_out
-i /home/shima/p1/tao_compare/frame.jpg

leading to this error:

Loading uff directly from the package source code
Traceback (most recent call last):
File “</home/shima/p1/venv38/lib/python3.8/site-packages/nvidia_tao_deploy/cv/detectnet_v2/scripts/inference.py>”, line 3, in
File “”, line 169, in
File “”, line 39, in main
File “”, line 21, in init
File “”, line 67, in init
File “”, line 92, in _add_source
NotADirectoryError: [Errno 20] Not a directory: ‘/home/shima/p1/tao_compare/frame.jpg’
2024-01-03 14:02:32,749 [INFO] nvidia_tao_deploy.cv.common.entrypoint.entrypoint_proto: Sending telemetry data.
2024-01-03 14:02:35,250 [WARNING] nvidia_tao_deploy.cv.common.entrypoint.entrypoint_proto: Telemetry data couldn’t be sent, but the command ran successfully.
2024-01-03 14:02:35,250 [WARNING] nvidia_tao_deploy.cv.common.entrypoint.entrypoint_proto: [Error]: <urlopen error EOF occurred in violation of protocol (_ssl.c:1131)>
2024-01-03 14:02:35,250 [WARNING] nvidia_tao_deploy.cv.common.entrypoint.entrypoint_proto: Execution status: FAIL

I use these versions:
nvidia-cublas-cu11 11.11.3.6
nvidia-cuda-nvrtc-cu11 11.8.89
nvidia-cuda-runtime-cu11 11.8.89
nvidia-cudnn-cu11 8.9.6.50
nvidia-eff 0.6.2
nvidia-eff-tao-encryption 0.1.7
nvidia-ml-py 12.535.133
nvidia-pyindex 1.0.9
nvidia-tao 4.0.1
nvidia-tao-deploy 4.0.0.1
nvidia-tensorrt 8.4.1.5

Topic		Replies	Views
PeopleNet not detecting bags TAO Toolkit tensorrt	10	1234	September 30, 2022
Peoplenet resnet34 v2.0 inference , bag detection TAO Toolkit	12	630	January 2, 2024
Run PeopleNet with tensorrt TAO Toolkit	35	9804	August 10, 2021
Jetson-inference detectnet.py --model=peoplenet , no engine file Jetson Xavier NX jetson-inference	11	590	February 13, 2024
PeopleNet. Coverage output is always zero TAO Toolkit	12	1801	October 12, 2021
Peoplenet model error on Jetson Orin Nano Jetson Orin Nano jetson-inference	6	481	February 29, 2024
Errors while running inference on Peoplenet Model (no training) TAO Toolkit	3	452	December 28, 2021
Custom TAO unet model classifying only two classes on Deepstream! TAO Toolkit	34	1700	May 12, 2022
Fine-tuning Peoplenet Resnet 34 on AWS. "failed to connect to vfs socket" TAO Toolkit	16	928	October 6, 2023
Update the docs for deepstream_tao_apps DeepStream SDK nvbugs , python , documentation , inference-server-triton , tao , deepstream	4	50	November 18, 2024

Run peoplenet on python virtual environment without jetson

Related topics