Can't make TAO

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

My system information:

• Hardware Platform : GPU
• DeepStream Version : 6.0.1
• TensorRT Version : 8.0.1
• NVIDIA GPU Driver Version : 470.103.01

I am trying to make tao apps from here.

But, when I make the tao apps, there is an error.

export CUDA_VER=11.4
make
make -C post_processor
make[1]: Entering directory '/opt/nvidia/deepstream/deepstream-6.0/deepstream_tao_apps/post_processor'
g++ -o libnvds_infercustomparser_tao.so nvdsinfer_custombboxparser_tao.cpp -I/opt/nvidia/deepstream/deepstream-6.0/sources/includes -I/usr/local/cuda-11.4/include -Wall -std=c++11 -shared -fPIC -Wl,--start-group -lnvinfer -lnvparsers -L/usr/local/cuda-11.4/lib64 -lcudart -lcublas -Wl,--end-group
make[1]: Leaving directory '/opt/nvidia/deepstream/deepstream-6.0/deepstream_tao_apps/post_processor'
make -C apps
make[1]: Entering directory '/opt/nvidia/deepstream/deepstream-6.0/deepstream_tao_apps/apps'
make -C tao_detection
make[2]: Entering directory '/opt/nvidia/deepstream/deepstream-6.0/deepstream_tao_apps/apps/tao_detection'
g++ -c -o deepstream_det_app.o -I/opt/nvidia/deepstream/deepstream-6.0/sources/includes `pkg-config --cflags gstreamer-1.0` -std=c++14 deepstream_det_app.c
deepstream_det_app.c:30:10: fatal error: nvds_yml_parser.h: No such file or directory
   30 | #include "nvds_yml_parser.h"
      |          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [Makefile:66: deepstream_det_app.o] Error 1
make[2]: Leaving directory '/opt/nvidia/deepstream/deepstream-6.0/deepstream_tao_apps/apps/tao_detection'
make[1]: *** [Makefile:24: all] Error 2
make[1]: Leaving directory '/opt/nvidia/deepstream/deepstream-6.0/deepstream_tao_apps/apps'
make: *** [Makefile:25: all] Error 2
3

Can you find nvds_yml_parser.h in sources/includes/

4

No, it’s not there:

ls /opt/nvidia/deepstream/deepstream-6.0/sources/includes/
cvcore_headers       gstnvdsmeta.h          nvds_latency_meta.h      nvds_tracker_meta.h      nvdsinfer_logger.h            nvdsmeta.h                  nvmsgbroker.h
gst-nvdssr.h         nvbufaudio.h           nvds_logger.h            nvds_version.h           nvdsinfer_tlt.h               nvdsmeta_schema.h
gst-nvevent.h        nvbufsurface.h         nvds_mask_utils.h        nvdsgstutils.h           nvdsinfer_utils.h             nvdspreprocess_interface.h
gst-nvmessage.h      nvbufsurftransform.h   nvds_msgapi.h            nvdsinfer.h              nvdsinferserver               nvdspreprocess_meta.h
gst-nvquery.h        nvds_analytics_meta.h  nvds_obj_encode.h        nvdsinfer_context.h      nvdsinferserver_common.proto  nvdstracker.h
gstnvdsbufferpool.h  nvds_audio_meta.h      nvds_opticalflow_meta.h  nvdsinfer_custom_impl.h  nvdsinferserver_config.proto  nvll_osd_api.h
gstnvdsinfer.h       nvds_dewarper_meta.h   nvds_roi_meta.h          nvdsinfer_dbscan.h       nvdsinferserver_plugin.proto  nvll_osd_struct.h
5

This file included since Deepstream 6.1
If you do not want to upgrade deepstream, please use tao app with 6.0 branch.

1 Like
6

It worked past that.

But now, when i try to run any app the execution gets stuck here:

 /opt/nvidia/deepstream/deepstream-6.0/deepstream_tao_apps# ./apps/tao_segmentation/ds-tao-segmentation -c configs/unet_tao/pgie_unet_tao_config.txt -i /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_720p.h264
Now playing: configs/unet_tao/pgie_unet_tao_config.txt
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:1484 Deserialize engine failed because file path: /opt/nvidia/deepstream/deepstream-6.0/deepstream_tao_apps/configs/unet_tao/../../models/unet/unet_resnet18.etlt_b1_gpu0_fp16.engine open error
0:00:00.791441536  6213 0x55e694508130 WARN                 nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1889> [UID = 1]: deserialize engine from file :/opt/nvidia/deepstream/deepstream-6.0/deepstream_tao_apps/configs/unet_tao/../../models/unet/unet_resnet18.etlt_b1_gpu0_fp16.engine failed
0:00:00.791494548  6213 0x55e694508130 WARN                 nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1996> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/deepstream/deepstream-6.0/deepstream_tao_apps/configs/unet_tao/../../models/unet/unet_resnet18.etlt_b1_gpu0_fp16.engine failed, try rebuild
0:00:00.791502885  6213 0x55e694508130 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1914> [UID = 1]: Trying to create engine from model files
WARNING: [TRT]: onnx2trt_utils.cpp:362: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
WARNING: [TRT]: Detected invalid timing cache, setup a local cache instead
7

Build engine may take some long time.

8

When I try to run peopleSegNet, process gets automatically killed after some time:

/opt/nvidia/deepstream/deepstream-6.0/deepstream_tao_apps# ./apps/tao_segmentation/ds-tao-segmentation -c conf
igs/peopleSegNet_tao/pgie_peopleSegNet_tao_config.txt -i /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_720p.h264
Now playing: configs/peopleSegNet_tao/pgie_peopleSegNet_tao_config.txt
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:1484 Deserialize engine failed because file path: /opt/nvidia/deepstream/deepstream-6.0/deepstream_tao_apps/configs/peopleSegNet_tao/../../models/peopleSegNet/peopleSegNet_resnet50.etlt_b1_gpu0_fp16.engine open error
0:00:00.577486621 139506 0x559dc6c8b330 WARN                 nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1889> [UID = 1]: deserialize engine from file :/opt/nvidia/deepstream/deepstream-6.0/deepstream_tao_apps/configs/peopleSegNet_tao/../../models/peopleSegNet/peopleSegNet_resnet50.etlt_b1_gpu0_fp16.engine failed
0:00:00.577531749 139506 0x559dc6c8b330 WARN                 nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1996> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/deepstream/deepstream-6.0/deepstream_tao_apps/configs/peopleSegNet_tao/../../models/peopleSegNet/peopleSegNet_resnet50.etlt_b1_gpu0_fp16.engine failed, try rebuild
0:00:00.577538590 139506 0x559dc6c8b330 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1914> [UID = 1]: Trying to create engine from model files
WARNING: [TRT]: Detected invalid timing cache, setup a local cache instead
Killed

I commented below line from config file
#model-engine-file=../../models/peopleSegNet/peopleSegNet_resnet50.etlt_b1_gpu0_fp16.engine

/opt/nvidia/deepstream/deepstream-6.0/deepstream_tao_apps# ./apps/tao_segmentation/ds-tao-segmentation -c configs/peopleSegNet_tao/pgie_peopleSegNet_tao_config.txt -i /opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_720p.h264
Now playing: configs/peopleSegNet_tao/pgie_peopleSegNet_tao_config.txt
0:00:00.362525168 141079 0x55ba8746d730 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1914> [UID = 1]: Trying to create engine from model files
WARNING: [TRT]: Detected invalid timing cache, setup a local cache instead
Killed

In both the cases, the process gets killed automatically after some time.

9

If you do not kill the process, and if system got to circumstances like extreme resource starvation, the process will be killed. you can monitor the system usage like cpu/memory.

10

Something 3500 MB of GPU memory and 11 GB of RAM is available before starting. There are no more processes to kill. Is there a workaround?

11

How about system usage when the process running?

12

On start it uses >1700MB of GPU, > 1GB of RAM. After some time RAM use starts increasing untill it uses all the RAM, and process gets killed automatically.

13

How about your total RAM?

14

RAM is 16GB. GPU is 4GB.

15

It’s fine with GPU memory 4G to run deepstream, but your app caused GPU memory up to 3.5G, did you still have any other process which needs GPU? I used unet model, and total GPU memory is aroung 1.7G

16

GPU memory is not overflown, only 1700 MB GPU is being used. Only RAM is overused. Also, UNET is working fine. I’m having issue with Mask RCNN (PeopleSegMet). Also, not other process is there when running PeopleSegNet with given apps/tao_segmentation/ds-tao-segmentation file.

17

Peoplesegnet is detection model. use detection app try again.

18

Will try, but PeopleSegNet is MaskRCNN, and is for segmentation, right?

19

Sorry, Peoplesegnet is for segmentation.

20

I think there is no meaning of trying detection app then. What should I try next?

21

20539 root 20 0 9.962g 2.539g 589552 S 14.6 2.0 0:04.27 ds-tao-segmenta
The process used total around 10G virtual memory, in physical memory it’s around 2.5G, the remaining 7.5G will be in swap area. did your os partition include swap partition?