Issues with running inference on multiple rtsp streams in deepstream-imagedata-multistream

• Hardware Platform (Jetson / GPU) - Jetson Orin Nano Developer Kit
• DeepStream Version - Docker Container - deepstream:7.0-samples-multiarch
• JetPack Version - 6.0
• TensorRT Version - 8.6.2.3
• NVIDIA GPU Driver Version (valid for GPU only) -
• Issue Type (questions, new requirements, bugs) - Question / Bug
• How to reproduce the issue ?

Ran the docker container using:

sudo docker run --runtime=nvidia -it --rm --net=host --privileged -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY -w /opt/nvidia/deepstream/deepstream-7.0 [nvcr.io/nvidia/deepstream:7.0-samples-multiarch](http://nvcr.io/nvidia/deepstream:7.0-samples-multiarch)

Cloned the deepstream_python_apps repo

cd /opt/nvidia/deepstream/deepstream-7.0/sources
git clone https://github.com/NVIDIA-AI-IOT/deepstream_python_apps

Built the python bindings using the instructions shared in the bindings’s README file

cd /opt/nvidia/deepstream/deepstream-6.4/sources/deepstream_python_apps/bindings

For my current application, the sample python app deepstream-imagedata-multistream would suit it the most. And I wanted the application to detect humans, so I used the Pretrained TAO model Peoplenet.

cd sources/deepstream_python_apps/apps/deepstream-imagedata-multistream

Used the TAO Pretrained model (peoplenet) and created the model engine files using trtexec.

cd /usr/src/tensorrt/bin
trtexec --onnx=/opt/nvidia/deepstream/deepstream-7.0/samples/models/tao_pretrained_models/peopleNet/resnet34_peoplenet_int8.onnx --saveEngine=/opt/nvidia/deepstream/deepstream-7.0/samples/models/tao_pretrained_models/peopleNet/resnet34_peoplenet_int8.engine

Here’s my dstest_imagedata_config.txt file:

# Following properties are mandatory when engine files are not specified:
#   int8-calib-file(Only in INT8)
#   Caffemodel mandatory properties: model-file, proto-file, output-blob-names
#   UFF: uff-file, input-dims, uff-input-blob-name, output-blob-names
#   ONNX: onnx-file
#
# Mandatory properties for detectors:
#   num-detected-classes
#
# Optional properties for detectors:
#   cluster-mode(Default=Group Rectangles), interval(Primary mode only, Default=0)
#   custom-lib-path,
#   parse-bbox-func-name
#
# Mandatory properties for classifiers:
#   classifier-threshold, is-classifier
#
# Optional properties for classifiers:
#   classifier-async-mode(Secondary mode only, Default=false)
#
# Optional properties in secondary mode:
#   operate-on-gie-id(Default=0), operate-on-class-ids(Defaults to all classes),
#   input-object-min-width, input-object-min-height, input-object-max-width,
#   input-object-max-height
#
# Following properties are always recommended:
#   batch-size(Default=1)
#
# Other optional properties:
#   net-scale-factor(Default=1), network-mode(Default=0 i.e FP32),
#   model-color-format(Default=0 i.e. RGB) model-engine-file, labelfile-path,
#   mean-file, gie-unique-id(Default=0), offsets, process-mode (Default=1 i.e. primary),
#   custom-lib-path, network-mode(Default=0 i.e FP32)
#
# The values in the config file are overridden by values set through GObject
# properties.

[property]
gpu-id=0
net-scale-factor=0.00392156862745098
offsets=0.0;0.0;0.0
maintain-aspect-ratio=0
tlt-model-key=tlt_encode
tlt-encoded-model=/opt/nvidia/deepstream/deepstream-7.0/samples/models/tao_pretrained_models/peopleNet/resnet34_peoplenet_int8.onnx
model-engine-file=/opt/nvidia/deepstream/deepstream-7.0/samples/models/tao_pretrained_models/peopleNet/resnet34_peoplenet_int8.engine
labelfile-path=/opt/nvidia/deepstream/deepstream-7.0/samples/models/tao_pretrained_models/peopleNet/labels.txt
# int8-calib-file=../../../../samples/models/Primary_Detector/cal_trt.bin
force-implicit-batch-dim=1
batch-size=30
process-mode=1
model-color-format=0
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=1
network-type=0
num-detected-classes=3
interval=0
gie-unique-id=1
uff-input-order=0
uff-input-blob-name=input_1
output-blob-names=output_cov/Sigmoid;output_bbox/BiasAdd
#scaling-filter=0
#scaling-compute-hw=0
cluster-mode=2
infer-dims=3;544;960
output-tensor-meta=0

[class-attrs-all]
pre-cluster-threshold=0.2
eps=0.2
group-threshold=1

I modified the python file to contain 3 labels as used by Peoplenet. No other modifications to deepstream_imagedata-multistream.py were made.

Now, when I run the application it works well for a single stream, but when I add more than one stream, it crashes:

For a single stream

python3 mod_deepstream_imagedata-multistream.py rtsp_link1 frames

Frames will be saved in frames
Creating Pipeline

Creating streamux

Creating source_bin 0

Creating source bin
source-bin-00
Creating Pgie

Creating nvvidconv1

Creating filter1

Creating tiler

Creating nvvidconv

Creating nvosd

Is it Integrated GPU? : 1
Creating nv3dsink

Atleast one of the sources is live
WARNING: Overriding infer-config batch-size 30 with number of sources 1

Adding elements to Pipeline

Linking elements in the Pipeline

Now playing…
1 : rtsp_link1
Starting pipeline

Setting min object dimensions as 16x16 instead of 1x1 to support VIC compute mode.
WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
0:00:06.844761407 151 0xaaab4d54e6a0 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:2095> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-7.0/samples/models/tao_pretrained_models/peopleNet/resnet34_peoplenet_int8.engine
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT input_1:0 3x544x960

1 OUTPUT kFLOAT output_cov/Sigmoid:0 3x34x60

2 OUTPUT kFLOAT output_bbox/BiasAdd:0 12x34x60

ERROR: [TRT]: 3: Cannot find binding of given name: output_cov/Sigmoid
0:00:07.242181049 151 0xaaab4d54e6a0 WARN nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:2062> [UID = 1]: Could not find output layer ‘output_cov/Sigmoid’ in engine
ERROR: [TRT]: 3: Cannot find binding of given name: output_bbox/BiasAdd
0:00:07.242240026 151 0xaaab4d54e6a0 WARN nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:2062> [UID = 1]: Could not find output layer ‘output_bbox/BiasAdd’ in engine
0:00:07.242256379 151 0xaaab4d54e6a0 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2198> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-7.0/samples/models/tao_pretrained_models/peopleNet/resnet34_peoplenet_int8.engine
0:00:07.260719379 151 0xaaab4d54e6a0 INFO nvinfer gstnvinfer_impl.cpp:343:notifyLoadModelStatus: [UID 1]: Load new model:dstest_imagedata_config.txt sucessfully
Decodebin child added: source

  • *PERF: {‘stream0’: 0.0}

Decodebin child added: decodebin0

Decodebin child added: rtppcmudepay0

Decodebin child added: mulawdec0

In cb_newpad

Decodebin child added: decodebin1

Decodebin child added: rtph265depay0

  • *PERF: {‘stream0’: 0.0}

Decodebin child added: h265parse0

Decodebin child added: capsfilter0

Decodebin child added: nvv4l2decoder0

Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 279
NvMMLiteBlockCreate : Block : BlockType = 279
In cb_newpad

Frame Number= 0 Number of Objects= 5 Person_count= 3 Face_count= 2

  • *PERF: {‘stream0’: 0.0}

Frame Number= 1 Number of Objects= 5 Person_count= 3 Face_count= 2
Frame Number= 2 Number of Objects= 5 Person_count= 3 Face_count= 2
Frame Number= 3 Number of Objects= 5 Person_count= 3 Face_count= 2

For more than one stream:

python3 mod_deepstream_imagedata-multistream.py rtsp_link1 rtsp_link2 frames

Frames will be saved in frames
Creating Pipeline

Creating streamux

Creating source_bin 0

Creating source bin
source-bin-00
Creating source_bin 1

Creating source bin
source-bin-01
Creating Pgie

Creating nvvidconv1

Creating filter1

Creating tiler

Creating nvvidconv

Creating nvosd

Is it Integrated GPU? : 1
Creating nv3dsink

Atleast one of the sources is live
WARNING: Overriding infer-config batch-size 30 with number of sources 2

Adding elements to Pipeline

Linking elements in the Pipeline

Now playing…
1 : rtsp://10.226.52.222:554/h264/ch1/sub/av_stream
2 : rtsp://10.226.52.226:554/h264/ch1/sub/av_stream
Starting pipeline

Setting min object dimensions as 16x16 instead of 1x1 to support VIC compute mode.
WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
0:00:06.917391091 206 0xaaab216c1c70 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:2095> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-7.0/samples/models/tao_pretrained_models/peopleNet/resnet34_peoplenet_int8.engine
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT input_1:0 3x544x960

1 OUTPUT kFLOAT output_cov/Sigmoid:0 3x34x60

2 OUTPUT kFLOAT output_bbox/BiasAdd:0 12x34x60

0:00:07.311114935 206 0xaaab216c1c70 WARN nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:2027> [UID = 1]: Backend has maxBatchSize 1 whereas 2 has been requested
0:00:07.311192729 206 0xaaab216c1c70 WARN nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2204> [UID = 1]: deserialized backend context :/opt/nvidia/deepstream/deepstream-7.0/samples/models/tao_pretrained_models/peopleNet/resnet34_peoplenet_int8.engine failed to match config params, trying rebuild
0:00:07.325072343 206 0xaaab216c1c70 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2109> [UID = 1]: Trying to create engine from model files
WARNING: INT8 calibration file not specified/accessible. INT8 calibration can be done through setDynamicRange API in ‘NvDsInferCreateNetwork’ implementation
WARNING: [TRT]: The implicit batch dimension mode has been deprecated. Please create the network with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag whenever possible.
ERROR: [TRT]: UffParser: Could not read buffer.
parseModel: Failed to parse UFF model
ERROR: Failed to build network, error in model parsing.
ERROR: Failed to create network using custom network creation function
ERROR: Failed to get cuda engine from custom library API
0:00:14.360852900 206 0xaaab216c1c70 ERROR nvinfer gstnvinfer.cpp:676:gst_nvinfer_logger: NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2129> [UID = 1]: build engine file failed
Segmentation fault (core dumped)

Please let me know how I can fix this and run multiple streams simultaenously?

Did you regenerate the engine? You can try to delete the engine file that have generated and let nvinfer generate the engine file itself.

I deleted the engine file generated by trtexec and commented out the engine file path in the dstest_imagedata_config.txt and ran the python program, it still doesn’t work…

python3 mod_deepstream_imagedata-multistream.py rtsp_link1 rtsp_link2 frames

Frames will be saved in frames

Creating Pipeline

Creating streamux

Creating source_bin 0

Creating source bin
source-bin-00
Creating source_bin 1

Creating source bin
source-bin-01
Creating Pgie

Creating nvvidconv1

Creating filter1

Creating tiler

Creating nvvidconv

Creating nvosd

Is it Integrated GPU? : 1
Creating nv3dsink

Atleast one of the sources is live
WARNING: Overriding infer-config batch-size 30 with number of sources 2

Adding elements to Pipeline

Linking elements in the Pipeline

Now playing…
1 : rtsp_Link1
2 : rtsp_link2
Starting pipeline

Setting min object dimensions as 16x16 instead of 1x1 to support VIC compute mode.
0:00:00.411207135 237 0xaaaad494fec0 INFO nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2109> [UID = 1]: Trying to create engine from model files
WARNING: INT8 calibration file not specified/accessible. INT8 calibration can be done through setDynamicRange API in ‘NvDsInferCreateNetwork’ implementation
WARNING: [TRT]: The implicit batch dimension mode has been deprecated. Please create the network with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag whenever possible.
ERROR: [TRT]: UffParser: Could not read buffer.
parseModel: Failed to parse UFF model
ERROR: Failed to build network, error in model parsing.
ERROR: Failed to create network using custom network creation function
ERROR: Failed to get cuda engine from custom library API
0:00:09.488596443 237 0xaaaad494fec0 ERROR nvinfer gstnvinfer.cpp:676:gst_nvinfer_logger: NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2129> [UID = 1]: build engine file failed
Segmentation fault (core dumped)

Looks like it’s a model problem. How do you get the model?

TAO Toolkit Integration with DeepStream — DeepStream documentation 6.4 documentation

I referred the README in package /opt/nvidia/deepstream/deepstream/samples/configs/tao_pretrained_models/README.md to obtain TAO toolkit config files and models mentioned in the above link.

I then went on to the below location in our docker container and used the model file provided…

root@ubuntu:/opt/nvidia/deepstream/deepstream/samples/models/tao_pretrained_models/peopleNet# ls
labels.txt  nvinfer_config.txt  resnet34_peoplenet_int8.onnx  resnet34_peoplenet_int8.txt

Did you have the following regenerated engine prints while running one source?
You set the network-mode=1(int8), but you didn’t provide the int8 cal file in your config file.

No, I didn’t have the trying rebuild print messages while running a single stream.

Also, I tried to provide it with the int 8 calibration file by modifying the dstest_imagedata_config.txt

int8-calib-file=/opt/nvidia/deepstream/deepstream-7.0/samples/models/tao_pretrained_models/peopleNet/resnet34_peoplenet_int8.txt

But I still get errors and it doesn’t work.

  1. Multiple Streams, Using old engine built with trtexec, int8 calib file provided

Output

Frames will be saved in  frames

(gst-plugin-scanner:80): GStreamer-WARNING **: 09:38:13.904: adding type GstEvent multiple times
(Argus) Error FileOperationFailed: Connecting to nvargus-daemon failed: No such file or directory (in src/rpc/socket/client/SocketClientDispatch.cpp, function openSocketConnection(), line 204)
(Argus) Error FileOperationFailed: Cannot create camera provider (in src/rpc/socket/client/SocketClientDispatch.cpp, function createCameraProvider(), line 106)

(gst-plugin-scanner:80): GStreamer-WARNING **: 09:38:14.170: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so': libtritonserver.so: cannot open shared object file: No such file or directory

(gst-plugin-scanner:80): GStreamer-WARNING **: 09:38:14.172: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_udp.so': librivermax.so.0: cannot open shared object file: No such file or directory
Creating Pipeline 
 
Creating streamux 
 
Creating source_bin  0  
 
Creating source bin
source-bin-00
Creating source_bin  1  
 
Creating source bin
source-bin-01
Creating Pgie 
 
Creating nvvidconv1 
 
Creating filter1 
 
Creating tiler 
 
Creating nvvidconv 
 
Creating nvosd 
 
Is it Integrated GPU? : 1
Creating nv3dsink 

Atleast one of the sources is live
WARNING: Overriding infer-config batch-size 30  with number of sources  2  

Adding elements to Pipeline 

Linking elements in the Pipeline 

Now playing...
1 :  rtsp_link1
2 :  rtsp_link2
Starting pipeline 

Setting min object dimensions as 16x16 instead of 1x1 to support VIC compute mode.
WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
0:00:08.648696761    79 0xaaab6d194a70 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:2095> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-7.0/samples/models/tao_pretrained_models/peopleNet/resnet34_peoplenet_int8.engine
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: [Implicit Engine Info]: layers num: 3
0   INPUT  kFLOAT input_1:0       3x544x960       
1   OUTPUT kFLOAT output_cov/Sigmoid:0 3x34x60         
2   OUTPUT kFLOAT output_bbox/BiasAdd:0 12x34x60        

0:00:09.071230853    79 0xaaab6d194a70 WARN                 nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:2027> [UID = 1]: Backend has maxBatchSize 1 whereas 2 has been requested
0:00:09.071298661    79 0xaaab6d194a70 WARN                 nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2204> [UID = 1]: deserialized backend context :/opt/nvidia/deepstream/deepstream-7.0/samples/models/tao_pretrained_models/peopleNet/resnet34_peoplenet_int8.engine failed to match config params, trying rebuild
0:00:09.087429362    79 0xaaab6d194a70 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2109> [UID = 1]: Trying to create engine from model files
WARNING: [TRT]: The implicit batch dimension mode has been deprecated. Please create the network with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag whenever possible.
ERROR: [TRT]: UffParser: Could not read buffer.
parseModel: Failed to parse UFF model
ERROR: Failed to build network, error in model parsing.
ERROR: Failed to create network using custom network creation function
ERROR: Failed to get cuda engine from custom library API
0:00:15.693025236    79 0xaaab6d194a70 ERROR                nvinfer gstnvinfer.cpp:676:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2129> [UID = 1]: build engine file failed
Segmentation fault (core dumped)
  1. Multiple Streams, letting nvinfer generate the engine (didn’t provide model engine file), int8 calib file provided

Output

Frames will be saved in  frames

(gst-plugin-scanner:80): GStreamer-WARNING **: 09:49:23.214: adding type GstEvent multiple times
(Argus) Error FileOperationFailed: Connecting to nvargus-daemon failed: No such file or directory (in src/rpc/socket/client/SocketClientDispatch.cpp, function openSocketConnection(), line 204)
(Argus) Error FileOperationFailed: Cannot create camera provider (in src/rpc/socket/client/SocketClientDispatch.cpp, function createCameraProvider(), line 106)

(gst-plugin-scanner:80): GStreamer-WARNING **: 09:49:23.487: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so': libtritonserver.so: cannot open shared object file: No such file or directory

(gst-plugin-scanner:80): GStreamer-WARNING **: 09:49:23.489: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_udp.so': librivermax.so.0: cannot open shared object file: No such file or directory
Creating Pipeline 
 
Creating streamux 
 
Creating source_bin  0  
 
Creating source bin
source-bin-00
Creating source_bin  1  
 
Creating source bin
source-bin-01
Creating Pgie 
 
Creating nvvidconv1 
 
Creating filter1 
 
Creating tiler 
 
Creating nvvidconv 
 
Creating nvosd 
 
Is it Integrated GPU? : 1
Creating nv3dsink 

Atleast one of the sources is live
WARNING: Overriding infer-config batch-size 30  with number of sources  2  

Adding elements to Pipeline 

Linking elements in the Pipeline 

Now playing...
1 :  rtsp_link1
2 :  rtsp_link2
Starting pipeline 

Setting min object dimensions as 16x16 instead of 1x1 to support VIC compute mode.
0:00:01.594154368    79 0xaaaaf801e3d0 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2109> [UID = 1]: Trying to create engine from model files
WARNING: [TRT]: The implicit batch dimension mode has been deprecated. Please create the network with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag whenever possible.
ERROR: [TRT]: UffParser: Could not read buffer.
parseModel: Failed to parse UFF model
ERROR: Failed to build network, error in model parsing.
ERROR: Failed to create network using custom network creation function
ERROR: Failed to get cuda engine from custom library API
0:00:08.949714405    79 0xaaaaf801e3d0 ERROR                nvinfer gstnvinfer.cpp:676:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:2129> [UID = 1]: build engine file failed
Segmentation fault (core dumped)

In my above reply where I am using the engine built with trtexec, I still got the below again even though I have now provided the int8 calib file. Is it because of some other parameter conflict? If so, then what parameter could be behind the issue?

Please refer to our demo code config_infer_primary_peoplenet.txt.
If you are using onnx model, you should not use the tlt-model-key and tlt-encoded-model parameters.

I think I figured out what the main problem was. The model engine built using trtexec was, unbeknownst to me, being built for batch sizes of 1, which I guess corresponds to the number of streams it can handle? I went through the forums and found this github resource which helped me in figuring out that I can rebuild the engine to handle multiple streams, say 6 streams by recreating the model engine through a command like this:

trtexec --onnx=/opt/nvidia/deepstream/deepstream-7.0/samples/models/tao_pretrained_models/peopleNet/resnet34_peoplenet_int8.onnx --int8 \
--calib=/opt/nvidia/deepstream/deepstream-7.0/samples/models/tao_pretrained_models/peopleNet/resnet34_peoplenet_int8.txt --saveEngine=/opt/nvidia/deepstream/deepstream-7.0/samples/models/tao_pretrained_models/peopleNet/resnet34_peoplenet_int8.onnx_b6_gpu0_int8.engine \
--minShapes="input_1:0":6x3x544x960 --optShapes="input_1:0":6x3x544x960 --maxShapes="input_1:0":6x3x544x960&

I then modified these parameters in the dstest_imagedata_config.txt as found in the deepstream-imagedata-multistream python application folder.

model-engine-file=/opt/nvidia/deepstream/deepstream-7.0/samples/models/tao_pretrained_models/peopleNet/resnet34_peoplenet_int8.onnx_b6_gpu0_int8.engine
# int8-calib-file=../../../../samples/models/Primary_Detector/cal_trt.bin
batch-size=6

And it worked. I could now simultaneously run 6 streams.

Thanks for pointing me in the right direction of it being due to a model problem and some parameter conflict!

Now though, I am facing an issue when I’m running more than a certain number of streams. When I’m trying to run 10 streams on my Jetson Orin Nano Developer Kit, it’s not working most of the times. It is inconsistent, as I ran the program about 10 times, and only once has it been able to properly initialize and show inference on all 10 streams. Even then the avg fps of those 10 streams was about 10 fps. Other times, half the streams don’t even appear on the screen, and the ones that do are just a static images of the moment I started the program as shown in the picture where 9 streams didn’t even appear and the 1 that did, is just a static image.

Jetson should be able to offer more performance than this, right, considering the online benchmarks of Jetson Orin Nano are way better than this…

Here’s the output I receive on the terminal:

Frames will be saved in  frames

(gst-plugin-scanner:76): GStreamer-WARNING **: 05:51:35.447: adding type GstEvent multiple times
(Argus) Error FileOperationFailed: Connecting to nvargus-daemon failed: No such file or directory (in src/rpc/socket/client/SocketClientDispatch.cpp, function openSocketConnection(), line 204)
(Argus) Error FileOperationFailed: Cannot create camera provider (in src/rpc/socket/client/SocketClientDispatch.cpp, function createCameraProvider(), line 106)

(gst-plugin-scanner:76): GStreamer-WARNING **: 05:51:35.726: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so': [libtritonserver.so](http://libtritonserver.so/): cannot open shared object file: No such file or directory

(gst-plugin-scanner:76): GStreamer-WARNING **: 05:51:35.728: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_udp.so': librivermax.so.0: cannot open shared object file: No such file or directory
Creating Pipeline

Creating streamux

Creating source_bin  0

Creating source bin
source-bin-00
Creating source_bin  1

Creating source bin
source-bin-01
Creating source_bin  2

Creating source bin
source-bin-02
Creating source_bin  3

Creating source bin
source-bin-03
Creating source_bin  4

Creating source bin
source-bin-04
Creating source_bin  5

Creating source bin
source-bin-05
Creating source_bin  6

Creating source bin
source-bin-06
Creating source_bin  7

Creating source bin
source-bin-07
Creating source_bin  8

Creating source bin
source-bin-08
Creating source_bin  9

Creating source bin
source-bin-09
Creating Pgie

Creating nvvidconv1

Creating filter1

Creating tiler

Creating nvvidconv

Creating nvosd

Is it Integrated GPU? : 1
Creating nv3dsink

Atleast one of the sources is live
Adding elements to Pipeline

Linking elements in the Pipeline

Now playing...
1 :  rtsp_link
2 :  rtsp_link
3 :  rtsp_link
4 :  rtsp_link
5 :  rtsp_link
6 :  rtsp_link
7 :  rtsp_link
8 :  rtsp_link
9 :  rtsp_link
10 :  rtsp_link
Starting pipeline

Setting min object dimensions as 16x16 instead of 1x1 to support VIC compute mode.
WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
0:00:08.391206182    75 0xaaab1d9b4c50 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:2095> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-7.0/samples/models/tao_pretrained_models/peopleNet/resnet34_peoplenet_int8.onnx_b10_gpu0_int8.engine
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: [Implicit Engine Info]: layers num: 3
0   INPUT  kFLOAT input_1:0       3x544x960

1   OUTPUT kFLOAT output_cov/Sigmoid:0 3x34x60

2   OUTPUT kFLOAT output_bbox/BiasAdd:0 12x34x60

ERROR: [TRT]: 3: Cannot find binding of given name: output_cov/Sigmoid
0:00:08.828073461    75 0xaaab1d9b4c50 WARN                 nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:2062> [UID = 1]: Could not find output layer 'output_cov/Sigmoid' in engine
ERROR: [TRT]: 3: Cannot find binding of given name: output_bbox/BiasAdd
0:00:08.828122965    75 0xaaab1d9b4c50 WARN                 nvinfer gstnvinfer.cpp:679:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:2062> [UID = 1]: Could not find output layer 'output_bbox/BiasAdd' in engine
0:00:08.834539544    75 0xaaab1d9b4c50 INFO                 nvinfer gstnvinfer.cpp:682:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2198> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-7.0/samples/models/tao_pretrained_models/peopleNet/resnet34_peoplenet_int8.onnx_b10_gpu0_int8.engine
0:00:08.873626035    75 0xaaab1d9b4c50 INFO                 nvinfer gstnvinfer_impl.cpp:343:notifyLoadModelStatus:<primary-inference> [UID 1]: Load new model:dstest_imagedata_config.txt sucessfully
Decodebin child added: source

Decodebin child added: source

Decodebin child added: source

Decodebin child added: source

Decodebin child added: source

Decodebin child added: source

Decodebin child added: source

Decodebin child added: source

Decodebin child added: source

Decodebin child added: source

- *PERF: {'stream0': 0.0, 'stream1': 0.0, 'stream2': 0.0, 'stream3': 0.0, 'stream4': 0.0, 'stream5': 0.0, 'stream6': 0.0, 'stream7': 0.0, 'stream8': 0.0, 'stream9': 0.0}

Decodebin child added: decodebin0

Decodebin child added: rtppcmudepay0

Decodebin child added: mulawdec0

In cb_newpad

Decodebin child added: decodebin1

Decodebin child added: rtppcmudepay1

Decodebin child added: mulawdec1

In cb_newpad

Decodebin child added: decodebin2

Decodebin child added: rtph265depay0

Decodebin child added: h265parse0

Decodebin child added: capsfilter0

Decodebin child added: nvv4l2decoder0

Decodebin child added: decodebin3

Decodebin child added: decodebin4
Opening in BLOCKING MODE

Decodebin child added: decodebin5

Decodebin child added: decodebin6

NvMMLiteOpen : Block : BlockType = 279
Decodebin child added: rtppcmudepay2

Decodebin child added: rtph265depay2

Decodebin child added: rtph265depay1
NvMMLiteBlockCreate : Block : BlockType = 279

Decodebin child added: rtph265depay3

Decodebin child added: mulawdec2

In cb_newpad

Decodebin child added: decodebin7

Decodebin child added: rtppcmudepay3

Decodebin child added: mulawdec3

In cb_newpad

Decodebin child added: decodebin8

Decodebin child added: rtph265depay4

Decodebin child added: decodebin9

Decodebin child added: rtph265depay5

Decodebin child added: h265parse1

Decodebin child added: capsfilter1

Decodebin child added: nvv4l2decoder1

Decodebin child added: decodebin10

Decodebin child added: rtph265depay6

Decodebin child added: decodebin12

Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 279
Decodebin child added: decodebin11
NvMMLiteBlockCreate : Block : BlockType = 279

Decodebin child added: rtppcmudepay5

Decodebin child added: rtppcmudepay4

Decodebin child added: decodebin13

Decodebin child added: mulawdec4

Decodebin child added: mulawdec5

In cb_newpad

Decodebin child added: h265parse2

Decodebin child added: capsfilter2

Decodebin child added: nvv4l2decoder2

In cb_newpad
Decodebin child added: rtppcmudepay6

Decodebin child added: mulawdec6

In cb_newpad
Opening in BLOCKING MODE
In cb_newpad

NvMMLiteOpen : Block : BlockType = 279
Decodebin child added: decodebin14

NvMMLiteBlockCreate : Block : BlockType = 279
Decodebin child added: rtph265depay7

Decodebin child added: decodebin15

Decodebin child added: rtppcmudepay7

Decodebin child added: mulawdec7

In cb_newpad

In cb_newpad

In cb_newpad

Frame Number= 0 Number of Objects= 0 Person_count= 0 Face_count= 0
Decodebin child added: decodebin16

Decodebin child added: decodebin17

Decodebin child added: rtph265depay8

Decodebin child added: rtph265depay9

Decodebin child added: decodebin18

Decodebin child added: rtppcmudepay8

Decodebin child added: mulawdec8

In cb_newpad

Decodebin child added: decodebin19

Decodebin child added: rtppcmudepay9

Decodebin child added: mulawdec9

In cb_newpad

- *PERF: {'stream0': 0.0, 'stream1': 0.0, 'stream2': 0.0, 'stream3': 0.0, 'stream4': 0.0, 'stream5': 0.0, 'stream6': 0.0, 'stream7': 0.0, 'stream8': 0.0, 'stream9': 0.0}

Decodebin child added: h265parse3

Decodebin child added: capsfilter3

Decodebin child added: nvv4l2decoder3

Decodebin child added: h265parse4

Opening in BLOCKING MODE
Decodebin child added: capsfilter4

NvMMLiteOpen : Block : BlockType = 279
NvMMLiteBlockCreate : Block : BlockType = 279
Decodebin child added: nvv4l2decoder4

Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 279
NvMMLiteBlockCreate : Block : BlockType = 279
Decodebin child added: h265parse5

Decodebin child added: capsfilter5

Decodebin child added: nvv4l2decoder5

Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 279
NvMMLiteBlockCreate : Block : BlockType = 279
In cb_newpad

In cb_newpad

In cb_newpad

Decodebin child added: h265parse6

Decodebin child added: capsfilter6

Decodebin child added: nvv4l2decoder6

Decodebin child added: h265parse7

Decodebin child added: capsfilter7

Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 279
Decodebin child added: nvv4l2decoder7

NvMMLiteBlockCreate : Block : BlockType = 279
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 279
NvMMLiteBlockCreate : Block : BlockType = 279
In cb_newpad

In cb_newpad

- *PERF: {'stream0': 0.0, 'stream1': 0.0, 'stream2': 0.0, 'stream3': 0.0, 'stream4': 0.0, 'stream5': 0.0, 'stream6': 0.0, 'stream7': 0.0, 'stream8': 0.0, 'stream9': 0.0}
- *PERF: {'stream0': 0.0, 'stream1': 0.0, 'stream2': 0.0, 'stream3': 0.0, 'stream4': 0.0, 'stream5': 0.0, 'stream6': 0.0, 'stream7': 0.0, 'stream8': 0.0, 'stream9': 0.0}
- *PERF: {'stream0': 0.0, 'stream1': 0.0, 'stream2': 0.0, 'stream3': 0.0, 'stream4': 0.0, 'stream5': 0.0, 'stream6': 0.0, 'stream7': 0.0, 'stream8': 0.0, 'stream9': 0.0}
- *PERF: {'stream0': 0.0, 'stream1': 0.0, 'stream2': 0.0, 'stream3': 0.0, 'stream4': 0.0, 'stream5': 0.0, 'stream6': 0.0, 'stream7': 0.0, 'stream8': 0.0, 'stream9': 0.0}
- *PERF: {'stream0': 0.0, 'stream1': 0.0, 'stream2': 0.0, 'stream3': 0.0, 'stream4': 0.0, 'stream5': 0.0, 'stream6': 0.0, 'stream7': 0.0, 'stream8': 0.0, 'stream9': 0.0}

^C
**PERF:  {'stream0': 0.0, 'stream1': 0.0, 'stream2': 0.0, 'stream3': 0.0, 'stream4': 0.0, 'stream5': 0.0, 'stream6': 0.0, 'stream7': 0.0, 'stream8': 0.0, 'stream9': 0.0}

Exiting app

NVMMLITE_NVVIDEODEC, <cbAllocPictureBuffer, 5003> ErrorInfo = VideoErrorInfo_OutputBufferUnavailable cctx = 0xc41461f0
NVMMLITE_NVVIDEODEC, <cbAllocPictureBuffer, 5003> ErrorInfo = VideoErrorInfo_OutputBufferUnavailable cctx = 0xbc17b250
NVMMLITE_NVVIDEODEC, <cbAllocPictureBuffer, 5003> ErrorInfo = VideoErrorInfo_OutputBufferUnavailable cctx = 0x3c14e4b0
NVMMLITE_NVVIDEODEC, <cbAllocPictureBuffer, 5003> ErrorInfo = VideoErrorInfo_OutputBufferUnavailable cctx = 0xb8134190
NVMMLITE_NVVIDEODEC, <cbAllocPictureBuffer, 5003> ErrorInfo = VideoErrorInfo_OutputBufferUnavailable cctx = 0x5414c900
NVMMLITE_NVVIDEODEC, <cbAllocPictureBuffer, 5003> ErrorInfo = VideoErrorInfo_OutputBufferUnavailable cctx = 0x4c1457d0

What can I do to rectify this? Is there some way to improve the performance, so as to run more streams, more smoothly?

  1. boost-the-clocks
  2. You can set the interval for the nvinfer, like interval=10

I had already boosted the clocks before. I modified the interval = 0 to interval = 10 in my config file but I am still facing the earlier issue…

You can use tegrastats to observe the loading of your board. If the loading is too heavy, there will be some performance problems.

The thing is when the above application is running and the above problem crops up, the jtop stats i see fluctuate heavily between 0-50% GPU Usage, which in other cases when the application runs properly is comparatively quite stable and fluctuates at max between a range of about +/- 5% about its central stable value.

Screenshot from 2024-06-14 17-27-00

It doesn’t seem like the loading is too heavy, but still just to make sure I built and ran the same application on a Jetson AGX Orin 64GB Developer Kit which should easily be able to handle a load of about 10 streams running peoplenet but it faced the same issue with half the screens not loading and the screens that do load are static in nature…

In fact, I even noticed that this happens sometimes even when there are less number of streams for example I noticed that this error with not all screens initializing and some screens being static occurred when I ran the application with 6 streams as well…