DeepStream Python Parallel Inference Pipeline Freezes

[nohara]

Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU) Tesla T4
• DeepStream Version 6.2 (docker image)
• TensorRT Version 8.5.2
• NVIDIA GPU Driver Version (valid for GPU only) 525.85.12

Hello, I’m constructing a DeepStream parallel inference pipeline in Python. The pipeline has 2 branches:
branch1: trafficcamnet → tracker → vehiclecolor → vehiclemake → vehicletype
branch 2: trafficcamnet → lpd

I was able to run these branches individually before combining them to a single pipeline.

Here’s my code:
ds_pipeline.py (15.2 KB)

Here are my config files:
branch1-pgie1-trafficcamnet.txt (3.0 KB)
branch1-sgie1-vehiclecolor.txt (3.2 KB)
branch1-sgie2-vehiclemake.txt (2.8 KB)
branch1-sgie3-vehicletype.txt (837 Bytes)
branch2-pgie1-trafficcamnet.txt (3.0 KB)
branch2-sgie1-lpd.txt (1.1 KB)

Here’s my pipeline log:
log-pipeline.txt (57.0 KB)

Pipeline graph:

I’m currently routing 1 video to branch 1 and 1 video to branch 2 for debugging purposes. In future, I intend to route specific set of video streams to each branch.

Problem: The pipeline freezes. I don’t think generating model engines should take more than 30 minutes.

[yuweiw]

Just from the log, it appears that the engine should have been generated, but it cannot be confirmed which plugin is frozen. Could you help to debug this with the mothod below?
Using fakesink to replace plugins from back to front to location which plugin is frozen. Thanks

[nohara]

Thanks for the brilliant idea!

I was able to run the branches individually after commenting out plugins of the other branch.
This worked:

This worked:

This doesn’t work:

Pipeline freezes at branch1-pgie1-trafficcamnet.txt. (All other nvinfers were successfully initialized)
PS. I am using the same config files for both branch1-pgie1-trafficcamnet.txt and branch2-pgie1-trafficcamnet.txt. The only difference between these config files is the gie-unique-id.

Few observations:

1. I have been ignoring these all this time. Please let me know if something needs to be fixed to resolve these warnings and errors.

WARNING: [TRT]: CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
WARNING: [TRT]: CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
ERROR: [TRT]: 3: [builder.cpp::~Builder::307] Error Code 3: API Usage Error (Parameter check failed at: optimizer/api/builder.cpp::~Builder::307, condition: mObjectCounter.use_count() == 1. Destroying a builder object before destroying objects it created leads to undefined behavior.

2. Even though I have provided the location for lpd engine in the config file, DS generates the lpd engine once again and it takes very long time (maybe 20-30 mins).

[nohara]

Hello @yuweiw, I would like to kindly follow up on my previous request. Could you please provide an update or any information regarding the status of this issue?

[yuweiw]

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

@Nohara

I have been ignoring these all this time. Please let me know if something needs to be fixed to resolve these warnings and errors.

This has no impact on the results. We’ll remove these prints in the new version.

@nohara

Even though I have provided the location for lpd engine in the config file, DS generates the lpd engine once again and it takes very long time (maybe 20-30 mins).

This is because you did not configure the following fields: model-engine-file.

About the freeze issue, could you use our config file and models in the following project with your python code to have a try? We need a reproducible environment to locate that.
https://github.com/NVIDIA-AI-IOT/deepstream_parallel_inference_app