Confirm two things.
1.
If I’m not mistaken, deepstream-test2 works fine using config_tracker_NvDCF_perf.yml if the TensorRT error is ignored?
2.If the answer to the above is yes, then the problem seems to be caused by python bindings.
Run the following command line to confirm the version of pyds. DS-6.4 should be 1.1.10
pip list | grep pyds
Hello
For config_tracker_NvDCF_perf.yml :
Works perfectly.
This seems logical given that this config file does not load any ReId model.
Here is what I get :
pyds 1.1.10
Note : I am installing pyds using this command :
pip install https://github.com/NVIDIA-AI-IOT/deepstream_python_apps/releases/download/v1.1.10/pyds-1.1.10-py3-none-linux_x86_64.whl
I found the same issue on a seperate project, spend a lot of time on it, waiting for a solution.
RTX3090
DS 6.4
CUDA 12.2
NVIDIA-SMI 535.154.05
python 3.10.12
pyds 1.1.10
In my case, the config_tracker_xxx.yml, ReID session does not support a .engine file. In your case, the second time failed due to the first time was based on an onnx or ‘uff’ file. This process will create an .engine file. The message is:
[NvMultiObjectTracker] Load engine failed. Create engine again.
[NvMultiObjectTracker] Begin building engine for tracker ReID…
[NvMultiObjectTracker] Serialized plan file cached at location: /path/model.onnx_b100_gpu0_fp16.engine
[NvMultiObjectTracker] Initialized
The second run it found the .engine file path was exist, tried to use it, and failed. Message was:
[NvMultiObjectTracker] Loading TRT Engine for tracker ReID…
[NvMultiObjectTracker] Loading Complete!
Segmentation fault (core dumped)
This issue did not exist in 6.3. In 6.3, if successful, it gives:
[NvMultiObjectTracker] Loading TRT Engine for tracker ReID…
[NvMultiObjectTracker] Loading Complete!
[NvMultiObjectTracker] Initialized
This should also not related to the tensorRT version, as I tried to provided an onnx for nvinfer, it creates an engine file as well, and is functional. This issue only exists in tracker ReID section, both for DeepSort and DCF.
The only ‘solution’ I found was to comment out modelEngineFile, it means everytime I run my project, a lot time had to spend on generating a useless engine file, definitely this is not what I wanted.
I suspect a bug is in initializing/loading engine file in ReID module, using onnx or uff, there was no engine loading process, only converting. Loading engine model in ReID is different with loading engine models in nvinfer module. In nvinfer model, I got a message:
NvDsInferContext[UID 2]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2195> [UID = 2]: Use deserialized engine model: /path/model.engine
while there is no such message for loading engine file in ReID (in 6.3) if engine in ReID was successfully loaded.
Hope I can get some help on solving this issue
I can see we have the exact same issue…
I am also looking for help from Nvidia cause I can’t build an engine everytime. Especially in production.
@chen8131928 You can open a new topic to discuss your problem
docker run --gpus all -it --rm --net=host --privileged -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY -w /opt/nvidia/deepstream/deepstream nvcr.io/nvidia/deepstream:6.4-triton-multiarch
I use T4 GPU and the software version is as follows
deepstream-app --version-all
deepstream-app version 6.4.0
DeepStreamSDK 6.4.0
CUDA Driver Version: 12.2
CUDA Runtime Version: 12.2
TensorRT Version: 8.6
cuDNN Version: 8.9
libNVWarp360 Version: 2.0.1d3
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.86.10 Driver Version: 535.86.10 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:03:00.0 Off | 0 |
| N/A 30C P8 9W / 70W | 485MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
2.Currently reid does not support the onnx model, so deployable_v1.2 cannot be used currently. You can use deployable_v1.1.
and my tests are based on v1.1.
So I don’t know how you use the onnx model.
3.Modify deepstream_test_2.py to use config_tracker_NvDCF_accuracy.yml (with reid v1.1 model)
I haven’t encountered the problem you mentioned.
4.Please help to use the following command to grab the crash stack.
gdb -ex run --args python3 deepstream_test_2.py /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264
Following your steps :
deepstream-app --version-all :
deepstream-app version 6.4.0
DeepStreamSDK 6.4.0
CUDA Driver Version: 12.2
CUDA Runtime Version: 12.2
TensorRT Version: 8.6
cuDNN Version: 8.9
libNVWarp360 Version: 2.0.1d3
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.154.05 Driver Version: 535.154.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 3060 ... Off | 00000000:01:00.0 On | N/A |
| N/A 51C P5 13W / 60W | 129MiB / 6144MiB | 11% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
Using deployable_v1.1 and gdb -ex run --args python3 deepstream_test_2.py /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264 with config_tracker_NvDCF_accuracy.yml :
1st execution : can successfully create engine and start pipeline (by the way, I don’t know why it uses the onnx2trt_utils.cpp code since I supply an etlt) :
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
~~ CLOG[src/modules/ReID/ReID.cpp, loadTRTEngine() @line 598]: Engine file does not exist
[NvMultiObjectTracker] Load engine failed. Create engine again.
WARNING: [TRT]: onnx2trt_utils.cpp:374: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
WARNING: ../nvdsinfer/nvdsinfer_model_builder.cpp:1208 INT8 calibration file not specified. Trying FP16 mode.
WARNING: [TRT]: TensorRT encountered issues when converting weights between types and that could affect accuracy.
WARNING: [TRT]: If this is not the desired behavior, please modify the weights or retrain with regularization to adjust the magnitude of the weights.
WARNING: [TRT]: Check verbose logs for the list of affected weights.
WARNING: [TRT]: - 90 weights are affected by this issue: Detected subnormal FP16 values.
WARNING: [TRT]: - 108 weights are affected by this issue: Detected values less than smallest positive FP16 subnormal value and converted them to the FP16 minimum subnormalized value.
[NvMultiObjectTracker] Serialized plan file cached at location: /opt/nvidia/deepstream/deepstream-6.4/sources/deepstream_python_apps/apps/deepstream-test2/resnet50_market1501_aicity156.etlt_b100_gpu0_fp16.engine
[NvMultiObjectTracker] Initialized
2nd execution (engine previously built) : error :
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
~~ CLOG[src/modules/ReID/ReID.cpp, loadTRTEngine() @line 598]: Engine file does not exist
[NvMultiObjectTracker] Load engine failed. Create engine again.
Thread 1 "python3" received signal SIGSEGV, Segmentation fault.
0x00007ffff65ae550 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Here is what’s happening when I try to run : gdb -ex run --args python3 deepstream_nvdsanalytics.py file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264 for the second time (first execution works perfectly fine) (following your steps with deployable_v1.1):
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
[NvMultiObjectTracker] Loading TRT Engine for tracker ReID...
[NvMultiObjectTracker] Loading Complete!
terminate called after throwing an instance of 'nvinfer1::APIUsageError'
what(): Parameter check failed at: runtime/rt/runtime.cpp::~Runtime::346, condition: mEngineCounter.use_count() == 1. Destroying a runtime before destroying deserialized engines created by the runtime leads to undefined behavior.
Thread 1 "python3" received signal SIGABRT, Aborted.
0x00007ffff7ce59fc in pthread_kill () from /usr/lib/x86_64-linux-gnu/libc.so.6
If you can’t get the crash stack using python, can you help me test it using the c version deepstream-test2?
gdb --args ./deepstream-test2-app /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264
When compiling deepstream-test2-app, please add the -g option in Makefile.
%.o: %.c $(INCS) Makefile
$(CC) -c -g -o $@ $(CFLAGS) $<
By the way, have you upgraded gcc? What is the output of the following command ？
ls -l /usr/lib/x86_64-linux-gnu/libstdc++.so.6
@junshengy I have noticed that T4 does not have this issue from other threads (Result Correction Issue with NvTracker in the DeepStream 6.4 Environment), while here 3060 and 3090 has the identical engine file issue, would it possible that only RTX30x0 cause this problem? I do not have a T4 unfortunately, Would it be possible that you test it in a RTX30X0?
As I said in my post (the first one) :
This error doesn’t seem to prevent the pipeline from starting up for the sample:
/opt/nvidia/deepstream/deepstream-6.4/sources/apps/sample_apps/deepstream-test2
But this seems to be a problem when using python.
Here is the log for that sample (regarding the tracker nvdcf_accuracy):
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
[NvMultiObjectTracker] Loading TRT Engine for tracker ReID...
[NvMultiObjectTracker] Loading Complete!
ERROR: [TRT]: 3: [runtime.cpp::~Runtime::346] Error Code 3: API Usage Error (Parameter check failed at: runtime/rt/runtime.cpp::~Runtime::346, condition: mEngineCounter.use_count() == 1. Destroying a runtime before destroying deserialized engines created by the runtime leads to undefined behavior.
)
Execution in c appears to continue without interruption. But this error could be the cause of the probeleme encountered in python…
Here is the result for ls -l /usr/lib/x86_64-linux-gnu/libstdc++.so.6 :
lrwxrwxrwx 1 root root 19 May 13 2023 /usr/lib/x86_64-linux-gnu/libstdc++.so.6 -> libstdc++.so.6.0.30
I think I know what’s going on, I’ve reproduced the issue. Since pyds has a lot of dependencies, just installing *.whl is not enough.
If you are using docker
cd /opt/nvidia/deepstream/deepstream
./user_deepstream_python_apps_install.sh --build-bindings - r master
This script will help you install all dependencies and install pyds correctly.
Just installing whl, I get crash
Hi
Thanks,
Can this error be ignored ? :
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
[NvMultiObjectTracker] Loading TRT Engine for tracker ReID...
[NvMultiObjectTracker] Loading Complete!
ERROR: [TRT]: 3: [runtime.cpp::~Runtime::346] Error Code 3: API Usage Error (Parameter check failed at: runtime/rt/runtime.cpp::~Runtime::346, condition: mEngineCounter.use_count() == 1. Destroying a runtime before destroying deserialized engines created by the runtime leads to
undefined behavior.
)
I’m now trying to integrate these changes into my custom Deepstream application.
I no longer have the error in the samples but I still have the problem in my custom application…
Here is an image of the pipeline:
Here is the backtrace :
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so [476/9116]
[NvMultiObjectTracker] Loading TRT Engine for tracker ReID...
[NvMultiObjectTracker] Loading Complete!
Thread 1 "python3" received signal SIGABRT, Aborted.
0x00007ffff7ce59fc in pthread_kill () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) backtrace
#0 0x00007ffff7ce59fc in pthread_kill () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffff7c91476 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ffff7c777f3 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007ffff2baa692 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007ffff2bb589f in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007ffff2bb59c7 in __gxx_personality_v0 () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00007ffff5dbafe9 in __libunwind_Unwind_Resume () from /lib/x86_64-linux-gnu/libunwind.so.8
#7 0x00007fff8da2cea8 in ?? () from /lib/x86_64-linux-gnu/libnvinfer.so.8
#8 0x00007fff8e9bbaf2 in ?? () from /lib/x86_64-linux-gnu/libnvinfer.so.8
#9 0x00007fff8e9bfa72 in ?? () from /lib/x86_64-linux-gnu/libnvinfer.so.8
#10 0x00007fff244752f2 in ReID::loadTRTEngine() () from /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
#11 0x00007fff2447a984 in ReID::initializeModel() () from /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
#12 0x00007fff24469c00 in EnsembleReID::initializeModel() () from /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
#13 0x00007fff243d3298 in NvMultiObjectTrackerBase::initialize(NvMotDataTypes::Size, unsigned long) () from /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
#14 0x00007fff243f7cfe in NvMOTContext::NvMOTContext(_NvMOTConfig const&, _NvMOTConfigResponse&) () from /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
#15 0x00007fff243f8adf in NvMOT_Init () from /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
#16 0x00007fff9ff5caff in NvTrackerProc::initTrackerContext(unsigned int) () from /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_tracker.so
#17 0x00007fff9ff5fe32 in NvTrackerProc::init(TrackerConfig const&) () from /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_tracker.so
#18 0x00007fff9ff5594b in ?? () from /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_tracker.so
#19 0x00007fffb73c3441 in ?? () from /lib/x86_64-linux-gnu/libgstbase-1.0.so.0
#20 0x00007fffb73c3675 in ?? () from /lib/x86_64-linux-gnu/libgstbase-1.0.so.0
#21 0x00007fffb781373f in ?? () from /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#22 0x00007fffb7813fec in gst_pad_set_active () from /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#23 0x00007fffb77fb155 in ?? () from /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#24 0x00007fffb780436b in gst_iterator_fold () from /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#25 0x00007fffb787229a in ?? () from /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#26 0x00007fffb77f8a56 in ?? () from /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#27 0x00007fffb77fb410 in ?? () from /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#28 0x00007fffb77fa729 in gst_element_change_state () from /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#29 0x00007fffb77fae35 in ?? () from /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#30 0x00007fffb77d37ec in ?? () from /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#31 0x00007fffb7826796 in ?? () from /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#32 0x00007fffb77fa729 in gst_element_change_state () from /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#33 0x00007fffb77fa76f in gst_element_change_state () from /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#34 0x00007fffb77fae35 in ?? () from /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#35 0x00007ffff7623e2e in ?? () from /lib/x86_64-linux-gnu/libffi.so.8
#36 0x00007ffff7620493 in ?? () from /lib/x86_64-linux-gnu/libffi.so.8
#37 0x00007ffff782f722 in ?? () from /usr/lib/python3/dist-packages/gi/_gi.cpython-310-x86_64-linux-gnu.so
#38 0x00007ffff782d826 in ?? () from /usr/lib/python3/dist-packages/gi/_gi.cpython-310-x86_64-linux-gnu.so
#39 0x00007ffff782da7d in ?? () from /usr/lib/python3/dist-packages/gi/_gi.cpython-310-x86_64-linux-gnu.so
#40 0x00005555556a4a7b in _PyObject_MakeTpCall ()
#41 0x000055555569d629 in _PyEval_EvalFrameDefault ()
#42 0x00005555556ae9fc in _PyFunction_Vectorcall ()
#43 0x000055555569745c in _PyEval_EvalFrameDefault ()
#44 0x00005555556ae9fc in _PyFunction_Vectorcall ()
#45 0x000055555569745c in _PyEval_EvalFrameDefault ()
#46 0x00005555556ae9fc in _PyFunction_Vectorcall ()
#47 0x000055555569726d in _PyEval_EvalFrameDefault ()
#48 0x00005555556939c6 in ?? ()
#49 0x0000555555789256 in PyEval_EvalCode ()
--Type <RET> for more, q to quit, c to continue without paging--RET
#50 0x00005555557b4108 in ?? ()
#51 0x00005555557ad9cb in ?? ()
#52 0x00005555557b3e55 in ?? ()
#53 0x00005555557b3338 in _PyRun_SimpleFileObject ()
#54 0x00005555557b2f83 in _PyRun_AnyFileObject ()
#55 0x00005555557a5a5e in Py_RunMain ()
#56 0x000055555577c02d in Py_BytesMain ()
#57 0x00007ffff7c78d90 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#58 0x00007ffff7c78e40 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#59 0x000055555577bf25 in _start ()
Thanks for your help!
I think there is no problem with the pipeline, can you share sample code so I can reproduce the problem ？
Thank you for your help.
Here is a folder to reproduce the error.
Here are the instructions:
cd $PATH_FOLDERdocker build . -t trackerdocker run --gpus all -it --rm --net=host --privileged -v /tmp/.X11-unix:/tmp/.X11-unix -v ./deepstream-python:/opt/nvidia/deepstream/deepstream/deepstream-python -e DISPLAY=$DISPLAY -w /opt/nvidia/deepstream/deepstream/deepstream-python/deepstream trackerdocker exec -it container_name bashpython3 run.py 'rtsp://user:pass@a.b.c.d/onvif-media/media.amp'python3 run.py 'rtsp://user:pass@a.b.c.d/onvif-media/media.amp'The reid model should encounter an error when loading.
Most of the time, the logs will indicate :
[NvMultiObjectTracker] Loading TRT Engine for tracker ReID...
[NvMultiObjectTracker] Loading Complete!
Segmentation fault (core dumped)
Run the code several times, and after about ~30 attempts, with a bit of luck, you should get the following error:
[NvMultiObjectTracker] Loading TRT Engine for tracker ReID...
[NvMultiObjectTracker] Loading Complete!
terminate called after throwing an instance of 'nvinfer1::APIUsageError'
what(): Parameter check failed at: runtime/rt/runtime.cpp::~Runtime::346, condition: mEngineCounter.use_count() == 1. Destroying a runtime before destroying deserialized engines created by the runtime leads to undefined behavior.
Aborted (core dumped)
This error message is the same as the one obtained in the samples (but which can apparently be ignored when obtained in the samples). Unfortunately, in this case, this error causes the application to stop…
Note that the error is nevertheless presented slightly differently:
Custom app :
terminate called after throwing an instance of 'nvinfer1::APIUsageError'
what(): Parameter check failed at: runtime/rt/runtime.cpp::~Runtime::346, condition: mEngineCounter.use_count() == 1. Destroying a runtime before destroying deserialized engines created by the runtime leads to undefined behavior.
Aborted (core dumped)
VS
Sample app:
ERROR: [TRT]: 3: [runtime.cpp::~Runtime::346] Error Code 3: API Usage Error (Parameter check failed at: runtime/rt/runtime.cpp::~Runtime::346, condition: mEngineCounter.use_count() == 1. Destroying a runtime before destroying deserialized engines created by the runtime leads to undefined behavior.
How can we overcome this problem?
It seems that the structure of the project, not being made up of a single python file, could be the cause?
Thanks!
Here is the folder :
Sorry for long delay because vacation.The link has expired.
Usually no problem
Hi,
Here is a valid link :
Thanks!
Hi @junshengy
Were you able to download the project this time ?
Thanks
Yes, I can. I will try to reproduce