The results in the deepstream-test2 example vary from run to run. Despite using the same TRT engine generated for the models, the classification model results differ from run to run. Occasionally, no classification model results are produced, while in other instances the results differ significantly. This occurs on different frames and boxes, despite the coordinates of the boxes themselves being the same (at least up to 6 decimal places).
Example of differences on frame 42 for the same box (coordinates in brackets). In the first run, there is a result from the model that determines the type of car, in the second run there is not.
< #42 found #5 (1603.571045, 543.909546, 84.856934, 62.139530) largevehicle Car
---
> #42 found #5 (1603.571045, 543.909546, 84.856934, 62.139530) Car
In this example there are different brands of cars.
< #1302 found #10 (738.833923, 479.535278, 638.922058, 463.091339) mercedes Car
---
> #1302 found #10 (738.833923, 479.535278, 638.922058, 463.091339) hyundai Car
Why do the results not match from run to run? How can we achieve deterministic results using the same engine models and input data?
• Hardware Platform (Jetson / GPU): rtx 2080
• DeepStream Version: 6.2
• TensorRT Version: 8.5.2
• NVIDIA GPU Driver Version (valid for GPU only): 525.60.11
• Issue Type( questions, new requirements, bugs): bugs
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing):
Docker image and use a small plugin to analyze the results from NvDsObjectMeta for easy comparison. Here are the steps:
- Download the archive. reproduce.zip (4.9 KB)
- Unzip the archive
unzip reproduce.zip && cd reproduce - Build the Docker image
docker build -t deepstream - Run the Docker image
docker run -it --rm --gpus all deepstream - Go to to the deepstream-test2 example folder
cd sources/apps/sample_apps/deepstream-test2/ - Run the example twice and write the results to different files.
gst-launch-1.0 uridecodebin uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! \
queue ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! \
queue ! nvvideoconvert ! queue ! \
nvinfer config-file-path="dstest2_pgie_config.txt" ! queue ! \
nvtracker tracker-width=640 tracker-height=384 ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so \
ll-config-file=../../../../samples/configs/deepstream-app/config_tracker_NvDCF_perf.yml enable-batch-process=1 ! \
nvinfer config-file-path="dstest2_sgie2_config.txt" ! \
nvinfer config-file-path="dstest2_sgie3_config.txt" ! \
queue ! dsmetapublish ! fakesink > res1.txt
gst-launch-1.0 uridecodebin uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! \
queue ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! \
queue ! nvvideoconvert ! queue ! \
nvinfer config-file-path="dstest2_pgie_config.txt" ! queue ! \
nvtracker tracker-width=640 tracker-height=384 ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so \
ll-config-file=../../../../samples/configs/deepstream-app/config_tracker_NvDCF_perf.yml enable-batch-process=1 ! \
nvinfer config-file-path="dstest2_sgie2_config.txt" ! \
nvinfer config-file-path="dstest2_sgie3_config.txt" ! \
queue ! dsmetapublish ! fakesink > res2.txt
- Compare the results
diff res1.txt res2.txt