Deepstream multi streams processing

[peeranat85]

Hi,

I’m building a DeepStream people tracking app based on deepstream_test_app3.c and deepstream_infer_tensor_meta_test.cpp. This app uses a multi-object tracker, ByteTrack, which is different from other DeepStream apps that the output detections are processed in a stateful fashion; for one input source, ByteTrack maintains a data structure that keeps track of the detections from the previous N frames and performs some post-processing logic. For this reason, my app maintains a BYTETracker object associated with one input source and depending on which input source the frame is coming from, the corresponding BYTETracker object is used.

Please have a look at https://github.com/cocoza4/ByteTrack/blob/main/deploy/TensorRT/cpp/src/ds_rtsp_bytetrack.cpp, it contains a CustomData struct which is hard-coded with 2 BYTETracker’s for debugging purposes.

To reproduce the issue, please follow the steps below

cd ByteTrack/deploy/TensorRT/cpp/
mkdir build && cd build
cmake ..
make

Run 1 input source

./ds_rtsp_bytetrack file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264

in this case, only 1 BYTETracker is used because only 1 input source is instantiated. Notice the tracker_id associated with each bounding box, everything works as expected.

Run 2 input sources

./ds_rtsp_bytetrack file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264 file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264

this time, 2 input sources are instantiated. Notice the tracker_id associated with each bounding box, they are different from the previous example. For some reason, the BYTETracker for each input source is not processed independently. So the question is, how to make BYTETracker process each one independently so that the output for each stream in the nv-tiler, matches the previous example?

The models and source code can be found at bytetrack_trt - Google Drive and GitHub - cocoza4/ByteTrack respectively.

I ran this experiment on docker image nvcr.io/nvidia/deepstream:6.0-devel, TensorRT v8001, and NVIDIA GeForce GTX 1650 Ti with Max-Q Design.

Thank you
Peeranat F.

[Fiona.Chen]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
• DeepStream Version
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

[peeranat85]

Architecture: x86_64
GPU: NVIDIA GeForce GTX 1650 Ti with Max-Q Design
NVIDIA GPU Driver: Driver Version: 495.29.05
DeepStream Version: 6.0 (running on docker image nvcr.io/nvidia/deepstream:6.0-devel)
TensorRT Version: v8001
Issue Type: Question

Thank you

[Fiona.Chen]

@peeranat85

For some reason, the BYTETracker for each input source is not processed independently. So the question is, how to make BYTETracker process each one independently so that the output for each stream in the nv-tiler, matches the previous example?

It depends on the BYTETracker. How does the BYTETracker handle the batch?

[peeranat85]

From my understanding, BYTETracker only handles batch_size=1, see line . However, I notice some weird model output; the model output for the second scenario(batch_size=2) is of shape [2, 13566, 6] but when I print info->inferDims.numElements it returns 81396 which is the same as the first scenario(batch_size=1). I checked the engine file, the right file was loaded correctly and the output dimension was output, [2, 13566, 6]. Is there something wrong with getting raw inference result here? Maybe DeepStream only allows decoding network result 1 batch_size at a time?

Back to your question: How does the BYTETracker handle the batch?
According to my configuration, batch_size=2 means that the first comes from stream1 and the second from stream2. When BYTETracker decodes network result see this, it selects the right one based on the whether it is stream1 or stream2. So, in this case, regardless of the batch size, it will always treat batch_size=1.

[Fiona.Chen]

@peeranat85

For some reason, the BYTETracker for each input source is not processed independently. So the question is, how to make BYTETracker process each one independently so that the output for each stream in the nv-tiler, matches the previous example?

You have identified the object belongs to which stream. So how to use BYTETracker has nothing to do with deepstream.

[peeranat85]

Thank you for your response. I have now identified the bug. It was due to BYTETracker’s handling of batch size 2 as you suspected in the first place. I’m now working on the decoding part. Thank you very much!