Pipetuner, DsApp score is 0

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Continuing the discussion from PipeTuner:

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 7.1
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only) 560.35.03

Hello, i’m trying to use the pipe tuner tool but encounter the same problem as the post linked above.
I am also using a custom model (a yolov4 model trained with TAO exported in onnx format).
I’ve used the model with deepstream before.

Here are my config files for pipe tuner and the inference :
config_files.zip (4.0 KB)

I made sure every files were mapped in the container even the custom-lib
I get this result :

Found avaliable ports: 51096 53017
Launch containers:
docker run --gpus all -itd --net=host -v /home/ia/pipe-tuner-sample:/home/ia/pipe-tuner-sample --name ds_2024-11-14_16-21-19 nvcr.io/nvidia/deepstream:7.0-triton-multiarch;
docker run --gpus all -itd --net=host --name tuner_2024-11-14_16-21-19 -v /var/run/docker.sock:/var/run/docker.sock -v /home/ia/pipe-tuner-sample:/home/ia/pipe-tuner-sample nvcr.io/nvidia/pipetuner:1.0;
94889f9626010fa65e1e61b494e6189838da0836dbcfa28e2fa264fb18dfa188
72eeb7b1cfc29f1580d4486edbdcddb2c3e012812065b9e11dc47de0da8bc0c1
Creating output directory...
mkdir -p /home/ia/pipe-tuner-sample/output; cp ../configs/config_PipeTuner/pipe_tuner_maritime.yml /home/ia/pipe-tuner-sample/output; sed -i "s/        containerImageID:*/containerImageID:ds_2024-11-14_16-21-19\n/" -i "s/    port: 51096\n/" /home/ia/pipe-tuner-sample/output/pipe_tuner_maritime.yml;
Installing dependencies...
Installing dependencies (1/2)
Installing dependencies (2/2)
Launch BBO client...
Launch BBO server...
PipeTuner started successfully!

!!!!! To stop tuning process in the middle, press CTRL+C !!!!!

  adding: DsAppServer (deflated 63%)
2024-11-14 07:33:19,294 root         INFO     seq_list: ['Boat_tracking_2', 'Boat_tracking_3', 'Boat_tracking_4', 'Boat_tracking_5', 'Boat_tracking_6', 'Boat_tracking_7', 'Boat_tracking_8', 'Boat_tracking_9']
2024-11-14 07:33:19,669 root         INFO     Writing configs to /home/ia/pipe-tuner-sample/output/pipe_tuner_maritime.yml_output/results/configs_11-14-2024_07-33-19
2024-11-14 07:33:19,676 root         INFO     send backend init
2024-11-14 07:33:19,678 root         INFO     creating optimizers...
2024-11-14 07:33:19,681 root         INFO     done. created 2
 * Serving Flask app 'ds_bbo_frontend_server'
 * Debug mode: on
2024-11-14 07:33:19,759 root         INFO     init jobs done
2024-11-14 07:33:19,774 root         INFO     progress: 0% (0/200)
Launching server on: http://0.0.0.0:51096
received /init call
2024-11-14 07:33:19,925 root         INFO     wait backend ready
[server core] initializing
2024-11-14 07:33:20,957 root         INFO     wait backend ready
2024-11-14 07:33:23,144 root         WARNING  DsApp score is 0. number of 0 scores:1
2024-11-14 07:33:25,281 root         WARNING  DsApp score is 0. number of 0 scores:2
2024-11-14 07:33:27,416 root         WARNING  DsApp score is 0. number of 0 scores:3
2024-11-14 07:33:29,239 root         WARNING  DsApp score is 0. number of 0 scores:4
2024-11-14 07:33:29,248 root         WARNING  DsApp score is 0. number of 0 scores:5
2024-11-14 07:33:29,920 root         INFO     progress: 2% (5/200) ETA 00:06:30
2024-11-14 07:33:31,440 root         WARNING  DsApp score is 0. number of 0 scores:6
2024-11-14 07:33:31,441 root         WARNING  DsApp score is 0. number of 0 scores:7
2024-11-14 07:33:33,564 root         WARNING  DsApp score is 0. number of 0 scores:8
2024-11-14 07:33:33,565 root         WARNING  DsApp score is 0. number of 0 scores:9
2024-11-14 07:33:35,695 root         WARNING  DsApp score is 0. number of 0 scores:10
2024-11-14 07:33:35,696 root         WARNING  DsApp score is 0. number of 0 scores:11
2024-11-14 07:33:37,534 root         WARNING  DsApp score is 0. number of 0 scores:12
2024-11-14 07:33:37,536 root         WARNING  DsApp score is 0. number of 0 scores:13
2024-11-14 07:33:39,709 root         WARNING  DsApp score is 0. number of 0 scores:14
2024-11-14 07:33:39,710 root         WARNING  DsApp score is 0. number of 0 scores:15
2024-11-14 07:33:40,027 root         INFO     progress: 7% (15/200) ETA 00:04:06
2024-11-14 07:33:41,836 root         WARNING  DsApp score is 0. number of 0 scores:16
2024-11-14 07:33:41,838 root         WARNING  DsApp score is 0. number of 0 scores:17
2024-11-14 07:33:43,685 root         WARNING  DsApp score is 0. number of 0 scores:18
2024-11-14 07:33:43,994 root         WARNING  DsApp score is 0. number of 0 scores:19
2024-11-14 07:33:45,821 root         WARNING  DsApp score is 0. number of 0 scores:20
2024-11-14 07:33:45,822 root         WARNING  DsApp score is 0. number of 0 scores:21
2024-11-14 07:33:46,132 root         ERROR    Too many 0 scores from DS app. We stop all processes. Please check DS app logs
2024-11-14 07:33:46,133 root         ERROR    OPTIMIZATION not completed!
[result sender] up and running!
[worker 0] up and running!
[worker 1] up and running!
received /reset call
[server core] resetting
stopping result senders...
Waiting for result sender to stop...
done.
stopping workers...
number of workers to stop: 2
number of workers to stop: 0
done.

!!!!! Press CTRL+C key to end PipeTuner !!!!!

I have tried the default configuration, it worked fine.
I tried, as suggested, to use the deepstream app in the DS container to make sure the output with my model is as expected and didn’t see any problem.

What can I do to try to find the problem?
Thank you

3

So you have tried all suggestions in PipeTuner - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums? We will check your configurations first.

4

@Alex6s You just provided two configuration files. Seems you changed more configurations and scripts to run your case. How did you run the pipetuner case? What did you changed for your cases?

5

I don’t think i’ve changed anything else, apart from the input data of course (that i’ve set according to the documentation). What make you think there is something else that changed ?
I modified the .yml pipetuner config file to include the path to my data and the pgiePath. I’ve also just changed the “checkClassMatch” parameter in the DCF config file, but that’s it. I run pipetuner with the command : >>bash launch.sh nvcr.io/nvidia/deepstream:7.0-triton-multiarch …/configs/config_PipeTuner/pipe_tuner_maritime.yml

6

Hi, I’m facing the exact same problem as the one presented both in this thread and the one referenced at the post. DsApp score returns always 0 when changing to a custom model.

For instance, I’m using YOLOv10 with the implementation of an open PR in this repo. This model and implementation has always worked perfectly when running barebones deepstream-app after building the library.

For using this implementation in PipeTuner, I’m using as DeepStream Docker image the one built with this Dockerfile:

FROM nvcr.io/nvidia/deepstream:7.0-triton-multiarch

RUN apt-get update && apt-get upgrade -y && \
    apt-get install -y git build-essential && \
    rm -rf /var/lib/apt/lists/*

RUN git clone https://github.com/davconde/DeepStream-Yolo.git \
    /opt/nvidia/deepstream/deepstream-7.0/DeepStream-Yolo
WORKDIR /opt/nvidia/deepstream/deepstream-7.0/DeepStream-Yolo
RUN CUDA_VER=12.2 make -C nvdsinfer_custom_impl_Yolo
RUN chmod -R 777 /opt/nvidia/deepstream/deepstream/DeepStream-Yolo
WORKDIR /opt/nvidia/deepstream/deepstream-7.0/

I build this as deepstream-yolo:7.0-triton-multiarch and then launch with:

sudo bash launch.sh deepstream-yolo:7.0-triton-multiarch ../configs/config_PipeTuner/road_MOT.yml

I attach my modified files in config_PipeTuner and config_PGIE:
configs.zip (4.2 KB)

Essentially I’ve just changed the pgiePath in the global PipeTuner config file and used a standard PGIE config file for YOLOv10 in YAML that I’ve always used in DeepStream.

After unsuccessful PipeTuner optimization, I’ve also tried attaching a shell to the DeepStream container created by launch.sh to run DeepStream with the same PGIE config file in order to check that the installation and set paths were correct. The deepstream-app ran correctly, tracking objects in the sample dataset. But when it comes to PipeTuner, it always gets DsApp score 0 when using my model.

I include the logs for reference, but as other users, only server and client logs are being generated:
logs.zip (3.9 MB)

7

@Alex6s
Have you tried all suggestions in PipeTuner - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums?

I noticed that you run the Pipetuner with your own dataset. Can you check whether the dataset is prepared correctly? Pipetuner Guide — DeepStream documentation

8

Hello,
Yes, here is what i’ve tried so far
Here is my data directory:

~/pipe-tuner-sample/data/maritime_videos$ tree
.
├── Boat_tracking_2
│   ├── gt
│   │   └── gt.txt
│   └── seqinfo.ini
├── Boat_tracking_2.mp4
├── Boat_tracking_3
│   ├── gt
│   │   └── gt.txt
│   └── seqinfo.ini
├── Boat_tracking_3.mp4
├── Boat_tracking_4
│   ├── gt
│   │   └── gt.txt
│   └── seqinfo.ini
├── Boat_tracking_4.mp4
├── Boat_tracking_5
│   ├── gt
│   │   └── gt.txt
│   └── seqinfo.ini
├── Boat_tracking_5.mp4
├── Boat_tracking_6
│   ├── gt
│   │   └── gt.txt
│   └── seqinfo.ini
├── Boat_tracking_6.mp4
├── Boat_tracking_7
│   ├── gt
│   │   └── gt.txt
│   └── seqinfo.ini
├── Boat_tracking_7.mp4
├── Boat_tracking_8
│   ├── gt
│   │   └── gt.txt
│   └── seqinfo.ini
├── Boat_tracking_8.mp4
├── Boat_tracking_9
│   ├── gt
│   │   └── gt.txt
│   └── seqinfo.ini
├── Boat_tracking_9.mp4
└── seqmap.txt

seqmap.txt:

name
Boat_tracking_2
Boat_tracking_3
Boat_tracking_4
Boat_tracking_5
Boat_tracking_6
Boat_tracking_7
Boat_tracking_8
Boat_tracking_9

seqinfo.ini (the lenght change depending on the video):

[Sequence]
seqLength=743

I launch the pipetuner tool with :

bash launch.sh nvcr.io/nvidia/deepstream:7.0-triton-multiarch ../configs/config_PipeTuner/pipe_tuner_maritime.yml

Once the containers started, i attach a terminal to the DS docker with (with the name of the docker that has been created) :

docker exec -it ds_2024-11-20_10-50-12 /bin/bash

I modified a config file from /opt/nvidia/deepstream/deepstream-7.0/samples/configs/deepstream-app to read one of my video in the data directory, and use my inference config file and model, it output a video with bboxes, i can confirm the detection is performed correctly.

I do have a question, in the MOT annotation format, the class is given as an id, in the label.txt file, labels are strings with the name of the class. Does pipetuner associate the “n” id in the MOT annotation file as the label at the nth line in the label file (not sure if my question is clear)?

9

The label.txt file is given by the model card which is generated when the model is trained. Take the PeopleNet | NVIDIA NGC model as the example, the model is trained to classify 3 classes of objects - person, bag and face. The coresponding class id of the three casses are “person -1 , bag - 2, face - 3”. In the Pipetuner “DS-based Perception tuning” sample, the class id 1 in SDG_1min_videos dataset means the person object.