My PipeTuner tool is failing to optimize using a ResNet34_PeopleNet model. I have my own sample video (one video only as a test) and have generated ground truth by running an object detection python script, although this was done with a different YOLO based model because I didn’t know how to interpret the output of ResNet34 (i.e how to get bounding boxes with the output object from session.run() in python). Regardless the bounding boxes for the ground truth are accurate and should have a similar IOU with what ResNet34 detects.
Setup:
- Operating System: Ubuntu-22.04
- Drivers: 535.161.07
- Hardware Platform: x86
- DeepStream Version: 7.1
- TensorRT Version: tensorrt-dev 10.3.0.26-1+cuda12.5
Log Files:
sk.yml_output.zip (11.8 MB)
Command:
bash launch.sh nvcr.io/nvidia/deepstream:7.0-triton-multiarch ../configs/config_PipeTuner/sk.yml
Output:
Directory mapped into docker: /home/nerd/nvidia-ws/pipetuner-data_1.0/pipe-tuner-sample
Found avaliable ports: 50919 53727
Launch containers:
docker run --gpus all -itd --net=host -v /home/nerd/nvidia-ws/pipetuner-data_1.0/pipe-tuner-sample:/home/nerd/nvidia-ws/pipetuner-data_1.0/pipe-tuner-sample --name ds_2025-10-02_14-26-50 nvcr.io/nvidia/deepstream:7.0-triton-multiarch;
docker run --gpus all -itd --net=host --name tuner_2025-10-02_14-26-50 -v /var/run/docker.sock:/var/run/docker.sock -v /home/nerd/nvidia-ws/pipetuner-data_1.0/pipe-tuner-sample:/home/nerd/nvidia-ws/pipetuner-data_1.0/pipe-tuner-sample nvcr.io/nvidia/pipetuner:1.0;
b62f1de6ea344c5e64b98d28a7e0c616c22612aef8af95117e4a80b7a0ca2c8b
140ed97c28cc0e10c58cd8e6292f2424377f1b76b397938154e6589b2c0cc183
Creating output directory…
mkdir -p /home/nerd/nvidia-ws/pipetuner-data_1.0/pipe-tuner-sample/output; cp ../configs/config_PipeTuner/sk.yml /home/nerd/nvidia-ws/pipetuner-data_1.0/pipe-tuner-sample/output; sed -i “s/ containerImageID:*/containerImageID:ds_2025-10-02_14-26-50\n/” -i “s/ port: 50919\n/” /home/nerd/nvidia-ws/pipetuner-data_1.0/pipe-tuner-sample/output/sk.yml;
Installing dependencies…
Installing dependencies (1/2)
Installing dependencies (2/2)
Launch BBO client…
Launch BBO server…
PipeTuner started successfully!
!!! To stop tuning process in the middle, press CTRL+C !!!
adding: DsAppServer (deflated 63%)
2025-10-01 21:40:23,052 root INFO seq_list: [‘workshop_person’]
2025-10-01 21:40:23,090 root INFO Writing configs to /home/nerd/nvidia-ws/pipetuner-data_1.0/pipe-tuner-sample/output/sk.yml_output/results/configs_10-01-2025_21-40-23
2025-10-01 21:40:23,090 root INFO send backend init
2025-10-01 21:40:23,090 root INFO creating optimizers…
2025-10-01 21:40:23,091 root INFO done. created 2
- Serving Flask app ‘ds_bbo_frontend_server’
- Debug mode: on
2025-10-01 21:40:23,110 root INFO init jobs done
2025-10-01 21:40:23,116 root INFO progress: 0% (0/100)
Launching server on: http://0.0.0.0:50919
received /init call
2025-10-01 21:40:23,123 root INFO wait backend ready
[server core] initializing
2025-10-01 21:40:24,124 root INFO wait backend ready
2025-10-01 21:40:33,340 root INFO progress: 0% (0/100)
2025-10-01 21:40:43,559 root INFO progress: 1% (1/100) ETA 00:33:00
2025-10-01 21:40:53,779 root INFO progress: 3% (3/100) ETA 00:16:10
2025-10-01 21:41:02,495 root WARNING DsApp score is 0. number of 0 scores:1
2025-10-01 21:41:04,009 root INFO progress: 4% (4/100) ETA 00:16:00
2025-10-01 21:41:14,225 root INFO progress: 5% (5/100) ETA 00:16:09
2025-10-01 21:41:24,447 root INFO progress: 6% (6/100) ETA 00:15:55
2025-10-01 21:41:29,557 root WARNING DsApp score is 0. number of 0 scores:2
2025-10-01 21:41:34,667 root INFO progress: 7% (7/100) ETA 00:15:43
2025-10-01 21:41:44,886 root INFO progress: 8% (8/100) ETA 00:15:31
2025-10-01 21:41:55,107 root INFO progress: 9% (9/100) ETA 00:15:20
2025-10-01 21:41:56,610 root WARNING DsApp score is 0. number of 0 scores:3
2025-10-01 21:42:05,329 root INFO progress: 10% (10/100) ETA 00:15:18
2025-10-01 21:42:14,648 root WARNING DsApp score is 0. number of 0 scores:4
2025-10-01 21:42:15,553 root INFO progress: 12% (12/100) ETA 00:13:41
2025-10-01 21:42:25,774 root INFO progress: 13% (13/100) ETA 00:13:36
2025-10-01 21:42:32,687 root WARNING DsApp score is 0. number of 0 scores:5
2025-10-01 21:42:36,003 root INFO progress: 14% (14/100) ETA 00:13:30
2025-10-01 21:42:46,223 root INFO progress: 15% (15/100) ETA 00:13:30
2025-10-01 21:42:50,730 root WARNING DsApp score is 0. number of 0 scores:6
2025-10-01 21:42:56,441 root INFO progress: 16% (16/100) ETA 00:13:23
2025-10-01 21:43:06,671 root INFO progress: 17% (17/100) ETA 00:13:15
2025-10-01 21:43:16,891 root INFO progress: 18% (18/100) ETA 00:13:08
2025-10-01 21:43:27,111 root INFO progress: 20% (20/100) ETA 00:12:12
2025-10-01 21:43:37,343 root INFO progress: 21% (21/100) ETA 00:12:09
2025-10-01 21:43:47,562 root INFO progress: 22% (22/100) ETA 00:12:03
2025-10-01 21:43:57,790 root INFO progress: 23% (23/100) ETA 00:11:56
2025-10-01 21:44:01,697 root WARNING DsApp score is 0. number of 0 scores:7
2025-10-01 21:44:08,009 root INFO progress: 24% (24/100) ETA 00:11:49
2025-10-01 21:44:18,223 root INFO progress: 24% (24/100) ETA 00:12:24
2025-10-01 21:44:28,458 root INFO progress: 26% (26/100) ETA 00:11:37
2025-10-01 21:44:38,686 root INFO progress: 27% (27/100) ETA 00:11:29
2025-10-01 21:44:48,902 root INFO progress: 28% (28/100) ETA 00:11:21
2025-10-01 21:44:59,120 root INFO progress: 29% (29/100) ETA 00:11:15
2025-10-01 21:45:09,338 root INFO progress: 30% (30/100) ETA 00:11:07
2025-10-01 21:45:19,557 root INFO progress: 31% (31/100) ETA 00:10:58
2025-10-01 21:45:29,788 root INFO progress: 32% (32/100) ETA 00:10:50
2025-10-01 21:45:40,005 root INFO progress: 34% (34/100) ETA 00:10:13
2025-10-01 21:45:50,224 root INFO progress: 35% (35/100) ETA 00:10:07
2025-10-01 21:45:58,038 root WARNING DsApp score is 0. number of 0 scores:8
2025-10-01 21:46:00,458 root INFO progress: 36% (36/100) ETA 00:09:59
2025-10-01 21:46:10,685 root INFO progress: 37% (37/100) ETA 00:09:50
2025-10-01 21:46:20,903 root INFO progress: 38% (38/100) ETA 00:09:42
2025-10-01 21:46:31,122 root INFO progress: 39% (39/100) ETA 00:09:35
2025-10-01 21:46:41,342 root INFO progress: 40% (40/100) ETA 00:09:27
2025-10-01 21:46:51,559 root INFO progress: 41% (41/100) ETA 00:09:18
2025-10-01 21:47:01,177 root WARNING DsApp score is 0. number of 0 scores:9
2025-10-01 21:47:01,794 root INFO progress: 43% (43/100) ETA 00:08:47
2025-10-01 21:47:12,016 root INFO progress: 44% (44/100) ETA 00:08:39
2025-10-01 21:47:22,244 root INFO progress: 45% (45/100) ETA 00:08:32
2025-10-01 21:47:32,464 root INFO progress: 46% (46/100) ETA 00:08:23
2025-10-01 21:47:42,684 root INFO progress: 47% (47/100) ETA 00:08:15
2025-10-01 21:47:46,291 root WARNING DsApp score is 0. number of 0 scores:10
2025-10-01 21:47:52,904 root INFO progress: 48% (48/100) ETA 00:08:06
2025-10-01 21:48:03,124 root INFO progress: 49% (49/100) ETA 00:07:58
2025-10-01 21:48:04,327 root WARNING DsApp score is 0. number of 0 scores:11
2025-10-01 21:48:13,356 root INFO progress: 50% (50/100) ETA 00:07:50
2025-10-01 21:48:23,577 root INFO progress: 51% (51/100) ETA 00:07:41
2025-10-01 21:48:33,801 root INFO progress: 52% (52/100) ETA 00:07:32
2025-10-01 21:48:40,412 root WARNING DsApp score is 0. number of 0 scores:12
2025-10-01 21:48:44,019 root INFO progress: 53% (53/100) ETA 00:07:23
2025-10-01 21:48:54,238 root INFO progress: 54% (54/100) ETA 00:07:15
2025-10-01 21:49:04,456 root INFO progress: 55% (55/100) ETA 00:07:06
2025-10-01 21:49:14,674 root INFO progress: 56% (56/100) ETA 00:06:57
2025-10-01 21:49:24,943 root INFO progress: 57% (57/100) ETA 00:06:48
2025-10-01 21:49:35,188 root INFO progress: 59% (59/100) ETA 00:06:23
2025-10-01 21:49:45,414 root INFO progress: 60% (60/100) ETA 00:06:14
2025-10-01 21:49:55,632 root INFO progress: 61% (61/100) ETA 00:06:05
2025-10-01 21:50:05,851 root INFO progress: 62% (62/100) ETA 00:05:56
2025-10-01 21:50:10,659 root WARNING DsApp score is 0. number of 0 scores:13
2025-10-01 21:50:16,089 root INFO progress: 63% (63/100) ETA 00:05:47
2025-10-01 21:50:26,312 root INFO progress: 64% (64/100) ETA 00:05:39
2025-10-01 21:50:28,717 root WARNING DsApp score is 0. number of 0 scores:14
2025-10-01 21:50:36,552 root INFO progress: 65% (65/100) ETA 00:05:30
2025-10-01 21:50:46,772 root WARNING DsApp score is 0. number of 0 scores:15
2025-10-01 21:50:46,772 root INFO progress: 67% (67/100) ETA 00:05:06
2025-10-01 21:50:55,804 root WARNING DsApp score is 0. number of 0 scores:16
2025-10-01 21:50:57,024 root INFO progress: 68% (68/100) ETA 00:04:57
2025-10-01 21:51:07,246 root INFO progress: 69% (69/100) ETA 00:04:49
2025-10-01 21:51:13,857 root WARNING DsApp score is 0. number of 0 scores:17
2025-10-01 21:51:17,464 root INFO progress: 70% (70/100) ETA 00:04:40
2025-10-01 21:51:22,874 root WARNING DsApp score is 0. number of 0 scores:18
2025-10-01 21:51:27,688 root INFO progress: 71% (71/100) ETA 00:04:31
2025-10-01 21:51:37,905 root INFO progress: 72% (72/100) ETA 00:04:22
2025-10-01 21:51:48,134 root INFO progress: 73% (73/100) ETA 00:04:13
2025-10-01 21:51:49,937 root WARNING DsApp score is 0. number of 0 scores:19
2025-10-01 21:51:58,363 root INFO progress: 74% (74/100) ETA 00:04:04
2025-10-01 21:51:58,964 root WARNING DsApp score is 0. number of 0 scores:20
2025-10-01 21:52:08,587 root INFO progress: 76% (76/100) ETA 00:03:42
2025-10-01 21:52:18,804 root INFO progress: 77% (77/100) ETA 00:03:33
2025-10-01 21:52:29,027 root INFO progress: 78% (78/100) ETA 00:03:24
2025-10-01 21:52:35,038 root WARNING DsApp score is 0. number of 0 scores:21
2025-10-01 21:52:35,345 root ERROR Too many 0 scores from DS app. We stop all processes. Please check DS app logs
2025-10-01 21:52:35,345 root ERROR OPTIMIZATION not completed!
[result sender] up and running!
[worker [worker 1] up and running!
0] up and running!
received /reset call
[server core] resetting
stopping result senders…
done.
stopping workers…
number of workers to stop: 2
number of workers to stop: 0
done.