I have a Python program that performs deep network inference on images using TensorFlow. When a single program runs the GPU is not fully utilized. However, when running several programs simultaneously, the inference time per image is slower compared to running a single program. What could be the reason for that and what can be done to improve the inference time for multiple programs? Currently I am on windows (can move to Linux if needed) with GTX 1070 and have set the gpu_options.per_process_gpu_memory_fraction parameter of TesorFlow.
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| the inference time increases linearly when running more than 2 tensorrt instance on single GPU | 1 | 1647 | April 4, 2019 | |
| Parallel inference | 1 | 1026 | November 27, 2022 | |
| Optimal Trt inference using threads/processes for peoplenet model for | 1 | 1228 | July 30, 2021 | |
| Cuda Kernels running slow | 0 | 518 | November 9, 2018 | |
| run multi Inference programs in same time | 4 | 663 | January 21, 2020 | |
| Profiling single-gpu multi-session tf inference | 0 | 837 | September 22, 2020 | |
| Running multiple parallel pipeline with nvinfer vs batched inference | 19 | 292 | August 25, 2026 | |
| Multiple threads running inference are causing a slowdown | 24 | 1568 | August 13, 2023 | |
| When running multiple inferences and benchmarking the time of each one, does the first inference longer than the other ones? | 3 | 515 | October 27, 2020 | |
| Running Multiple DetectNets on Jetson TX1 | 1 | 931 | March 30, 2018 |