# Low gpu utilization during training for custom object detection (TLT / Faster\_RCNN) **URL:** \ **Category:** TAO Toolkit\ **Created:** [April 7, 2020, 1:00pm UTC](https://forums.developer.nvidia.com/t/low-gpu-utilization-during-training-for-custom-object-detection-tlt-faster-rcnn/118841 "2020-04-07T13:00:21Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![nrsw5c06](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@nrsw5c06](https://forums.developer.nvidia.com/u/nrsw5c06)\ **Post date:** [April 7, 2020, 1:00pm UTC](https://forums.developer.nvidia.com/t/low-gpu-utilization-during-training-for-custom-object-detection-tlt-faster-rcnn/118841/1 "2020-04-07T13:00:21Z")
Hello, Im using tlt object detection for a custom dataset and was wondering if there are any advice on how to increase the gpu utilization. I noticed that the Volatile GPU-util jumps around and that the GPU memory are not fully utilized but remains static. I looked at [Training process is slow, GPU is not fully utilized - #5 by quansm](https://forums.developer.nvidia.com/t/training-process-is-slow-gpu-is-not-fully-utilized/80405/5) and increased the batch size, which made the Volatile GPU-util increase on average, but it still jumps all around which made me think about prefetching workers. The aforementioned issue nicely shows how to configure more workers but i dont see it in the Faster RCNN config for object detection. help is greatly appreciated! ![image](https://global.discourse-cdn.com/nvidia/original/3X/b/c/bc7e382d527667f487c423ce3372cad3f2298251.png) ---
**Author:** ![quansm](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@quansm](https://forums.developer.nvidia.com/u/quansm)\ **Post date:** [April 8, 2020, 12:39am UTC](https://forums.developer.nvidia.com/t/low-gpu-utilization-during-training-for-custom-object-detection-tlt-faster-rcnn/118841/3 "2020-04-08T00:39:42Z")
Sorry, I am not familiar with tlt object detection. In my case, it seems use tensorflow’s python script function to preprocess 3D medical image data, and it cost a lot of time. So I think this is the bottleneck. Before confirm that problem, I also use profile tool to measure the timeline to confirm the problme , and try to use tfrecord and remove python op , but not finished. If the problem only happen in your custom data, you may check it or think about the difference between it with public data. Or use top command to check cpu memory usage or use time function to log function execution time may be helpful. ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [April 8, 2020, 9:44am UTC](https://forums.developer.nvidia.com/t/low-gpu-utilization-during-training-for-custom-object-detection-tlt-faster-rcnn/118841/4 "2020-04-08T09:44:15Z")
Which tlt docker did you use? ---
**Author:** ![nrsw5c06](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@nrsw5c06](https://forums.developer.nvidia.com/u/nrsw5c06)\ **Post date:** [April 8, 2020, 9:47am UTC](https://forums.developer.nvidia.com/t/low-gpu-utilization-during-training-for-custom-object-detection-tlt-faster-rcnn/118841/5 "2020-04-08T09:47:19Z")
I used [nvcr.io/nvidia/tlt-streamanalytics:v1.0.1\_py2](http://nvcr.io/nvidia/tlt-streamanalytics:v1.0.1_py2) When i did docker run i changed runtime=nvidia with --gpus all ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [April 8, 2020, 9:52am UTC](https://forums.developer.nvidia.com/t/low-gpu-utilization-during-training-for-custom-object-detection-tlt-faster-rcnn/118841/6 "2020-04-08T09:52:36Z")
From my experience for experiments, it is related to different training work. I can see some of jobs get high GPU utiliztion. Some are lower. ---
**Author:** ![TomNVIDIA](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/tomnvidia/32/14181_2.png) [@TomNVIDIA](https://forums.developer.nvidia.com/u/TomNVIDIA)\ **Post date:** [October 12, 2021, 4:24pm UTC](https://forums.developer.nvidia.com/t/low-gpu-utilization-during-training-for-custom-object-detection-tlt-faster-rcnn/118841/8 "2021-10-12T16:24:43Z")