Im using tlt object detection for a custom dataset and was wondering if there are any advice on how to increase the gpu utilization. I noticed that the Volatile GPU-util jumps around and that the GPU memory are not fully utilized but remains static.
I looked at Training process is slow, GPU is not fully utilized and increased the batch size, which made the Volatile GPU-util increase on average, but it still jumps all around which made me think about prefetching workers. The aforementioned issue nicely shows how to configure more workers but i dont see it in the Faster RCNN config for object detection.
help is greatly appreciated!