cuDNN memory exception when using more than 2 channels

Hello experts,

I am using openCV 4.3 with CUDA. openCV is compiled with CUDA 10.2 and cuDNN 7.6.5. I am using GeForce GTX 1050 with Driver version 451.67.

The issue that I am observing is with Yolo when used with cuDNN. Everything works as expected when using 2 cameras. But if I use more than 2, openCV cv::dnn::Net function forward() throws exception.

Any ideas why it is happening?
Any kind of clue would be greatly appreciated.

Thanks in advance.