installing CUDA Toolkit and driver to a Yocto image

I have a Target which has x86_64 CPU. I have built a Yocto linux image which boots up this CPU board. We will soon get a Nvidia GPU daughter board which will connect to the CPU board through PCIe. I want to be able to run CUDA programs on the Target.

How can I integrate NVidia driver for the GPU into the Yocto image? One option is to include the entire CUDA toolkit and driver installer in the Yocto image and install CUDA directly on the Target. But that increases the size of the image by 2.15 GB.

So I want to have everything in the Yocto image using only the minimum needed drivers and tools.

Appreciate any help in this regard.