Probably I would just write a CUDA kernel to convert RGB 565 to RGB 888, then call a function from the npp library if further conversion was needed (e.g. RGB to YUV).
I wouldn’t be able to answer “why” questions. I don’t think there is any “fundamental” reason you don’t find library implementations for RGB 565 (lets say in npp). It might be just not popular enough.
Regarding NVENC, that is a HW engine. So the hardware designers are not going to just provide hardware support for a large variety of little-used formats. I don’t know if that is actually the reason, but it might be. That would have real silicon costs, which are going to be weighed carefully against the perceived need or use case. I haven’t seen a large number of questions like this, which suggests to me that there are not a lot of people who are chafing at the lack of direct support for RGB 565.
If you want to see support for something in CUDA, a good way to express that is to file a bug