It is generally straightforward to control which encoder to use if you have enabled nvenc and vaapi in your system. For example, in obs-studio you can choose via a pop-up menu if you’d wish to use vaapi or nvenc and, in the former case, which device to use.
But when talking about decoding you’d normally don’t control this feature (at least I am not aware of such). For example if your system has an nvidia card AND an intel iGPU, the first capable of vdpau and the second capable of vaapi, which encoder will you be using? One would guess the answer is simple: if your monitor is attached to the nvidia card, then you’ll use vdpau, else vaapi or viceversa.
Even if this is the case, the plot thickens when you also setup an “Optimus” system. If your monitor is connected to the iGPU (intel) but the work is offloaded to the nvidia card, who would be responsible to decode the signal? The primary guess is that the nvidia card will, using vdpau, but I am not sure.
Anyone? just curious.