If the kernel Image file has the driver already, then loading and unloading a module will always fail. You would have to recompile the kernel Image itself to not integrate the option before module commands would work. Keep in mind that not all features can be built as a module (don’t know about your case).
If you want to see what your kernel Image was compiled with, then see:
zcat /proc/config.gz | grep 'CONFIG_VIDEO_OV9281'
If this shows as “=y”, then your version of the driver will never load and never be unloaded.