hello shaorui,
you should review the driver implementation, did you shutdown the camera device completely before remove the kernel module? what’s the kernel messages reported while running rmmod?
please also check developer guide for reference, Loadable Kernel Module (LKM).
thanks