Opinions may differ :) I find it more convenient to keep in my linux config file the list of the drivers I need for my target, compile only those drivers and produce a single Image file containing all the needed drivers. Compilation is faster, there is no risk of module/kernel mismatch and management of backups is easier.