The question is a bit too general to just answer. Broadly speaking, I’m not sure if kernel options during build would help, but at least in R24.1 I noticed some debug options set. You might want to load the “/proc/config.gz” for a kernel compile and go through “make menuconfig” to see what can be changed regarding debug support…removing debug support and using optimization may be a possibility for kernel speed boost, but I do not know exactly what is there now.
Typically supporting a given driver in module format would be slightly lower performance than integrating in the kernel. Options in module format which you know will be used could be converted to integrated format.
Typically having a driver supported as a module but not loaded will have no effect on performance, just file system use.
You might want to install htop and browse what processes are running…most will not have a significant impact on CPU use or RAM, and the ones which do mostly will be required. If you spot something you know isn’t needed, you could remove the package, e.g., if you don’t use wireless, then remove everything wireless in both the kernel and user space packages.
Determining the source of any bottleneck is really required if you truly want to tune for performance…all of the above may tweak things a bit, but not address specific issues. For example, if the issues are something like hardware device drivers competing for each other, then hardware IRQ servicing rate may have a significant impact by increasing it or decreasing it (only CPU0 can service hardware IRQ…sometimes responsiveness through faster IRQ polling helps, sometimes slower IRQ helps due to reducing overhead).
Running in a desktop GUI is often a big increase in latency if you can run and view something from a text-mode shell.