Hi,
It may not work well if you build only modules. Please build kernel and modules:
Kernel Customization — NVIDIA Jetson Linux Developer Guide
Kernel Customization — NVIDIA Jetson Linux Developer Guide
Here is a real practice on Jetpack 6 Orin:
No data from Joystick Logitech-f710 - #10 by DaneLLL
The steps are similar on Jetpack 7 Thor.