A while ago I put up an experimental module build for the FTDI driver on the Jetson TX1: https://github.com/jetsonhacks/TX1FTDIModule
The scripts downloads the kernel source, decompresses the config.gz file, adds the local version number, adds the FTDI module and then makes the module.
Before doing all that you’ll have to install the 64 bit extensions to gcc, there’s a script in there for that too.
In your case, you’ll have to modify the script and config file for the modules you’ll need, but at least this is an outline.
Just to be safe, I would suggest a fresh install in L4T 23.1, build your modules, save your modules off of the Jetson after they are built, reinstall L4T 23.1, and then install the modules appropriately. If you want to live on the edge, you won’t do the reinstall, but you’ll want to save the module and config file off board in any case.
I think the intent is that you build everything on a host which makes it easier to deal with a 64 bit kernel/32 bit OS on the TX1, but I haven’t encountered any issues with the FTDI driver that I surreptitiously built.