Clearly, people have built carrier boards, because you can buy carrier boards from places like Connect Tech, Auvidea, and so forth.
I don’t see them posting on these boards much, though, so maybe you could try sending them email directly or something to see if you can get in touch with their designers.
In general, as far as I can tell, the hard parts are the high-data-rate busses – PCI Express, USB 3, and MIPI CSI for example. If you haven’t designed high-data-rate PCBs before, and don’t know how to match timing, use impedance controlled PCB manufacturing, and so forth, then you’re unlikely to have an easy time of making a well working Jetson carrier board. It’s not quite as bad as a modern multi-core PC motherboard, but it’s getting up there.
The other hard part is the linux device tree configuration to go with your board. As soon as you diverge from the default Jetson carrier board in any way, you will need to design a ‘dts’ device tree source file that matches your new configuration, and make sure that new bootloader and kernel builds use that file. If you’re not used to Linux low-level development, this will be a pain in and of itself.