[35.4.1] LAN Tx performace is getting lower when HDMI is plug out

Probably the developer kit is somewhere physically broken. If it is in warranty, you can consider do RMA process:
Jetson FAQ | NVIDIA Developer

The latest test we do is to connect two Xavier NX developer kits in Jetpack 5.1.2, and do iperf3 test. While plugging/unplugging HDMI port, performance drops for a short while and goes back to normal. If you see different behavior, it seems possible the board has minor physical damage.