Is the host a VM, or native install? If a VM, then this is an issue of VMs seen a lot. However, VMs do tend to see the device in the beginning, and only later fail to see the device.
Is this a dev kit? Dev kits come with a known high quality micro-B USB cable. Charger cables, and many cables out in the wild, were designed to be cheap and tend to fail for data. Try a different cable if you can.
Is this the dev kit carrier board, or some other carrier board? I would expect either to be somewhat reliable at achieving recovery mode, but if you have different carrier boards, then I would suggest swapping the module to another carrier board to see if the issue follows board or module.
It isn’t high on the list, but you might also try a different power supply in case it is power related.
If there is a signal quality issue, then consider trying (a) directly to the host PC port, and then again using a HUB (the HUB may work with signals of lower quality). Note that if you use a HUB, then be sure only the Jetson is on that HUB.