The EDID wire is required for auto configuration. VGA does not have this. There is a steep learning curve to manually creating modelines and X configuration, but it can be done if you have detailed information and a lot of time and patience. Otherwise a mode which works is just from getting lucky (something may have set a working mode by luck, and the VGA could inherit this).
HDMI is hot plug, so you may test if unplugging and re-plugging gets things working, or if not having the monitor connected during boot, and then connecting after boot helps (think of it as debug steps, not as a solution).
Do you have ssh or serial console access? Chances are good you can ssh in and see if the drivers are correct:
sha1sum -c /etc/nv_tegra_release
Should you be able to get in, see if the “edid” file has content (and post the content if you can…this is the data the EDID wire mentioned above provides):
sudo -s