Reading through the Xavier SoC TRM I noticed that there were a few mentions of virtualization support.
Over the weekend I decided to experiment with enabling KVM virtualization, and my experiment was a success! :-)
If you’re interested in enabling KVM for your Xavier, you can follow a guide I have written here.
For best results, I would recommend using qemu >=5.0 as it has added support for ARMv8.1 and ARMv8.2 extensions.
Here’s a screenshot of Fedora 31 running with virgl-backed virtio-gpu: