There appears to be some issues with symbolic links. /usr/lib/aarch64-linux-gnu/libdrm.so.2 appears to be a symbolic link which points to the wrong file. It’s hard to tell if it gets overwritten or is setup incorrectly to begin with. As a workaround, you can try:
$ sudo rm /usr/lib/aarch64-linux-gnu/libdrm.so.2
$ sudo -H ln -s /usr/lib/aarch64-linux-gnu/libdrm.so /usr/lib/aarch64-linux-gnu/libdrm.so.2
Installing gazebo from the repositories worked correctly once that change was made on this Xavier (L4T 31.0.2, JetPack 4.1). The link must be changed after gazebo is installed.