I took a look at your bug report log and I can see in /var/log/Xorg.0.log.old that the X server started with just your Intel device, and then when the eGPU was plugged in it treated it as a hotplug and loaded the modesetting driver. The intention behind that is to be able to extend your desktop onto the monitors on the eGPU without having to restart Xorg.
The Xorg server does have a path to support hot unplug, but it’s triggered by udev’s device removal path. To trigger that I think you have to write a 1 to /sys/class/drm/card1/device/remove. Unfortunately, nvidia-drm doesn’t currently have a way to respond to device removal requests; currently you have to rmmod nvidia-drm completely. I have a change to add that removal path that I’m working on internally, but I should be able to provide it as a patch if you’re interested in testing it.
I think you can disable this behavior in the X server by setting
Option "AutoAddGPU" "false"
in the ServerFlags section of xorg.conf.