This isn’t a real answer, but one thing I suggest is use the serial console first to explore environment variables before you actually remove serial console output.
If you run “env” you will see for example “bootdelay=2”. You might try to set “bootdelay” to 0. You will also see the start of actual boot runs/loops through detecting a series of boot devices…look at variable “distro_bootcmd” which loops through everything in $boot_targets. The default for boot_targets is “mmc1 mmc0 usb0 pxe dhcp”…you could reorder this to start with mmc0, although it’ll never get to the other targets in most cases (you could just make mmc0 the only boot_target).
You could recompile U-Boot to not use serial console, but perhaps the environment variable cbootargs could just have the two console arguments removed (and in Linux console could be removed from extlinux.conf)…I have not tested, but this might be easier than recompiling U-Boot.
Within U-Boot console see “help env”. Once you permanently set something to not access serial console you might end up having to flash to get it back to the original setting.