Your usb port will not work properly one day in some cases when there is no usb2.0 bounded.
Didn’t hit error now does not mean it will always work fine. You are not the first one doing such thing and those users all come back and report error. Thus, please try not to do that.
USB3 must have a usb2 to work along with. This is the spec.