The Xavier is likely to be good enough for making decisions about racing on its own (gas/brake/steer/gear.)
Once you have those decisions, you need to physically actuate the steering, gas, brake, and gearbox. You need something that can do this – servo motors, solenoids, or similar (depending on the specifics of your physical interface.)
Those actuators typically come with some form of communications interface – UART, RS-485, CAN, Ethernet, or something like that – if they are “servo actuators.” If this is the case, you need to output the right commands using the right communications protocol from the Xavier to your actuator. This should generally be straightforward, although for certain protocols that aren’t supported directly by the Xavier, you may need some kind of intermediary electronics.
Sometimes, those actuators are just “raw” motors (DC motors, or solenoids, or stepper motors,) which need a “driver” to actually switch the high-power driving current on/off. Those controllers are typically boxes with big MOSFETs and some other circuitry in them. The Xavier does not have the capability to do this switching itself. Those drivers, in turn, will need to be driven by some kind of input signal. For simple stepper motor drivers, you typically provide “step and direction” inputs. The Xavier could probably provide those signals over the I/O interface port (the “raspberry pi compatible” pins.) However, some motor drivers (H-bridges) will need PWM signals to modulate power delivery, and the Xavier does not have a simple API to generate suitable PWM signals for an H-bridge. If you need to generate those yourself, you should probably use a microcontroller to do so. (I’d recommend the Teensy microcontroller board series, connected over USB, but you can do whatever you want, including PIC over I2C, Arduino over serial, or whatever.)
Finally, you may also need to interface with various kinds of sensors. Some sensors (linear actuator potentiometers, for example,) are analog, and the Xavier does not have a good instrumentation-type analog input. (The audio inputs are DC de-coupled, which is bad in this case.) Some sensors are digital signals of high data rates, using no specific protocol, for example wheel encoder sensors. The Xavier does not have a simple package for tying counters to encoder inputs to figure out how far they have spun, although it may be possible to dive into the ARM peripheral reference manuals and configure a timer/counter to do this for you, once the technical reference manual is available. Typically, you’ll be better off using a microcontroller for those kinds of sensors.
Other kinds of sensors (such as IMUs, and various magnetic position sensors) will interface with I2C or SPI or UART, and thus you can talk to them straight from the Xavier, as it has those busses available.
So, the answer depends entirely on what your physical setup is, and how “smart” your controllers or actuators are on their own.