Hey everyone!
Let’s dive into the problem of interfacing critical subsystems with a neural network that’s running on a PC under an operating system. These OSes are generally ill-suited for such tasks because they were designed a long time ago for completely different purposes — servers, desktops, and cloud environments — where the priorities were flexibility, multitasking, and support for a huge number of devices and programs, not hard real-time constraints.
In real-time systems (RTS), it’s not enough for a task to execute correctly; it must also complete within a strictly defined deadline. Missing a deadline can lead to catastrophe. In automotive applications, for example, the time from detecting an obstacle to actuating the brakes needs to be in the millisecond range (typically 10–50 ms for emergency braking). Any extra delay can mean disaster.
Consider a concrete example: an autonomous vehicle controlled by a neural network (NN). The car is cruising down the highway when a pedestrian suddenly steps into the road. Here’s how the entire chain from detection to braking typically plays out in a system where everything is managed by a complex OS (e.g., Linux-based with real-time extensions):
Obstacle detection: Sensors (lidar, cameras, radars) register the object. Raw data flows into the system. The neural network, running under the OS, analyzes it: identifies the pedestrian, calculates speed, trajectory, and distance. This requires heavy computation — the NN might process thousands of frames per second, but on a powerful processor (like NVIDIA Orin or Qualcomm Snapdragon) under an OS, it still takes time. The OS controls everything: allocates CPU/GPU resources, schedules tasks via its scheduler, and handles sensor interrupts.
Analysis and decision-making: The OS coordinates the NN with other modules (e.g., path planner). The NN outputs its verdict: “danger — brake now.” But the OS is a multitasking environment. Other processes are running concurrently: map updates, cloud connectivity, battery monitoring, microphone audio processing, etc. The OS scheduler decides which task runs next. In typical server-grade OSes (like Ubuntu or other Linux distros), the scheduler is optimized for throughput — average performance across all tasks — not for hard deadlines. It can pause a critical task to let another run, or introduce delays from context switches. This creates jitter — unpredictable latencies ranging from 1 ms to 100 ms or more.
Command to actuators: Once analyzed, the OS sends the signal to the brakes (hydraulic or electromechanical). But the command goes through drivers, buses (CAN-bus, Ethernet), and again through the OS scheduler. If the OS is overloaded (e.g., the NN is handling a complex scene with many objects) or hits a fault (kernel panic, deadlock in multithreading), the command can be delayed. In the worst case, the OS hangs for seconds, and braking never happens.
Why is this a problem? Server-grade operating systems were never designed for hard real-time. Their codebase spans millions of lines with complex logic: virtual memory, filesystems, networking stacks. The scheduler (Linux’s CFS, for example) focuses on fair sharing of CPU time, not worst-case latency. Even applying patches (like PREEMPT_RT for Linux) doesn’t turn it into a true RTS. Here’s why:
Non-determinism: Patches improve preemptibility, but don’t eliminate all sources of delay. In PREEMPT_RT Linux, non-preemptible kernel sections (spinlocks, certain interrupt handling) remain. This leads to bounded but still significant latencies (100–200 µs in ideal cases, often worse in practice due to hardware).
Code complexity: A server OS is a massive monolith with thousands of modules. Patches add overhead rather than simplify. True RTS demand minimalism: code that’s easily verifiable and predictable. Server OSes have too much legacy and third-party code that isn’t certified for safety-critical use (e.g., ISO 26262 in automotive).
Uncontrolled delays: In hard RTS, deadlines are guaranteed (miss = failure). In patched Linux, they’re soft — the system tries its best but offers no guarantees. Page faults, cache misses, or I/O can add unpredictable milliseconds.
Certification: Automotive requires ASIL-D safety levels. Patched Linux rarely passes full certification because its complexity makes auditing practically impossible. Genuine RTS (QNX, VxWorks) are built from the ground up for this, often with microkernels and task isolation.
In the end, because of the OS’s internal logic and scheduler behavior, the time from danger detection to brake actuation can be excessively long — 100 ms or more, up to seconds in bad cases. In an autonomous vehicle, that means braking either happens too late or not at all if the OS hangs.
How to solve it? A hybrid architecture: use simple but specialized microcontrollers (MCUs) for the critical parts. MCUs (automotive-grade ones from NXP/TI, STM32, etc.) are programmed at a low level (C or assembly), with no OS or minimal bare-metal code. They have at most a few thousand lines of code — not millions — and interact directly with hardware, no middlemen.
To achieve maximum responsiveness, we need specialized MCUs optimized for hardware-level signal processing on input: advanced mechanisms for signal acquisition and handling, fast inter-MCU communication (e.g., CAN-FD or Ethernet TSN), and the ability to process large numbers of events in parallel. For example, multiple NVIC (Nested Vectored Interrupt Controllers) that can independently handle dozens of interrupts simultaneously — essential when many sensors fire events at once and need instant response without bottlenecks.
Furthermore, imagine companies like NVIDIA — already dominant in AI and automotive SoCs (Orin, DRIVE platforms) — developing their own specialized MCUs. These could be perfectly tailored for integration with onboard computers running neural networks: seamless data exchange with the NN, possibly direct access to AI accelerators. In such an ecosystem, the MCUs could even communicate with data centers (for updates or logging via cloud protocols), but the core principle remains: they always run low-level code, delivering predictability and minimal latency without the overhead of complex OSes.
Back to our vehicle example:
The main OS on a powerful SoC handles high-level analysis: NN, planning, UI.
The MCU connects directly to the braking system (via GPIO, PWM, or bus).
The MCU receives data not only from the OS but also from additional sensors. For instance, “smart” proximity sensors with basic on-chip preprocessing (e.g., edge detection) so the MCU isn’t burdened with heavy computation. This lets the MCU monitor the situation independently, but only to the extent needed for speed — no full-blown NN.
Process: Sensors detect the obstacle. Both OS and MCU get the data
The MCU doesn’t blindly slam the brakes. It waits for confirmation from the OS within a strict time window (say, 20 ms).
If the OS responds in time with “yes, brake” — MCU triggers emergency braking.
If “no, false alarm” — normal driving continues.
If the OS fails to respond (hung or overloaded) — MCU switches to stabilization mode: gentle braking , speed reduction, or complete stop if the object is already in the critical zone but the operating system has not sent a response , hazard lights. This buys time for the OS to recover. If a “stop” command arrives later, the vehicle is already prepped for instant full stop. If “continue,” it returns to normal.
Thus, the MCU serves as a safety net: mitigating risks from the OS’s complexity and potential unreliability. Even if the OS crashes, the system retains basic control — the MCU provides fallback safety.
The same applies to robotics: for arms/legs in humanoid robots, MCUs directly control servos with minimal latency. They monitor commands from the central OS (motion coordination, NN-based balance) but have their own sensors (IMUs, encoders) for local control. If the OS lags, the MCU stabilizes posture to prevent falling.
What do you think?
Are NVIDIA Microcontrollers possible?This would be especially interesting in the context of NVIDIA’s smaller Jetson boards like the Jetson Nano or Jetson Orin Nano — platforms that are already hugely popular among hobbyists, makers, students, and small-scale developers for AI and robotics projects. If NVIDIA paired these Jetsons with accessible, dedicated MCUs (or exposed low-level programming on existing safety/realtime cores inside the SoC), it would open up incredible possibilities for ordinary users