During a recent discussion, we realized that even with std::thread, it’s impossible to guarantee the simultaneous launch of two CUDA kernels. It turns out the problem isn’t just with the GPU, but with the host CPU’s architecture itself.
According to the Von Neumann architecture, CPU code always executes sequentially. std::thread calls happen one after the other. The operating system then introduces additional non-determinism by deciding when and on which core to run those threads. This creates delays—even if they’re just microseconds—that are simply unacceptable for safety-critical systems like those in nuclear or aerospace applications.
We know that NVIDIA has already taken a huge step with the Grace-Hopper platform, which unifies the CPU and GPU on a single chip for tighter integration.
But the question remains: Is NVIDIA planning to develop its own line of CPUs with deterministic characteristics to provide hard real-time guarantees for kernel launches and data processing? This would be a massive leap toward integrating CUDA into fields where reliability and predictability are absolutely critical.
(1) Historically, with rare exceptions, NVIDIA has not commented on future product developments. The earliest the public hears about a new product is at its introduction. This was the case with CUDA as well.
(2) Hard real-time requires a suitable processor plus a suitable RTOS, meaning substantial investment is required to offer a system solution. If I look at the many CPU designs offered by ARM, only a few models (Cortex-R and a few Cortex-M) are offered for hard real-time applications, which seems to indicate that hard real-time applications are a small niche. Since NVIDIA appears to be looking for areas of both rapid growth and large volume in the compute universe, it seems highly unlikely that they would be interested in this.
(3) Many of the features that make high-performance processor high performance also cause them to be unsuitable for hard real-time applications. As a result, processors suitable for hard real-time applications tend to be of lower performance. Custom-tailored solutions using FPGAs or custom ASICs might be a more suitable approach.
You’ve rightly pointed out that the market for processors in hard real-time applications has its own specifics, and choosing the right solution requires careful analysis. But let me ask: why pay thousands of dollars for an AArch64 processor from NVIDIA when you can find something similar on the market for $150?
Today, nearly all hardware manufacturers offer more or less the same thing. We could exchange plenty of fancy words and compliments, but in this case, it won’t bring any real value to the company. Only a clear-eyed view of the situation can help strengthen its position in the market. The von Neumann architecture, essentially undermines these efforts. The company would spend significant resources developing something that turns its strengths into weaknesses. Such CPUs stifle the GPUs, which are NVIDIA’s core advantage.
In other words, why spend so much effort and resources developing a CPU that could bottleneck your GPUs when paired with them, especially since you’re leaders in the GPU market?
Unless I am misinformed, NVIDIA does not sell standalone ARM processors, nor have I heard rumors that it intends to do so. From what I can see, that is not a business model that would make sense for them. NVIDIA does sell integrated solutions that include their ARM processors, including Jetson platforms for embedded applications suitable for various soft real-time applications. This seems smart as it reduces dependency on 3rd party CPU vendors.
When I joined NVIDIA in 2003, the company had about 2,500 employees and annual revenues of $2B. Now it has 36,000 employees and $130B in annual revenue. This all happened under the guidance of one CEO. Given that, it does not look like NVIDIA is in need of strategic advice from people outside the company. Technology companies like NVIDIA often lose their bearings and get into financial trouble once the original founder(s) and CEO depart, but given that Jensen is still quite young, I would think the company has a prosperous future ahead of it for the foreseeable future (in the absence of force majeure events such as the destruction of Taiwanese semiconductor manufacturing).
I don’t have the expertise to debate njuffa’s various points, but I do concur with him that I think it is generally unlikely for anyone from NVIDIA to discuss future product plans here. There are various reasons for this. This is a community forum; you’re of course welcome to post such questions/topics, and as already has happened, it may be that the community responds.
In addition to posting here, concrete suggestions for future improvements may be submitted in the form of a bug. I wouldn’t be able to set expectations for a suggestion of this sort, but I believe most or nearly all bugs receive a response of some sort.
NVIDIA does offer products in some possibly tangential spaces; the Drive product family comes to mind. But I don’t doubt that it differs meaningfully from the original focus here in this thread.
I think we should distinguish between the predictibilty of the host system and full determinism and hard realtime of the GPUs.
To improve the host system makes sense for Nvidia. Use a predictable setup that does not introduce spurious, but long delays. That is probably more about software and operating system running on the host than hardware.
However, the GPU part, while being non-deterministic, is quite predictable. Only specific kernels are launched. Those are typically short and have a repeatable behavior over multiple launches. So they can be used in soft realtime applications needing high performance = throughput.