Does Nvidia have a roadmap with an “adding more registers” item? Sometimes fusing two or more kernels require a bit more registers than the one with highest register-pressure, and this causes difficulties in finding better performance points that can be useful in many areas of computing.
For example, sorting 64k-elements (256kB) in SM unit fused with stream-compaction of their key-value counterparts. 256kB already takes all of registers and spills. Then stream-compactions uses warp shuffle that is bottlenecked by shared-memory use (and not enough smem for async-loading another 256kB at once). So, there’s no smem to load async at large chunk size, and no register left to manual load, data is not overlapping during sort (which also touches warp shuffle and smem a bit). If there were 512 registers per thread, or even 1024 registers, some more advanced algorithms would be applicable to speedup sorting more than 256 registers can.
If register increase is not possible, is there a possibility of increasing shared-memory capacity? Or a cluster-setting that does not reduce SM units available for cluster kernel? Because sometimes combining SM units for more SMEM is useful except when it reduces available SM units from 132 to 120 or 112.
Are there debug registers that are writable like a normal register?
By historical observation, NVIDIA does not publicly discuss specifications of new architectures until they are ready to ship. If you are one of NVIDIA’s huge corporate customers with NDAs the situation might be different.
To clarify: Your comments pertain to the current Blackwell architecture. You want to know whether there will be more registers in the upcoming Rubin architecture, for which some specifications have been stated publicly, but I have yet to see any data on the size of the register file.
Generally speaking, building large register files operating at high frequencies is a hard problem from a hardware engineering perspective. This implies to me that one should not expect significant growth in the number of registers available per thread in the near future, and that it is best to organize one’s code to best work around the current register limitations. One method might be the use of ad-hoc data compression, e.g. use of some simple packing scheme. This then requires additional code to pack and unpack data, but it may still be a performance win overall.
Thank you for response. I mean, some algorithms perform like there are 50% more cuda cores or maybe 100% more cuda cores, with extra local storage that is faster than global memory. Are registers more expensive than cuda cores?
I do not think anyone outside of NVIDIA’s hardware engineering department can provide a reliable assessment of the relative cost of adding registers versus adding CUDA cores. Maybe a third-party with access to high-resolution die photos and recent processor design experience might be able to provide guesstimates, but I do not fall into that category.
That is presumably precisely because building large register files operating at high frequencies is a hard problem, as I stated.
You are asking for two different things: More than 256K of register file per SM or more than 256 registers per thread.
I don’t think that Nvidia would just add expensive registers for a small class of applications, they more likely would utilize them better or in a more flexible way.
E.g. one possible development is whether Nvidia will increase beyond 4 SM partitions per SM. If each brings 16K registers (=64 KiB) you could have 512KiB with 8 SMSPs.
But
would the shared memory size stay the same (so less shared memory per computational capacity)? Perhaps not what you want.
would the shared memory increase, too? Then the access bandwidth to shared memory could be a real bottleneck with 8 SM partitions fighting for a single access
The 64K registers have to be divided by 4 partitions and 32 lanes. So you actually currently have up to 512 registers per thread (for the other registers no direct connection exists).
Nvidia could introduce a mode that a thread can use 256 more registers not directly, but as a fast storage.
It would be neat, if we could get some way of dynamic indexing, too. At least selecting a register bank.
Any fast storage that does not use registers (like tmem) would be good. Tensor-memory copy operation has to go through registers always before reaching global mem or before used in computations generally and is not a direct replacement for a chunk of registers. Smem is shared by all warps and tends to be a bandwidth bottleneck as you said.
I think some memory space like this would be good:
runtime indexing allowed even if its private to a thread
lower latency than distributed-shared-memory (even when all threads utilize it)
higher bandwidth than L2 cache when all SM units are utilized
even if just 128kB extra, it would help a lot but +1MB would unlock even better algorithms, fusion patterns that would keep energy spent in SM units instead of high-bandwidth-memory, yielding higher GPU frequency → extra performance per transistor. Maybe HBM wouldn’t even be required for some algorithm, data would be loaded once and a small amount of data would be streamed slowly while all of power would go into SM units for higher frequency
then I wouldn’t even need a cluster launch to increase effective density of calculations happening per HBM memory byte fetched.
How does Cerebras create & run a 20kW chip without issues? Maybe they have issues and not telling. But they say they utilize gigabytes of sram in chip.
Nvidia’s gpu diagrams always show register file, cuda cores, tensor cores, shared memory / L1 as side by side. Maybe in future if they can be stacked like HBM or Ryzen’s extra L3 cache, cuda cores could have more connectivity for their private registers right? I don’t know if this is impossible, but if data goes upwards for tensor-memory and downwards for registers, they wouldn’t compete for connectivity like the warp-shuffle competing with shared-memory access.
By using an entire wafer, at an estimated cost of about $3M for the finished product. I would assume very low yields despite the presumed use of redundancy features. This results in a niche product.
NVIDIA’s game is the creation of a mass-market capable architecture that can cover everything from consumer SKUs retailing for about $500 to high-end datacenter parts selling for around $50K, a price span of two orders of magnitude.
The result: the annual revenue rate of NVIDIA is currently around $250B vs. $1B for Cerebras.
It should be noted that the majority of NVIDIA’s value proposition consist in the gigantic software ecosystem that it both creates and enables. Some years back people coined the term “CUDA moat” for this, but it has since grown far beyond anything directly related to CUDA. NVIDIA realized the value of software early. At least as early as the year 2000, when I first interacted with NVIDIA engineering while working for AMD. It was still a 3D graphics company then that derived most of its revenue from the gaming market. When I retired from NVIDIA in 2014, the company already employed more software engineers than hardware engineers according to their own public statements.
NVIDIA is playing a very long game. General purpose GPU computing took about ten years (2004 - 2014) of continuous investment to contribute a sizeable share of revenue, and another couple of years beyond that before it became the basis for the AI revolution, at which point NVIDIA’s revenue exploded.
Nvidia currently seems to create (or has created) an added hardware advantage in the area of networking between GPUs. In 2020 Nvidia acquired Mellanox and since then has expanded its switching architecture.
The question (within AI) is not anymore who can create the best (performance, efficiency, …) chip, but who can provide the whole infrastructure for the supercomputing data center.
Of course those datacenters also need clever software to “seamlessly” (without specific optimizations by the user) run the same code or networks on single GPUs or whole clusters.
Yes, this is good logic for business to survive or to stay top. Also without software stack, hardware could stay unused like some experiments like Itanium, or Bulldozer, or VLIW, or some Playstation core design. It’s easy for me to talk like that, I’ve nothing to lose, but Nvidia’s risking trillions of $ must be difficult ofcourse.
Register files and shared memory is not on the same level.
Within the SM partitions you have 32 independent lanes.
Data crossing the lane boundary needs extensive logic (many transistors) or a bus or a crossbar architecture to be exchanged.
That is the reason the shuffle instructions are using the same logic as shared memory.
The tensor memory probably (IMHO) will be specific for Blackwell (or even later) datacenter GPUs. It helps accelerating the tensor cores, which are the main application of the datacenter GPUs. Consumer GPUs don’t have as fast tensor cores. Even tensor memory is organized by lanes. It is 128 separate 512 Byte memories per SM.
What is wrong with distributed shared memory for your application? Not enough bandwidth? Too much latency? (The argument with the SM units for consumer GPUs possibly be reduced by 10-15% is not a real argument. Let’s be happy, we got distributed shared memory at all.)
Trying to reduce L2-bandwidth dependency. For example, some data can be cached locally, so that other SM units can get their data from L2 faster. But dsmem has lower bandwidth than L2 cache. So offloading it to dsmem becomes slower, at least some algorithms.
For signaling, sending short message its good but sending an output data to neighbor SM in cluster takes more time than doing for global mem. Even its direction changes performance. Not all SM units have equally fast connection.
Latencywise its doing good, but performance is also sensitive to chunk size sent between sm units. It goes up to 3 TB/s on H100 while L2 has 4 times more bandwidth.
Also similarly TMEM has good bandwidth, but its latency cost makes some algorithms difficult to use it as a scratchpad, on top of register-cost of communicating TMEM.
Some algorithms can’t hide latency, and need low latency of dsmem + high bandwidth of tmem. Register is good but can’t get more capacity.
Distributed shared memory is faster in latency than sending to L2 and polling, until the result is ready and receiving back from L2 on another SM.
Its bandwidth is different between consumer GPUs and datacenter GPUs.
Basically you currently have 1 KiB non-addressable data per thread within a lane (256 registers), 32 KiB for a warp (if you use warp shuffle) and nearly 512 KiB for a SM (if datacenter class GPU), for a whole cluster of 16 SMs it is 8 MiB (only 4 MiB shared memory can be accessed directly, the registers of other threads have to be accessed cooperatively).
Tensor memory on datacenter blackwell increases this to a total of 3 KiB for a single thread, 96 KiB for a warp, 768 KiB for a SM and 12 MiB for a maximum cluster.
Nearly all exchanges beyond a thread have to go through shuffle or shared memory.
TMEM probaby is the nearest for what you need:
extra per lane scratch-space of 2 KiB (64 KiB per SMSP, 256 KiB per SM)
dynamically addressable
data exchange between same lane (and same SMSP), but different warps
→ so mostly the high latency and that it is asynchronous could be problematic
Either one of the following for TMEM would be very good for us:
no register required for general purpose math / reduction / storage to global memory
50% less latency than today
2x capacity (so that I can apply pipelining by cloning capacity to hide latency)
Otherwise, one of these:
2x registers
2x smem
higher dsmem bandwidth (with 16-SM cluster that does not reduce maximum clusters to 7 in H100 due to deactivated circuits for high yield)
I’m sorry for asking too much. This is the shortest description. Adding these would eat from power budget of chip, transistor budget, and R & D cost ofcourse.
It is perfectly fine to ask for enhancements. I worked in some partially customer-facing positions during my industry career, and part of that involved trying to understand what architectural changes in processors would help to accelerate customers’ applications. Things like integer-multiply throughput, cache organization, memory latency, specialized instructions, etc.
In all companies I worked for, customers’ enhancement requests were internally propagated to appropriate departments, collated in lists or databases, and ranked for consideration in future product development.
For any desired improvements, costs such as one-time development cost (NRE charges) and impact on production cost (e.g. increase in die size) have to be weighed against projected increases in TAM (total addressable market) and projected revenue increases in specific product lines.
The customer requests that are most likely to be considered are typically those that can reasonably be construed as being representative of entire industries and not just specific to one vendor. Example: Generally, in image processing, the number of concurrent data streams is such that data caches should be at least 4-way set associative. If the performance impact of the desired enhancement can be quantified, that is even more valuable input. Example: we estimate, across a collection of typical image processing tasks ([list]), that performance would increase by 25% when switching from a 2-way associative cache to a 4-way associative cache.