So has anyone been able to reproduce the increased IMAD performance?
The throughput table seems to have been updated a second time. Now the throughput for CC 12.x is only doubled for compare, minimum, maximum, and 64 instead of multiple instructions for 64-bit integer add compared to smaller CCs.
and doubled warp vote throughput.
It can do 64 x 64-bit integer add, but not 128 x 32-bit integer-add.
Also compare/min/max is actually often implemented with integer addition, but it is faster (128) than integer add.
So, with the white paper still showing Peak INT32 TOPS (non-Tensor) performance twice that of Ada, is it a case of there being an implementation failure somewhere?
Table has been updated again. It now shows throughput of 128 for 32-bit integer addition.
You just beat me to it. I wonder what’s next?
1. Introduction — CUDA C++ Programming Guide
what is 32-bit integer add, extended-precision add, subtract, extended-precision subtract meaning here?
Here is the list of possible operations:
I guess we can draw a line under this dead horse by noting the whitepaper has been amended, page 12:
“INT operation update added in v1.1 of this whitepaper >> Note that the number of possible
integer operations in Blackwell GB20x GPUs are doubled for many integer instructions compared to Ada, by fully unifying the INT32 cores with the FP32 cores, as depicted in Figure 6 below.
However, the unified cores can only operate as either FP32 or INT32 cores in any given clock
cycle. While many common INT operations can run at up to 2x throughput, not all INT operations can attain 2x speedups. For more details, please refer to the NVIDIA CUDA Programming Guide.”
I do not think that it draws a line.
Nobody (?) expected the unified cores to execute a FP32 and a INT32 instruction in the same clock cycle.
(At least not start one. What about pipeline length? Can the pipeline contain both types of instructions at the same time?)
The problem is that not even one INT operation could be benchmarked with 2x throughput or speedup, unlike what the whitepaper and the Programming Guide state.
The added note in the whitepaper rather doubles down on that INT computations would be faster. (E.g. instead of saying it was all a marketing gag or a computational error. ;-) )
Given the Guide, (currently), lists only 32bit integer add/subtract and cmp/max/min with 2x throughput, I don’t think we’ve seen benchmarks targeted there.
There’s Norbert’s mixed one, (LOP3, PRMT, ADD) and a few IMAD/MAD based ones.
Fair enough.
Perhaps we can use that one by tonywu93 and modify the mad into simple additions?
Hi, I’m part of the team responsible for the instruction throughput table in the CUDA programming guide, and I wanted to give you an update on this thread from my point of view:
First, thank you all for raising these issues!
With CUDA 13.0, we will make significant changes to the table: (1) it will be moved to the CUDA best practices guide as it is less relevant to the programming model and (2) it will be re-structured, and will have example PTX instructions which will hopefully provide a little more clarity, although it is obviously far from perfect.
Concerning Blackwell integer instruction throughput specifically:
- As pointed out in the thread already, some instructions have not been improved/changed: this applies to IMAD, LOP3, PRMT for example, as well as IADD3.
- The main improved instructions relevant to this thread are IADD, IMNMX/VIMNMX, FSETP/ISETP: addition of 2 operands and min/max/compare.
- Concerning integer addition specifically, it is even more complicated: previous architectures already had the possibility to achieve 2x throughput by combining e.g. IADD3 with IMAD.IADD or VIADD (for 9.0/10.0). Blackwell 12.0 now allows achieving this 2x throughput with a single instruction: IADD. But note that it can be difficult to get a sequence of instructions achieving higher throughput: for previous architectures, this is because of constraints in the instructions as well as compiler, which we cannot disclose publicly. For Blackwell 12.0, the compiler often outputs IADD3 instead of IADD: this should be improved soon.
Because of the above, it is unlikely that current benchmarks are able to achieve this 2x throughput, unless they are hand-crafted without relying on the compiler.
Note that this is the case for some other entries in the instruction throughput table: it only lists the theoretical maximum throughput, but we cannot always disclose how to precisely achieve this in practice.
If you have further questions, I can try answering them.
Thanks for the clarification. That’s quite a sad conclusion to the saga.
Many thanks for clarifying and the offer for additional information.
I agree that the actual feature is a bit underwhelming after the announcements. So is the current support by nvcc and ptxas for it.
So there are more SASS opcodes now, which can be processed by more than one type of pipe. We kind of have something like it with FMAheavy and FMAlite.
Even with the compiler tuning code generation towards the higher throughput operations, I would not expect any major speed-ups resulting from this hardware change.
Outside of sorting networks, I am not aware of any code bottlenecked on IMIN/IMAX, and I don’t know any code bottlenecked on comparison operators (ISETP, FSETP). I assume the fact that these operations now have higher throughput is not because of a targeted hardware optimization but simply a side-effect of introducing a high-throughput two-input IADD, i.e. these operations share the same ALUs.
Based on the description provided, splitting IADD3 with three non-zero source operands into two IADDs would never make sense, since the former offers the same throughput but lower dynamic instruction count. In terms of register pressure there could be a slight advantage either way, but overall its likely a toss-up.
So the only optimization that seems to make sense is to convert all current instances of IADD3 with an RZ source operand into a two-input IADD. But in code dominated and bottlenecked by integer addition, I would expect a high percentage of IADD3 to have no zero (RZ) operand, lessening the impact of this transformation. Conversely, where there are few integer additions and most IADD3 at present likely have a zero source, any positive performance impact will be limited by the small number of integer additions.
Overall I would expect this hardware change to result in real-life performance improvements to integer-intensive code in the single-digit percentage range if and when fully exploited by the compiler.
One thing I’ll add to this is that the ability to have a higher-throughput two-operand addition simplifies compiler optimization quite a bit, which is why we expect many more workloads to benefit from this, especially kernels with a mix of IADD3 / IMAD.IADD / VIADD / IMUL: this could for example apply to short-lived memory-bound kernels where the first cycles are almost always bound by address calculations (note that latency hiding through occupancy cannot ramp up instantly out of the blue).
I agree that this will not be a 2x kind of improvement, but making a small improvement for many kernels is valuable.
Sorting networks and some integer-arithmetic-bound kernels will benefit more directly, because it is not always possible to have all integer additions with 3 operands (again, it is often a mix of 3-operand and 2-operand addition, so not a 2x benefit, but it can be significant).
Great! So these compiler changes will be coming in CUDA 13?
Currently my benchmarks still show no change in performance for IADD with CUDA 12.9.
I’m not sure whether all optimizations will make it into CUDA 13, but the most important ones are indeed coming with 13.0, driver 580.