FP64 Performance - Power Limitation - H100 vs A100

Accelerated Computing CUDA CUDA Programming and Performance
1

Hello,

You might not get the theoretical 3x FP64 performance increase for H100 over A100 since the Power Ratio is only 700/400 = 1.75 (even if you have enough memory bandwidth and with process size being more efficient 4 nm vs 7nm).

I pinged Yves Gallot creator of the Integer NTT program Genefer from PrimeGrid and also did some searches in this forum and read the same.

That is why for 7.5MB FP64 FFT for PRPLL NTT the iteration time ratio was only 1.92 and not 3.

Note: PRPLL NTT is OpenCL so I am unable to use Nsight Compute.

At Google Colab (colab.research.google.com), you can rent the A100 or now also the new, hard to get H100.

! nvidia-smi reveals the following versions:

NVIDIA A100-SXM4-40GB (400 Watts)

NVIDIA H100 80GB (700 Watts)

Here are the complete details about both straight from the official H100 whitepaper:

https://resources.nvidia.com/en-us-hopper-architecture/nvidia-h100-tensor-c

L2 cache size:
A100: 40MB
H100: 50MB

Memory Bandwidth
A100: 1555 GB/sec
H100: 3352 GB/sec

FP64 TFLOPS
A100: 9.7
H100 33.5

On Google Colab, A100 has an iteration time of 412 microseconds after tuning with latest version of PRPLL NTT (that supports both FP64 FFT and Integer NTT) using 7.5MB FP64 FFT (INPLACE).
On Google Colab, H100 has an iteration time of 215 microseconds after tuning with latest version of PRPLL NTT (that supports both FP64 FFT and Integer NTT) using 7.5MB FP64 FFT (INPLACE).

If my math is correct, memory bandwidth calculations (ignoring the twiddle factors storage and bandwidth)

A100: 7.5 MB * 8 bytes (FP64 FFT) * 4 pass * 2 (read/write) [4 passes per iteration, each pass a read and write now INPLACE, not buffer to buffer] / 1024 * 1000000 / 412 = 1137.74 GB / sec
H100: 7.5 MB * 8 bytes (FP64 FFT) * 4 pass * 2 (read/write) [4 passes per iteration, each pass a read and write now INPLACE, not buffer to buffer] / 1024 * 1000000 / 215 = 2180.23 GB / sec

Similarly, Yves said “We can see this limitation with Zen-5 and AVX-512. In theory, each core is able to execute 2 FMA and 2 FADD per clock; but if a benchmark executes this code then the clock frequency drops. It is faster than Zen-4 but +30/40% and not 2x.”

Thus Nvidia advertises 3x FP32 and FP64 increase and 2x memory bandwidth increase of H100 over A100, but even with plenty of memory bandwidth available the 3x FP64 increase may not happen since power increase is only 1.75x.

I have a follow up question to Yves to see if the above is related since I always wondered why A100 for FP64 wasn’t even more faster than consumer RTX 4060. RTX 4060 with FP64:FP32 ratio 1:64 has 236.2 FP64 GFLOPS. (4060 Memory Bandwidth is 272 GB/sec). So comparing A100 to RTX 4060 on just theoretical FP64 FLOPS yields 9700/236.2 = 41.1 ratio. Yet RTX 4060 7.5MB FP64 is 5959 microseconds per iteration so it’s “only” 5959/412 = 14.46 slower and not 41.1. Thankfully now that PRPLL support integer NTT using for same size number 4MB M31+M61 integer NTT (12 bytes) brings its iteration time down to 1421 microseconds per iteration (memory bound 4 * 12 * 4 * 2 / 1024 * 1000000 / 1421 = 263.898 GB / sec).

Before INPLACE improvement, the cache improvement from George Woltman was “PRPLL does 4 passes over the data. Previously, each pass accessed all the data. Now one pass accesses all the data (unavoidable), then PRPLL does 3 passes on the M31 data, then three passes on the M61 data. The three M31 passes access 4MB * 4 bytes * 2 buffers = 32MB. The three M61 passes access 64MB. If the L2 cache size is between 32MB and 64MB these three passes can operate out of the cache (either partly or mostly).” Now, in addition INPLACE has been implemented and both help out even RTX 4060.

Great Internet Mersenne Prime Search (GIMPS)

mersenne.org

Current Version: Windows PRPLL NTT

https://www.dropbox.com/scl/fi/s8q2v1v8j0q3grcwaw3dd/prpll-win-20251209.zip?rlkey=zq46cmn58yie9s0mppb37buii&dl=0

Current Version: Linux PRPLL NTT

https://www.dropbox.com/scl/fi/y5cnumf4w3rc3kf8o0ift/prpll-linux-20251209.gz?rlkey=qrjhftun68xyawd8gu0fao42c&dl=0

https://github.com/gwoltman/gpuowl

https://github.com/preda/gpuowl

2

Why do you assume that power/performance stayed the same, even with a different semiconductor process?

Power efficiency was one of the advertised characteristics, when the last generations were introduced.

3

Good Point.

Why is H100 FP64 compute performance not higher over A100, especially with plenty of memory bandwidth?

4

Memory bandwidth

=================

Is memory performance plenty? The stated memory bandwidth is under theoretical optimum conditions.

You can run FFT with Cuda code and see Nsight Compute output.

SMs compared

============

The H100 does not have (beside higher frequency) double the SMs of the A100, but each SM can execute FP64 computations faster.

64/cycle instead of 32/cycle (the same speed doubling is true for FP64 on Tensor Core: 128/cycle for H100 instead of 64/cycle for A100).

That means all resources per SM need double the realized performance to serve the higher FP64 performance:

Intermittent non-FP64 instructions, data loading/storing, register pressure, caches, shared memory.

You could - even for OpenCL programs - fix the frequency of both at a low clock and compare performance to see whether it is thermal/power overload or an architectural limit (the program would have to be optimized better for H100 in that case).

5

To get close to the theoretical peak performance you are referring to (whether on A100 or H100), you would need basically a matrix multiply code. Suggesting that another algorithm or computational sequence also meets the necessary criteria is immediately doubtful to me, based on my own experience.

The vast majority of codes are memory bound.

The 1.92 ratio you report between the two GPUs would be consistent with the memory bandwidth ratio, and as you have already pointed out, inconsistent with the referenced compute throughput ratio.

If I wanted to test the claim I am implying here, since FP64-non-TC cannot be easily accessed via libraries like CUBLAS (on A100 and H100), I would seek to construct a code that nevertheless had many back-to-back DFMA sequences, and see how that compares between the two GPUs.

6

@Robert_Crovella - What’s your thought process about the below reported H200 iteration time?

On https://www.mersenneforum.org/ a user on the PRPLL NTT spreadsheet tracking performance data over various GPUs report a NVIDIA H200 7.5MB FP64 FFT iteration time of 193 microseconds per iteration.

They reported using the Nov. 16th and not Dec. 9th version that had INPLACE. Thus for 7.5MB FP64 FFT to fit entirely in cache for the Nov. 16th buffer to buffer version you need 120MB (unlike the later INPLACE version that needed half at 60MB to fit in cache - both have same memory bandwidth). A100 iteration time between the versions was small: 432 (Nov. 16) vs 412 (Dec. 9).

Anyway, the memory bandwidth calculation for H200 is (ignoring the twiddle factors storage and bandwidth):

H200: 7.5 MB * 8 bytes (FP64 FFT) * 4 pass * 2 (read/write) [4 passes per iteration, each pass a read and write, buffer to buffer] / 1024 * 1000000 / 193 = 2428.756 GB / sec

Thus H200 to A100 ratio (Nov. 16th version) is 432 / 193 = 2.238 x

H200 has same theoretical Teraflops as H100 but theoretical memory bandwidth is now 4.8 Terabytes / second.

FP64 TFLOPS
A100: 9.7
H200 34

Memory Bandwidth

A100: 1555 GB/sec
H200 4800 GB/sec

H200 / A100 FP64 ratio 34/9.7 = 3.505 x

H200 / A100 memory bandwidth ratio = 4800 / 1555 = 3.0868 x

H200 to A100 iteration time ratio = 2.238 x

What’s your sense about what this data means?

7

For FFTs in particular, it has been known for a long time that they are bound by memory bandwidth (at whatever the applicable level of the memory hierarchy is), and that has also applied to FFTs on GPUs for the past twenty years.

The issues with x86-64 CPUs downclocking under intensive use of AVX-512 is a completely different issue and to my knowledge there is no direct equivalence in the GPU world. That said, it is possible to hit either the power or thermal walls on various NVIDIA GPUs, and that results in lower clocks being used. nvidia-smi is the easiest way to diagnose such an issue, and aggressive cooling (that usually means liquid cooling) can incrementally improve performance in such situations. I do not have a comprehensive overview, but in scenarios I have seen, “incremental improvement” equated to 5% to 6%.

8

So the H200 and H100 have basically the same FP64-non-TC performance(&). And yet when you compare H200 to H100, on your code, the H200 is faster by 2.238/1.95 = 1.15 (or 215/193 if you prefer, it results in almost the same ratio - 1.14). I would say based on that data point, memory performance could certainly matter, and FP64 performance does not play a role in explaining the speedup/difference. If the code were FP64 bound, I would expect a best case increase of 34/33.7 = 1.01. (I’m just using numbers from this thread. I actually think the FP64 predicted speed up ratio is 1.0).

(& yes, its true: 1 2 And it is worth noting, at least if we look at the SXM variants, there is no difference in the max power envelope between H100 and H200, therefore positing a power difference to explain the perf difference also doesn’t hold up, in my view. The 34TF numbers for FP64 throughput bouncing around in this thread would pertain to the SXM variants.)

9

In practice this also means FP64 comparisons should always report power cap and achieved clocks, not just nominal specs.

10

The power cap probably is no issue. But the frequencies and thermal load yes.

This should be true for any benchmark, not just for FP64 measurements.

Alternatively set the clock to the base clock to have stable comparable results.

Or normalize results by dividing by clock and SM count for compute heavy results.

11

I agree @linzheng0428 and @Curefab – but I am a “beginner” with Google Colab and not a true tech person like you two are and don’t know if possible to run 2 things simultaneously so I can see this data.

The PRPLL NTT performance tracker spreadsheet they maintain has all the necessary columns for users to input (Core Clock, Mem Clock, Total Power, Transform Type, Operating System, etc).

Thus when I entered my data for RTX 4060 Integer NTT data, for instance, that yielded 1421 microseconds per iteration time with PRPLL NTT (GPU and CPU fans on full power, all non-Windows background processes in Task Manager that can be closed, closed), I used Nvidia App, HWMonitor, etc to get this data. In my case my RTX 4060 GPU in my HP Victus desktop tower using Windows 11 was running at 2700 Mhz, Mem was 8251 Mhz, and Power was 115 Watts.

The user who reported 193 microseconds per iteration using FP64 FFT on LInux on H200 reported 1980 GPU Core Clock, 3200 Mem Clock, and 700 W Power. I don’t know if they measured that or simply got specs somewhere like Nvidia’s site or TechPowerUp. May be legit since for FP64 FFT my H100 on Linux Colab was 215.

For the NVIDIA A100-SXM4-40GB and NVIDIA H100 80GB HBM3 on Google Colab, while I can use, for instance, ! nvidia-smi -q -d CLOCK,POWER I have to stop PRPLL NTT on the Jupyter notebook and run the command right after. If you can guide me if it is possible to run PRPLL NTT on Colab and also run the ! nvidia-smi command at same time like I can on Win 11 desktop?

12

@njuffa - Yeah your probably right and I think it’s easier to compare iteration time ratios between GPUs (after factoring in things like L2 cache size and memory bandwidth) when you use NTT instead of FFT.

That is what I did in that other thread over here that you are on also:

https://forums.developer.nvidia.com/t/integer-ntt-on-rtx-20xx-a100-vs-rtx-30xx-40xx-50xx/350124/23?u=wfgarnett3

As you know, for Nvidia Integer MAD runs on FP32 while other Integer Operations runs on INT32 for NTT for PRPLL NTT where the code is roughly 50 / 50 split.

As I posted in that thread, A100 architecture (streaming multiprocessor of 64 CUDA cores) supports concurrent FP32/INT32 so you can take the FP32 FLOP number and multiple by 2 for our Integer NTT.

For 4060 for Integer NTT you can use the 15.11 FP32 number because its streaming multiprocessor of 128 CUDA Cores has half that do FP32 only and the other half that can do FP32 or INT32. So for 50/50 code used in the NTT, the FP32 only CUDA cores can do FP32 (Integer MAD), and the FP32 or INT32 cores can do INT32 instructions, giving same 15.11 Tera Operations Per Second value.

Using the Nov. 16 versions of PRPLL NTT (M31 + M61 integer NTT):

A100/4060 Memory Bandwidth Ratios
1555/272=5.72

Compute Ratios
38.98/15.11=2.58

A100/4060 timings Ratios (M31*M61 NTT)

1M M31*M61 NTT (data size fits in L2 cache of both GPUs)
294.8/150.2=1.96

2M
713.4/279.2=2.56

4M (World Record 140M Exponent Mersenne Prime Candidate)
1608.7/535.1=3.01

64M
30112.9/9547.9=3.15

Everything for NTT between 4060 and A100 lines up with expected 2.58 integer compute ratio for that Nov. 16 version, even with the large 5.72 memory bandwidth ratio since as you say NTT is not as memory bound as FFT.

When using 1M transform that fits in the L2 cache for both, iteration time ratio is 1.96

At the extreme end for 64M where its memory bound for 4060 (30112.9 microseconds per iteration means 199.25 GB/sec out of theoretical 272 GB/sec), the ratio is 3.15 which is expected. A100 iteration value of 9547.9 corresponds to 628.410 GB/sec so it’s not memory bound.

Is your understanding the same, it may make sense to compare Operations Per Second ratios when using NTT instead of FFT?

13

@Robert_Crovella @Curefab @njuffa @linzheng0428

I got the H100 again at Google Colab and here are the complete files for H100 for the entire exponent range for both FFT and NTT in case you are interested.

When I stop the program and run nvidia-smi -q -d POWER,CLOCK it looks like GPU clock and memory clock are as advertised, and power sampling when stopping program shows a max power sample near 700 W on H100 (653 W).

Testing Mersenne Prime 136279841 on PRPLL NTT on Google Colab after tuning (either 7.5 MB FP64 FFT or 4 MB M31+M61 Integer NTT)

NVIDIA H100 80GB HBM3
NVIDIA A100-SXM4-40GB

7.5 MB FP64 FFT

H100: 215 microseconds per iteration
Memory Bandwidth not including twiddle factors storage and bandwidth: 2180.23 GB / sec [ 7.5 * 8 * 4 * 2 / 1024 * 1000000 / 215 ]

A100: 412 microseconds per iteration
Memory Bandwidth not including twiddle factors storage and bandwidth: 1137.74 GB / sec [ 7.5 * 8 * 4 * 2 / 1024 * 1000000 / 412 ]

4 MB M31+M61 Integer NTT

H100: 298 microseconds per iteration ( NO_ASM )
Memory Bandwidth not including twiddle factors storage and bandwidth: 1258.39 GB / sec [ 4 * 12 * 4 * 2 / 1024 * 1000000 / 298 ]

A100: 525 microseconds per iteration
Memory Bandwidth not including twiddle factors storage and bandwidth 714.29 GB / sec [ 4 * 12 * 4 * 2 / 1024 * 1000000 / 525 ]

H100_FP64_20260101_gpuowl-0 and H100_NTT-NO_ASM_20260101_gpuowl-0.log (34.7 KB)

H100_FP64_20260101_tune.txt (4.3 KB)

H100_FP64_20260101_config.txt (526 Bytes)

H100_NTT-NO_ASM_20260101_tune.txt (939 Bytes)

H100_NTT-NO_ASM_20260101_config.txt (578 Bytes)

14

I figured out the answer to my other question above “Yet RTX 4060 7.5MB FP64 is 5959 microseconds per iteration so it’s “only” 5959/412 = 14.46 slower and not 41.1.”. As with most applications today there is plenty of compute power and these applications can be memory bound. So in this case regardless of the theoretical 41.1 times FP64 FLOP value of the A100 vs RTX 4060 the iteration time ratio will be smaller since memory bound on A100.

I tested this by downloading for the first time Nsight Compute on my 4060 desktop, and also using it on the A100 at Google Colab where I downloaded the output report so I can view on my desktop Nsight Compute GUI. Since PRPLL NTT is OpenCL (and not CUDA) I used the old CUDALucas program which is 1.5x slower than PRPLL NTT when doing Lucas-Lehmer testing of Mersenne numbers with FP64 FFT.

It’s really cool seeing these technical concepts in action with this profiler tool.

@Robert_Crovella - not to beat a dead horse but for FP64 my Nov. 22 comment at the first link below “Even if it is 1:16 or 1:8 ratio on consumer GPUs instead of 1:2 that would be such a benefit over 1:64 ratio.” is so true. Being memory bound you won’t need 1:2 on a consumer GPU - 1:8 or 1:16 ratio would be great without physically needing all the compute units 1:2 entails. Where our project was able to do integer NTT on consumer GPUs because of talented developers who created these prime number search projects other types of FP64 work done by others on consumer GPUs may not have that luxury to develop workarounds. It would be nice in the future for consumer GPUs to have a better ratio; even on the data center side I read the recent news articles where data center HPC customers (1:2) are not satisfied with the small FP64 gains generation over generation.

https://forums.developer.nvidia.com/t/integer-ntt-on-rtx-20xx-a100-vs-rtx-30xx-40xx-50xx/350124

https://www.hpcwire.com/2025/12/09/nvidia-says-its-not-abandoning-64-bit-computing/