I dont achieve peak performance of 4090 in Pytorch

Accelerated Computing CUDA CUDA Programming and Performance
1

I’m unable to achieve peak matmul performance on an RTX 4090 (FP16/FP16) using PyTorch (eager, JIT, and AOT) as well as custom CUDA kernels.

I’ve implemented several approaches and benchmarks, but the achieved throughput is still significantly below the expected theoretical performance for this GPU.

This is the repository I’ve been working on:

would really appreciate any insights on:

  • Potential bottlenecks (compute vs memory vs kernel launch)

  • PyTorch configuration or flags I might be missing

  • CUDA kernel optimization strategies for FP16 tensor cores

  • Any known limitations or pitfalls on Ada Lovelace GPUs

I evaluated hardware response under increasing workloads. This included small (1024, 2048), medium (4096, 8192), and large (16384, 32768) matrix sizes.

The second phase analyzed non-square matrices under a “partial workload” scheme with sustained arithmetic intensity. For this, the inner dimension was fixed at K = 8192.

Thanks in advance for your help!