Thor torch.mm benchmark results (float32/float16/float8_e3m2fn)

Robotics & Edge Computing Jetson Systems Jetson Thor
, ,
1

Float32 mamf

Jetson Thor

** Command line:
/home/ptrck/pytorch/venv/bin/python mamf-finder.py --m_range 0 4096 512 --n_range 0 4096 512 --k_range 0 4096 512 --dtype float32

** Dtype: torch.float32

** Platform/Device info:
Linux flatbrick 6.8.12-tegra #1 SMP PREEMPT Thu Aug 21 17:27:43 PDT 2025 aarch64 aarch64
_CudaDeviceProperties(name='NVIDIA Thor', major=11, minor=0, total_memory=125772MB, multi_processor_count=20, uuid=, pci_bus_id=1, pci_device_id=0, pci_domain_id=0, L2_cache_size=32MB)

** Critical software versions:
torch=2.9.0a0+gitec2c137
cuda=13.0

** Additional notes:
benchmark version: 2


--------------------------------------------------------------------------------


Warming up the accelerator for 30 secs ... accelerator warmup finished

Tried  343 shapes => the best outcomes were:
mean:   5.3 TFLOPS @ 1536x2560x2048 (MxNxK)
median: 5.3 TFLOPS @ 1536x2560x2048 (MxNxK)
max:    5.3 TFLOPS @ 1536x2560x2048 (MxNxK)

geomean: 4.6 TFLOPS for 343 shapes in range: m=[0, 4096, 512] | n=[0, 4096, 512] | k=[0, 4096, 512]

Legend: TFLOPS = 10**12 FLOPS
Elapsed time: 0:05:08

Blackwell 6000

Tried  343 shapes => the best outcomes were:
mean:   65.9 TFLOPS @ 2560x3584x2048 (MxNxK)
median: 66.0 TFLOPS @ 2560x3584x2048 (MxNxK)
max:    66.3 TFLOPS @ 2560x3584x2048 (MxNxK)

geomean: 46.9 TFLOPS for 343 shapes in range: m=[0, 4096, 512] | n=[0, 4096, 512] | k=[0, 4096, 512]

Legend: TFLOPS = 10**12 FLOPS
Elapsed time: 0:00:55

Float16

Jetson Thor

Tried  343 shapes => the best outcomes were:
mean:   155.5 TFLOPS @ 3584x3072x3584 (MxNxK)
median: 156.0 TFLOPS @ 3584x3072x3584 (MxNxK)
max:    163.3 TFLOPS @ 3584x3072x3584 (MxNxK)

geomean: 66.5 TFLOPS for 343 shapes in range: m=[0, 4096, 512] | n=[0, 4096, 512] | k=[0, 4096, 512]

Legend: TFLOPS = 10**12 FLOPS
Elapsed time: 0:01:48

Blackwell 6000

If I increase the searchable range this is over 400, but keeping it fair

Tried  343 shapes => the best outcomes were:
mean:   357.7 TFLOPS @ 1536x3584x1536 (MxNxK)
median: 359.0 TFLOPS @ 1536x3584x1536 (MxNxK)
max:    363.2 TFLOPS @ 1536x3584x1536 (MxNxK)

geomean: 215.0 TFLOPS for 343 shapes in range: m=[0, 4096, 512] | n=[0, 4096, 512] | k=[0, 4096, 512]

Legend: TFLOPS = 10**12 FLOPS
Elapsed time: 0:00:41

Float8_e4m3fn

Jetson Thor

Tried  343 shapes => the best outcomes were:
mean:   276.5 TFLOPS @ 3584x3072x2048 (MxNxK)
median: 278.8 TFLOPS @ 3072x3584x2048 (MxNxK)
max:    288.1 TFLOPS @ 3072x3584x2048 (MxNxK)

geomean: 88.6 TFLOPS for 343 shapes in range: m=[0, 4096, 512] | n=[0, 4096, 512] | k=[0, 4096, 512]

Legend: TFLOPS = 10**12 FLOPS
Elapsed time: 0:02:03

Blackwell 6000

Tried  343 shapes => the best outcomes were:
mean:   658.4 TFLOPS @ 2048x2560x3072 (MxNxK)
median: 659.4 TFLOPS @ 3584x1536x3072 (MxNxK)
max:    690.4 TFLOPS @ 2048x2560x3072 (MxNxK)

geomean: 353.8 TFLOPS for 343 shapes in range: m=[0, 4096, 512] | n=[0, 4096, 512] | k=[0, 4096, 512]

Legend: TFLOPS = 10**12 FLOPS
Elapsed time: 0:00:40

pytorch’s support of float4 is too broken atm to actually benchmark in this fashion atm on either platform

3

Hi,
By default AGX Thor runs in 120W mode:

Jetson Thor Product Family — NVIDIA Jetson Linux Developer Guide

Please execute the command to do profiling in MAXN mode:

$ sudo nvpmodel -m 0
$ sudo jetson_clocks

There may be some improvement.

4

This was with mode 0 MAXN and jetson_clocks. These numbers are about where I expected them; before I got mine I was curious about these numbers so figured I would share.

The only surprise was how bad the 6000 Blackwell f16 performance was relative to the Thor but that’s off topic for this thread.

5

Just out of morbid curiosity I expanded the search space on jetson to see how much of the peak was left untapped. These are unlikely geometries for me to use just due to the limited memory bandwidth but might be of interest to someone. Also, with these larger matrix’s in MAXN mode (Peak power draw was reported at 136w).

Screenshot From 2025-09-08 23-21-03
Screenshot From 2025-09-08 23-21-032473×942 480 KB

I also realized I didn’t share these relevant settings previously, they are the same for these numbers

  matmul.allow_fp16_reduced_precision_reduction: True
  cudnn.allow_tf32: True
  matmul.allow_tf32: True

float8_e4m3fn

Tried  60 shapes => the best outcomes were:
mean:   341.9 TFLOPS @ 4096x4096x4096 (MxNxK)
median: 351.4 TFLOPS @ 4096x4096x8192 (MxNxK)
max:    380.2 TFLOPS @ 4096x12288x4096 (MxNxK)

geomean: 204.1 TFLOPS for 60 shapes in range: m=[0, 8193, 4096] | n=[0, 65536, 4096] | k=[0, 8193, 4096]

Legend: TFLOPS = 10**12 FLOPS
Elapsed time: 0:02:51

float16

Tried  60 shapes => the best outcomes were:
mean:   160.0 TFLOPS @ 4096x4096x4096 (MxNxK)
median: 162.3 TFLOPS @ 4096x4096x4096 (MxNxK)
max:    172.1 TFLOPS @ 4096x4096x4096 (MxNxK)

geomean: 93.2 TFLOPS for 60 shapes in range: m=[0, 8193, 4096] | n=[0, 65536, 4096] | k=[0, 8193, 4096]

Legend: TFLOPS = 10**12 FLOPS
Elapsed time: 0:05:14

float32

Tried  12 shapes => the best outcomes were:
mean:   5.8 TFLOPS @ 8192x12288x8192 (MxNxK)
median: 5.8 TFLOPS @ 8192x12288x8192 (MxNxK)
max:    5.8 TFLOPS @ 8192x12288x8192 (MxNxK)

geomean: 5.5 TFLOPS for 12 shapes in range: m=[0, 8193, 4096] | n=[0, 16384, 4096] | k=[0, 8193, 4096]

Legend: TFLOPS = 10**12 FLOPS
Elapsed time: 0:03:56
6

Hi,

Thanks for sharing the result.

We recommended benchmarking Thor’s performance with LLM.
Matrix multiplication depends more on the memory bandwidth since the calculation is relatively simpler.

For example, below are some LLMs scores from our side:

Thanks.