Thor DriveAGX Maximum Load Test

DRIVE OS Version: DRIVE OS Linux version 7.0.2

Hi All,

I would like to measure the maximum power consumption of the Thor DriveAGX platform by applying the highest possible load on the Thor chipset. My goal is to utilize all available resources within the chip, including CPU, GPU, and DDR bandwidth, while monitoring power consumption.

I have the following questions:

  1. I am running a simple CUDA matrix operation, and tegrastats reports GR3D_FREQ at 99%. However, the GPU temperature hardly increases. How can I verify whether the GPU is actually under full load?

  2. As I understand, Thor can deliver up to approximately 1,000 TOPS of INT8 performance. If I only run the CUDA matrix workload and observe a high GR3D_FREQ value in tegrastats, does that mean the GPU is being loaded close to its maximum INT8 processing capability?

  3. Does NVIDIA provide any reference test applications or benchmark tools for conducting this type of heavy-load and maximum power consumption testing?

Thank you for your support.

Dear @sk.lee ,
Could you please check the power measurement tool at Power Measurement Tool — NVIDIA DriveOS 7.2.5 Linux SDK Early Access Developer Guide

Dear @SivaRamaKrishnaNV

Thank you for share PowerManagement tool.

By the way, could you please let me know about Q2 which GR3D_FREQ and full TOPS support?

Just to clarify, if GR3D_FREQ is 70%, would it be reasonable to interpret that as roughly 70% utilization of the available INT8 performance (e.g., about 700 TOPS out of 1000 TOPS)?

Regards,

sk.lee

It can be like if CPU/GPU utilisation is 90%, it indicates, out of 100 samples that are collected, 90 samples indicates CPU/GPU is used.
Could you also check User Guide — Nsight Systems helps?

From the NSight, the GPU load seems 100% while runnign very simple CUDA matrix multiplier (attached Nsight analysis capture)

But I don’t know this is really same with 1,000 TOPS usage and this is right method to generate full power comsumption.

Does your CUDA matrix multiplier uses INT8 operations?
Can you check running any DL model in INT8 or samples like cuda-samples/cpp/3_CUDA_Features/immaTensorCoreGemm at v13.3 · NVIDIA/cuda-samples · GitHub to check INT8 usage.

As recommended, I ran several additional GEMM-related tests.

For INT8, I added and tested the immaTensorCoreGemm sample: cuda-samples/cpp/3_CUDA_Features/immaTensorCoreGemm at v13.3 · NVIDIA/cuda-samples · GitHub

Since my primary goal is not only performance benchmarking but also measuring power consumption under full load, I additionally ran FP8 GEMM tests using CUTLASS.

According to the Nsight results, both tests maintained a GPU clock of 1.54 GHz, while GR Active remained at 99.9% throughout execution.

Given these results, I would like to confirm whether I am effectively driving the Thor GPU to its maximum performance capability. If so, would it be reasonable to consider these benchmarks as achieving the full INT8 or FP8 1,000 TOPS performance that Thor is specified to deliver?

Below are the results from the FP8 GEMM test, showing how many GEMM iterations were completed per second.

The test was performed using the following CUTLASS example:

The purpose of this measurement was to monitor the sustained execution rate of the GEMM workload under continuous load conditions and to correlate it with power consumption and GPU utilization metrics.

I understand that there is a comment indicating that the examples under cutlass/examples are not suitable for benchmarking. Nevertheless, I would like to confirm whether it is expected that the performance converges to approximately 120 TFLOPS, as shown below.

  Load mode: looping FP8 GEMM 8192x8192x8192 for 60s
  [ 1s]     64 GEMMs  70.4 TFLOPS (64 GEMM/s)
  [ 2s]    192 GEMMs  105.6 TFLOPS (96 GEMM/s)
  [ 3s]    256 GEMMs  93.8 TFLOPS (85 GEMM/s)
  [ 4s]    384 GEMMs  105.6 TFLOPS (96 GEMM/s)
  [ 5s]    512 GEMMs  112.6 TFLOPS (102 GEMM/s)
  [ 6s]    640 GEMMs  117.3 TFLOPS (107 GEMM/s)
  [ 7s]    704 GEMMs  110.6 TFLOPS (101 GEMM/s)
  [ 8s]    832 GEMMs  114.3 TFLOPS (104 GEMM/s)
  [ 9s]    960 GEMMs  117.3 TFLOPS (107 GEMM/s)
  [10s]   1088 GEMMs  119.6 TFLOPS (109 GEMM/s)
  [11s]   1152 GEMMs  115.1 TFLOPS (105 GEMM/s)
  [12s]   1280 GEMMs  117.3 TFLOPS (107 GEMM/s)
  [13s]   1408 GEMMs  119.1 TFLOPS (108 GEMM/s)
  [14s]   1536 GEMMs  120.6 TFLOPS (110 GEMM/s)
  [15s]   1600 GEMMs  117.3 TFLOPS (107 GEMM/s)
  [16s]   1728 GEMMs  118.7 TFLOPS (108 GEMM/s)
  [17s]   1856 GEMMs  120.0 TFLOPS (109 GEMM/s)
  [18s]   1984 GEMMs  121.2 TFLOPS (110 GEMM/s)
  [19s]   2048 GEMMs  118.5 TFLOPS (108 GEMM/s)
  [20s]   2176 GEMMs  119.6 TFLOPS (109 GEMM/s)
  [21s]   2304 GEMMs  120.6 TFLOPS (110 GEMM/s)
  [22s]   2432 GEMMs  121.5 TFLOPS (111 GEMM/s)
  [23s]   2496 GEMMs  119.3 TFLOPS (109 GEMM/s)
  [24s]   2624 GEMMs  120.2 TFLOPS (109 GEMM/s)
  [25s]   2752 GEMMs  121.0 TFLOPS (110 GEMM/s)
  [26s]   2880 GEMMs  121.8 TFLOPS (111 GEMM/s)
  [27s]   2944 GEMMs  119.9 TFLOPS (109 GEMM/s)
  [28s]   3072 GEMMs  120.6 TFLOPS (110 GEMM/s)
  [29s]   3200 GEMMs  121.3 TFLOPS (110 GEMM/s)
  [30s]   3328 GEMMs  122.0 TFLOPS (111 GEMM/s)

Could you check the suggestion from 【Jetson Thor】Cutlass FP4/FP8/FP16 Performance Test if it helps.

Thank you for sharing another thread for getting maximum power consumption similar with me.

But as I mentioned before I’m already testing with CUTLASS GEMM test to make GPU running fully but I don’t know it’s really full load or not.

From the GEMM test result, it shows only about 120 TFLOPS but Thor-X datasheet tells over 2,000TFLOPS with FP8.

So I wonder there is no power limitation or performance blocker like ‘nvpmodel’ from Jetson.

Thank you

Could you please provide any update for this topic?

DRIVE platform always runs in performance mode unlike Jetson where we can configure mode using nvpmodel.

Could you please provide any update for this topic?

Could you please provide any update for this topic?

Hi, I’m not received update yet too.