GA10B GPU Metrics (DRAM Bandwidth) returns all zeros in Nsight Systems on Jetson Orin Nano — limitation or configuration issue?

Robotics & Edge Computing Jetson Systems Jetson Orin Nano
1

Background:

I am profiling a model on the Jetson Orin Nano dev kit for a memory-bandwidth study. The workload executes a GPU-resident model on the Orin Nano’s . I am trying to quantify how much DRAM bandwidth consumes during execution, I need per-source DRAM throughput measurements.

Environment:

  • Jetson Orin Nano Developer Kit (8 GB)
  • JetPack 6.2.2 (L4T R36.5.0)
  • CUDA 12.6
  • Nsight Systems 2024.5.4
  • Nsight Compute 2024.3.1.0
  • GPU: Ampere GA10B (1024 CUDA cores, 32 Tensor Cores)
  • Device path: /sys/devices/17000000.ga10b/devfreq/17000000.ga10b/

The ga10b-gfxt GPU metrics config file at /opt/nvidia/nsight-systems/2024.5.4/target-linux-tegra-armv8/GpuMetrics/ga10b-gfxt.config defines exactly the metrics I need. I also noted that the ga10x.config file defines GPU-internal DRAM counters (dram__read_throughput, dram__write_throughput) which are absent from the ga10b.config file.

I tried the following

  • captured a profile with ga10b-gfxt at 10 kHz as below

sudo nsys profile --trace=cuda,nvtx,osrt --gpu-metrics-device=0 --gpu-metrics-set=ga10b-gfxt --gpu-metrics-frequency=10000 --sample=cpu --output=run_dram ./myscript.py

  • Verified the GPU_METRICS table exists with the correct metric definitions in TARGET_INFO_GPU_METRICS
metricId 0 → CPU-L3 Reads [Throughput %] 

metricId 1 → CPU-L3 Writes [Throughput %] 

metricId 2 → GPU Reads [Throughput %] 

metricId 3 → GPU Writes [Throughput %] 

metricId 4 → SoC Peripheral Reads [Throughput %] 

metricId 5 → SoC Peripheral Writes [Throughput %]
  • Queried the actual values , shows AVG=0.0 and MAX=0 across allsamples
  • Verified that other GPU compute metrics in the same capture do have non-zero values
metricId 6 (GPU Active): AVG=34.1%, MAX=100% 

metricId 10 (SM Throughput): AVG=4.9%, MAX=86% 

metricId 91 (L1 Hit Rate): AVG=4.0%, MAX=100% 

metricId 101 (L2 Hit Rate): AVG=14.8%, MAX=100%

Questions:

  1. Is the GA10B GPU on the Jetson Orin Nano expected to support the mcc__dram_throughput_* counters through Nsight Systems’ periodic GPU metrics sampling (--gpu-metrics-set=ga10b-gfxt)?
  2. If the counters should work, is there any settings or configurations that needs to be enabled on JetPack 6.2.2 to expose these counters to nsys? I have already tried running with sudo and setting perf_event_paranoid to -1.
  3. Is there a newer version of Nsight Systems (or a patched ga10b-gfxt.config) that correctly collects these counters on the Orin Nano, or should I rely on ncutool for per-kernel DRAM bandwidth and tegrastats EMC_FREQ% for system-wide DRAM utilization?
  4. Are the dram__read_throughput / dram__write_throughput counters (present in ga10x.config but absent from ga10b.config) available on GA10B at all, through either nsys or ncu?

Any guidance on the supported counter set for GA10B on the Orin Nano would be appreciated. Thank you

4

Hi,

Please try to add --cuda-memory-usage=true and --gpu-metrics-devices=all and try it again:

For more details, please find the document below:

Thanks

5

It is the same with --cuda-memory-usage=true and --gpu-metrics-devices=all , doesn’t seem to help.

6

Thanks for the update. The flags you tried are the standard configuration steps, so let’s narrow down whether this is a hardware limitation, a tool version issue, or a configuration gap.

To help diagnose this, could you please provide:

  1. Metric availability check: Run nsys stats --report gputrace run_dram.nsys-rep on your captured profile and share the output. This will confirm whether the DRAM metrics are actually available in the profile or if they were never collected.

  2. Minimal reproduction: Test with a simple CUDA kernel (e.g., a basic memory copy like cudaMemcpy in a loop) instead of your full model. If DRAM metrics show non-zero values with the minimal kernel, the issue is workload-specific; if they remain zero, it’s a device/tool configuration issue.

  3. Nsight Compute cross-check: Try profiling the same workload with ncu (Nsight Compute) and check if it can access the dram__read_throughput and dram__write_throughput counters on your Orin Nano. This will tell us if the counters are available at the hardware level.

These three data points will help us determine whether GA10B on your Orin Nano supports DRAM bandwidth counters through periodic sampling, or if you should rely on alternative methods (e.g., ncu for per-kernel DRAM stats or tegrastats for system-wide EMC utilization).

8

Hi ,
Thanks for the suggestion

Metric availability check:
Had to use nsys stats --report cuda_gpu_trace g_1784253199.nsys-rep instead . gputrace did not work. Attached the output g_1784253199.cuda_gpu_trace.zip

g_1784253199.cuda_gpu_trace.zip (11.7 MB)

Minimal reproduction: Issue seem to persist. Attachment has the sample memcpy_test.cu file and CLI.txt which tells how it was executed and the outputs pf each step .
memcpy_run.cuda_gpu_trace. I find zeroes for the the metrics mentioned (CLI.txt)

Step2-Minimal-Reproduction.zip (764.6 KB)

Nsight Compute cross-check: In the same Step2-Minimal-Reproduction.zip attachment
Please refer to ncu_experiment.txt and dram_kernel_test.cu

10

Hi @AastaLLL , @carolyuu ,

I have attched the necessary data asked for . PLease let me know if you need anything else to take this debuging forward . Do let me know if you want me to try out anythign else in the meantime.

11

Hi,

Sorry for the late update.
Could you try the command mentioned in the topic below:

Thanks.