Unable to profile GPU metrics in nsight system

Developer Tools Nsight Systems Profiling Linux Targets
1

Hello, I want to profile vllm in a docker container. I followed the steps in User Guide — nsight-systems 2025.2 documentation and NVIDIA Development Tools Solutions - ERR_NVGPUCTRPERM: Permission issue with Performance Counters | NVIDIA Developer, but still can’t see GPU metrics in Nsight System. I tried some solutions in this forum but didn’t work out. Could anyone help me?

image
image1940×538 59.2 KB

  • NVIDIA Nsight Systems version 2024.6.2.225-246235244400v0
  • nsys profile --gpu-metrics-devices=help
    Possible --gpu-metrics-devices values are:
    0: NVIDIA H20 PCI[0000:6b:02.0]
    1: NVIDIA H20 PCI[0000:6b:03.0]
    2: NVIDIA H20 PCI[0000:69:02.0]
    3: NVIDIA H20 PCI[0000:69:03.0]
    4: NVIDIA H20 PCI[0000:67:02.0]
    5: NVIDIA H20 PCI[0000:67:03.0]
    6: NVIDIA H20 PCI[0000:65:02.0]
    7: NVIDIA H20 PCI[0000:65:03.0]
    all: Select all supported GPUs
    cuda-visible: Select GPUs that match CUDA_VISIBLE_DEVICES
    none: Disable GPU Metrics [Default]
    nsys status -e:
    Timestamp counter supported: Yes

CPU Profiling Environment Check
Root privilege: enabled
Linux Kernel Paranoid Level = 4
Linux Distribution = Ubuntu
Linux Kernel Version = 5.15.0-91-generic: OK
Linux perf_event_open syscall available: OK
Sampling trigger event available: OK
Intel(c) Last Branch Record support: Not Available
CPU Profiling Environment (process-tree): OK
CPU Profiling Environment (system-wide): OK

Nsight Systems UI on MacOS: 2025.2.1

2

It looks like the version of Nsys on the device is only 2024.6, can you update the one on the device?

@rknight does this status look reasonable?

3

What docker run command do you use?

5

the command I used to install nsys:
apt update
apt install -y --no-install-recommends gnupg
echo “deb http://developer.download.nvidia.com/devtools/repos/ubuntu$(source /etc/lsb-release; echo “$DISTRIB_RELEASE” | tr -d .)/$(dpkg --print-architecture) /” | tee /etc/apt/sources.list.d/nvidia-devtools.list
apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
apt update
apt install nsight-systems-cli

which nsys version do you suggest?

6

the docker run command I use:
docker run --name=xxx --user=root --pid host --network=host --privileged=true --cap-add=SYS_ADMIN --gpus=all --runtime=runc --detach=true -t …