CPU sampling in privileged Docker container via `sudo nsys`

[max.gerlach]

Using nsys I would like to to profile an application executed inside a Docker container running with --privileged (following the user guide, chapter 6, User Guide :: Nsight Systems Documentation). Even though the kernel paranoia level is set to 4, I would expect to be able to collect a profile that includes CPU sampling data by running under sudo. This, however, does not work:

$ nsys --version
NVIDIA Nsight Systems version 2022.2.1.31-5fe97ab

$ cat /proc/sys/kernel/perf_event_paranoid
4

$ sudo nsys status --environment
Timestamp counter supported: Yes

Sampling Environment Check
Root privilege: enabled
Linux Kernel Paranoid Level = 4: Fail
Linux Distribution = Ubuntu
Linux Kernel Version = 5.8.0-44-generic: OK
Linux perf_event_open syscall available: OK
Sampling trigger event available: OK
Intel(c) Last Branch Record support: Not Available
Sampling Environment: OK

Note: When Root privilege is enabled, the Linux Kernel Paranoid Level is overridden.

$ sudo nsys profile python -c 'print("foo")'
WARNING: CPU context switch tracing not supported, disabling.
Try the ‘nsys status --environment’ command to learn more.

WARNING: CPU sampling not supported, disabling.
Try the ‘nsys status --environment’ command to learn more.

foo
Generating ‘/tmp/nsys-report-37c1.qdstrm’
[1/1] [========================100%] report1.nsys-rep
Generated:
…/report1.nsys-rep

I can lower the paranoia level and apparently profiling works with CPU sampling, even without sudo, but I’d prefer not to as that affects processes running outside of my container, too:

$ sudo sysctl -w kernel.perf_event_paranoid=2
kernel.perf_event_paranoid = 2

$ cat /proc/sys/kernel/perf_event_paranoid
2

$ nsys status --environment
Timestamp counter supported: Yes

Sampling Environment Check
Root privilege: disabled
Linux Kernel Paranoid Level = 2: OK
Linux Distribution = Ubuntu
Linux Kernel Version = 5.8.0-44-generic: OK
Linux perf_event_open syscall available: OK
Sampling trigger event available: OK
Intel(c) Last Branch Record support: Not Available
Sampling Environment: OK

$ nsys profile python -c 'print("foo")'
Warning: LBR backtrace method is not supported on this platform. DWARF backtrace method will be used.
foo
Generating ‘/tmp/nsys-report-1106.qdstrm’
[1/1] [========================100%] report2.nsys-rep
Generated:
…/report2.nsys-rep

[hwilper]

@rknight can you respond to this.

[rknight]

In this situation (‘nsys status’ command run inside Docker container), the ‘nsys status --environment’ command result is incorrect - i.e. sudo permission does not override the paranoid level which is set by the host OS. This is a bug in nsys.

You will need to set the host OS’ paranoid level to 2 or less to enable nsys to collect CPU IP/backtrace samples or trace CPU context switches.

[max.gerlach]

Thanks for the explanation and clarifying that the output of nsys status --environment is misleading here!