Nsight Systems on Ubuntu, trying to profile Xavier NX

Developer Tools Nsight Systems Profiling Embedded Targets
1

Hi,

I am trying to connect Nsight Systems on Ubuntu, installed from the .deb package, to a Xavier NX, where I have installed the arm cli and moved it to the correct directory

On xavier:

USER@DEVICE: ls
/opt/nvidia/nsight-systems/2021.2.1/target-linux-armv8: ls
GpuMetrics.config       libcupti-tegra.so.10.2  libToolsInjectionCuBLAS64_10.so  nvtx
launcher                libcupti-tegra.so.11.0  libToolsInjectionCuBLAS64_11.so  python
libcupti-sbsa.so.10.2   libcupti-tegra.so.11.1  libToolsInjectionCuda64.so       reports
libcupti-sbsa.so.11.0   libcupti-tegra.so.11.2  libToolsInjectionCuDNN64_7.so    rules
libcupti-sbsa.so.11.1   libcupti-tegra.so.11.3  libToolsInjectionCuDNN64_8.so    sqlite3
libcupti-sbsa.so.11.2   libLauncher64.so        libToolsInjectionNvMedia64.so    vulkan-layers
libcupti-sbsa.so.11.3   libnvlog.so             libToolsInjectionProxy64.so
libcupti-tegra.so.10.0  libnvperf_host.so       nsys
libcupti-tegra.so.10.1  libToolsInjection64.so  nsys-launcher

However Nsight is giving the following error when I try to connect via SSH w/ password:

DirectoryNotFoundError (150) {
    OriginalExceptionClass: N5boost16exception_detail10clone_implIN11QuadDCommon26DirectoryNotFoundExceptionEEE
    OriginalFile: /build/agent/work/20a3cfcd1c25021d/QuadD/Host/Analysis/PosixDeviceValidator.cpp
    OriginalLine: 24
    OriginalFunction: bool QuadDAnalysis::PosixDeviceValidator::CheckHostSupport(const DevicePtr&)
    Filename: /opt/nvidia/nsight-systems/2021.2.1/target-linux-armv8
    ErrorText: Deploy directory does not exist
}

Clearly the directory exists, and the issue persists even if I widen permissions to something crazy like +777.

Does anyone have a solution or workaround for this issue? Please do not comment on the arm binary vs the sdkmanager, as that is unrelated to the error that I am being given.

Thanks,

2

Hi, the build you have installed 2021.2.1, is the ARM server version of Nsight Systems, rather than the version that works on embedded systems like Xavier NX.

Nsight Systems for the Xavier NX, is shipped in JetPack, and the installation instructions are at How to Install JetPack :: NVIDIA JetPack Documentation.

I found an issue with this documentation though. If you just click through to Nsight Systems page from there, you get the download page for the workstation version. And the “other platforms” link on the workstation page is broken. So you see the ARM server version and think it is correct.

Unfortunately, right now, there isn’t a good way to get Nsight Systems for tegra systems without getting JetPack (or CodeWorks (for Android)). What I will do is file a feature request to add Nsight Systems to the list of components that are available as separate apt-get components in that package. We already have apt-get options for the workstation product, so I would think it would be relatively simple.

1 Like
3

Hi hwilper,

Thanks for your help here.

That would certainly be helpful in the long run. Are you able to request that Nsight Compute binaries also be made available for the Xavier NX? (Again, outside of JetPack)

That said, I think I am also having another issue, as I would expect the ssh session of Nsight Systems on my host to be able to find the directory, irrespective of whether the binary itself on the target supports the platform.

Two followup questions:

  1. Does Nsight Systems on the host expect its ssh user to be root? Or is it sufficient to be a sudoer?
  2. Nsight Systems allows one to attach to a running process by specifying its pid. I’ve installed the Nsight Systems Cli package, but cannot find the capability to ask the nsys binary to attach to a pid in the documentation. Is there a way to attach nsys to a running process?
4

Is there a way to attach nsys to a running process?

No, this is not supported with the CLI. What you can do is use nsys launch command together with nsys start and nsys stop. This way you can start your application pre-injected using nsys launch ... ./my-app, and then use nsys start and nsys stop to start and stop the collection.

Does Nsight Systems on the host expect its ssh user to be root? Or is it sufficient to be a sudoer?

For remote connections, the user on Jetson should be a sudoer.

5

Has there been an update on this? Do we have debian packages of nsight to install on jetson devices?
I was looking at this documentation for updating on jetson device and host:
NVIDIA Jetson Linux Developer Guide : Over-the-Air Update | NVIDIA Docs
Here, the packages for jetson device does not include nsight-systems, but it is there for the host system. I have Jetson AGX Xavier with Jetpack 4.4.1. How do I install nsight-systems on it to access from linux host? Thanks in advance for any guidance.

6

@Andrey_Trachenko

7

Is there anyone else who can update about this? I went through the route of installing SDK manager on the host and installing packages on Xavier through it, however it runs into very specific issues of nvidia package version and ending up not completing the installation.
If there is a more direct way of getting nsight packages installed on the target through jetpack, I would really love to learn about it. Thanks!

8

OP Here,

Would love a response from the Jetpack team - adding in that my team would very much still like this capability as well.

9

I hope that we can get a package for the profiler! This will help make it so much easier to quickly dig into a problem when something comes up. Thanks.

10

Hi unphased / all,
For nsight on Jetson, with Jetpack installation, you should select on host which include Nsight, when you first time to connect/debug, it will automatically install nsys on Jetson

11

I am currently getting the same error with Jetpack 4.6.1-b110

12

Automatic target configuration doesn’t work if my jetpack version is <5.0.0 and my system is ubuntu 20.04. I managed to run nsight systems in a docker container based on 18.04 and this way the target device got properly configured.

13

Xavier NX jetpack version: 32.7.1
Xavier NX nsys version: 2021.5.4.19-e642d4b
Host nsys version: 2021.5.1
Host ubuntu version: 18.04

Still receiving a similar error to @PEJOE-G:

DirectoryNotFoundError (150) {
    OriginalExceptionClass: N5boost16exception_detail10clone_implIN11QuadDCommon26DirectoryNotFoundExceptionEEE
    OriginalFile: /build/agent/work/20a3cfcd1c25021d/QuadD/Host/Analysis/PosixDeviceValidator.cpp
    OriginalLine: 24
    OriginalFunction: bool QuadDAnalysis::PosixDeviceValidator::CheckHostSupport(const DevicePtr&)
    Filename: /opt/nvidia/nsight-systems/2021.5.1/target-linux-tegra-armv8
    ErrorText: Deploy directory does not exist
}

Any insights on how to get this working?

14

@pkovalenko

15

Please upgrade to the latest NSys release and see if that fixes the problem.

16

Upgraded host nsys version to: 2022.4.1
Received a similar error:

DirectoryNotFoundError (150) {
    OriginalExceptionClass: N5boost10wrapexceptIN11QuadDCommon26DirectoryNotFoundExceptionEEE
    OriginalFile: /build/agent/work/323cb361ab84164c/QuadD/Host/Analysis/PosixDeviceValidator.cpp
    OriginalLine: 24
    OriginalFunction: bool QuadDAnalysis::PosixDeviceValidator::CheckHostSupport(const DevicePtr&)
    Filename: /home/hommus/nsight-systems-2022.4.1/target-linux-tegra-armv8
    ErrorText: Deploy directory does not exist
}

How do I get the target-linux-tegra-armv8 folder on the host computer?

17

As this thread does not seem to be solved or closed:
I’m Having the same problem as @Hommus and I’m running nsight-systems-2023.2.1 @pkovalenko .

DirectoryNotFoundError (150) {
    OriginalExceptionClass: N5boost10wrapexceptIN11QuadDCommon26DirectoryNotFoundExceptionEEE
    OriginalFile: /dvs/p4/build/sw/devtools/Agora/Rel/QuadD_Main/QuadD/Host/Analysis/PosixDeviceValidator.cpp
    OriginalLine: 24
    OriginalFunction: bool QuadDAnalysis::PosixDeviceValidator::CheckHostSupport(const DevicePtr&)
    Filename: /home/epi1/nsight-systems-2023.2.1/target-linux-tegra-armv8
    ErrorText: Deploy directory does not exist
}

I’m trying to profile a target running Jetpack 4.6 r32.6.1
My host OS is ubuntu 22.04

18

Hey all,

having the same Problem. Host ist MAC with NsightSystems-macos-public-2023.1.1.127-3236574

DirectoryNotFoundError (150) {
OriginalExceptionClass: N5boost10wrapexceptIN11QuadDCommon26DirectoryNotFoundExceptionEEE
OriginalFile: /dvs/p4/build/sw/devtools/Agora/Rel/QuadD_Main/QuadD/Host/Analysis/PosixDeviceValidator.cpp
OriginalLine: 26
OriginalFunction: bool QuadDAnalysis::PosixDeviceValidator::CheckHostSupport(const QuadDAnalysis::DevicePtr &)
Filename: /Applications/NVIDIA Nsight Systems.app/Contents/target-linux-tegra-armv8
ErrorText: Deploy directory does not exist
}

19

I think part of the problem may be build disambiguation.

Nsight system versions are a four digit year, followed by a one digit release number, followed by a platform release number(and then an actual release number, which is usually irrelevant for your purposes.

So, 2023.2 (for example) is the second release of Nsys in 2023.

year.release.1 - the web version for workstation, windows, and HPC release, this is downloaded from developer.nvidia.com. This version support ARM servers, but not tegra (Jetson) targets. This is the most up to date version of the tool available, as it ships more often than CUDA toolkit.

year.release.2 - the version of the workstation, windows, and HPC release that is shipping in the CUDA toolkit. This version is usually just behind the year.release.1. This version supports ARM servers but does not support Tesla (Jetson) targets.

year.release.3+ - the version of the release for embedded or drive targets. These are shipped in the drive or Jetpack SDKs, they are not available for separate download on developer.nvidia.com. These are the only versions that support QNX or Tegra (Jetson) chips. These releases do not support x86 or ARM server targets.

While it looks like the first user was genuinely having an issue with a Jetson build on a Jetson board, it looks to me like @epi1 and @torsten.labs are trying to use the workstation build on a Jetson system.