Nsight System not generating CUDA traces for AGX Xavier

Developer Tools Nsight Systems Profiling Embedded Targets
1

I’m using the JetPack 4 beta on Ubuntu 16.04, and profiling an application on TX1 works fine. However, when I try to do the profile it on the AGX Xavier, only CPU traces show up in the resulting report, with the following diagnosis information available:

CUDA injection initialization failed.
CUDA profiling might have not been started correctly.
Zero CUDA events were collected. Does the application use CUDA?

The profiling projects for the two systems are completely the same, with CUDA traces checked. What’s wrong?

1 Like
2

Hello KireinaHoro,

I’m sorry that you experience this issue. Could you please help us with more info so that we could understand and reproduce the problem. On both Jetson devkits (TX1 and AGX Xavier), please run the following commands and let us know the output:

  • cat /etc/nv_tegra_release | head -1
  • cat /usr/local/cuda/version.txt
  • /opt/nvidia/nsight_systems/quadd_d --build_id

Feel free to post the info here on the forum, or send a direct message to me.

Looking forward to hearing back from you,

3

Output from TX1:

jsteward@tegra-ubuntu:~$ cat /etc/nv_tegra_release | head -1
# R28 (release), REVISION: 2.0, GCID: 10567845, BOARD: t210ref, EABI: aarch64, DATE: Fri Mar  2 04:58:16 UTC 2018
jsteward@tegra-ubuntu:~$ cat /usr/local/cuda/version.txt
CUDA Version 9.0.252
jsteward@tegra-ubuntu:~$ /opt/nvidia/nsight_systems/quadd_d --build_id
dad90c37ad13ad4d70740adde6deee51b37d4a4c
jsteward@tegra-ubuntu:~$

Output from Xavier:

jsteward@jetson-0423718016929:~$ cat /etc/nv_tegra_release | head -1
# R31 (release), REVISION: 0.2, GCID: 12860113, BOARD: t186ref, EABI: aarch64, DATE: Sat Sep 29 05:14:38 UTC 2018
jsteward@jetson-0423718016929:~$ cat /usr/local/cuda/version.txt
CUDA Version 10.0.117
jsteward@jetson-0423718016929:~$ /opt/nvidia/nsight_systems/quadd_d --build_id
dad90c37ad13ad4d70740adde6deee51b37d4a4c
jsteward@jetson-0423718016929:~$
1 Like
4

Hello KireinaHoro,

It looks like there is a mismatch between the version of CUDA Driver on Jetson AGX Xavier (10.0.117) and the version of CUDA Toolkit used by Nsight Systems (10.0.96), which unfortunately caused the failure.

Could you please download the latest available JetPack (JetPack SDK | NVIDIA Developer) and give it a go?

Thank you,

5

Hi,

I do face the same issue.
My target info is

ubuntu@gpucoder-hyd-tegratx2-2:~$ cat  /etc/nv_tegra_release | head -1
# R32 (release), REVISION: 5.1, GCID: 26202423, BOARD: t186ref, EABI: aarch64, DATE: Fri Feb 19 16:50:29 UTC 2021
ubuntu@gpucoder-hyd-tegratx2-2:~$ cat /usr/local/cuda/version.txt
CUDA Version 10.2.89
ubuntu@gpucoder-hyd-tegratx2-2:~$  /opt/nvidia/nsight_systems/quadd_d --build_id
-bash: /opt/nvidia/nsight_systems/quadd_d: No such file or directory
ubuntu@gpucoder-hyd-tegratx2-2:~$ nsys --version
NVIDIA Nsight Systems version 2021.4.1.73-08591f7
ubuntu@gpucoder-hyd-tegratx2-2:~$

Can anyone help on this? I am unable to grab the CUDA events on Tegra due to this issue. I am working on TX2.

Thanks,
MRK

6

@anikiforov can you answer this or refer it to someone that can?