Nsight System not generating CUDA traces for AGX Xavier

I’m using the JetPack 4 beta on Ubuntu 16.04, and profiling an application on TX1 works fine. However, when I try to do the profile it on the AGX Xavier, only CPU traces show up in the resulting report, with the following diagnosis information available:

CUDA injection initialization failed.
CUDA profiling might have not been started correctly.
Zero CUDA events were collected. Does the application use CUDA?

The profiling projects for the two systems are completely the same, with CUDA traces checked. What’s wrong?

Hello KireinaHoro,

I’m sorry that you experience this issue. Could you please help us with more info so that we could understand and reproduce the problem. On both Jetson devkits (TX1 and AGX Xavier), please run the following commands and let us know the output:

  • cat /etc/nv_tegra_release | head -1
  • cat /usr/local/cuda/version.txt
  • /opt/nvidia/nsight_systems/quadd_d --build_id

Feel free to post the info here on the forum, or send a direct message to me.

Looking forward to hearing back from you,

Output from TX1:

jsteward@tegra-ubuntu:~$ cat /etc/nv_tegra_release | head -1
# R28 (release), REVISION: 2.0, GCID: 10567845, BOARD: t210ref, EABI: aarch64, DATE: Fri Mar  2 04:58:16 UTC 2018
jsteward@tegra-ubuntu:~$ cat /usr/local/cuda/version.txt
CUDA Version 9.0.252
jsteward@tegra-ubuntu:~$ /opt/nvidia/nsight_systems/quadd_d --build_id
dad90c37ad13ad4d70740adde6deee51b37d4a4c
jsteward@tegra-ubuntu:~$

Output from Xavier:

jsteward@jetson-0423718016929:~$ cat /etc/nv_tegra_release | head -1
# R31 (release), REVISION: 0.2, GCID: 12860113, BOARD: t186ref, EABI: aarch64, DATE: Sat Sep 29 05:14:38 UTC 2018
jsteward@jetson-0423718016929:~$ cat /usr/local/cuda/version.txt
CUDA Version 10.0.117
jsteward@jetson-0423718016929:~$ /opt/nvidia/nsight_systems/quadd_d --build_id
dad90c37ad13ad4d70740adde6deee51b37d4a4c
jsteward@jetson-0423718016929:~$

Hello KireinaHoro,

It looks like there is a mismatch between the version of CUDA Driver on Jetson AGX Xavier (10.0.117) and the version of CUDA Toolkit used by Nsight Systems (10.0.96), which unfortunately caused the failure.

Could you please download the latest available JetPack (https://developer.nvidia.com/embedded/jetpack) and give it a go?

Thank you,