Nsight Compute crash with message: free(): invalid pointer

tnwilly · November 7, 2024, 10:49am

Hi, my nsight compute crashed unexpectedly.
The application was built correctly and worked well but crashed during profiling with Nsight Compute.

Device: AGX Orin
Nsight Compute Version: 2024.3.1 (from SDK Manager)

Error messages:

/opt/nvidia/nsight-compute/2024.3.1$ sudo ncu-ui
free(): invalid pointer
2024-11-07 18:42:33: stackwalker_arm64.cc:219: ERROR: Unable to read caller_fp from last_fp: 0x1

Reason: SIGABRT
Address: 0xc0ee

Thread 0 (crashed)
[0] libc.so.6 + 0x7f200
[1] libc.so.6 + 0x7f1e8
[2] libc.so.6 + 0x3a678
[3] libc.so.6 + 0x2712c
[4] libc.so.6 + 0x73304
[5] libc.so.6 + 0x89578
[6] libc.so.6 + 0x8b2c0
[7] libc.so.6 + 0x8dc80
[8] libstdc++.so.6 + 0xb869c
[9] libstdc++.so.6 + 0xb875c
[10] libRebelPlugin.so + 0x13f5e58
[11] libRebelPlugin.so + 0x7172cc
[12] libQt6Core.so.6 + 0x351670
[13] libQt6Core.so.6 + 0x351670
[14] libQt6Core.so.6 + 0x351670
[15] libQt6Core.so.6 + 0x31fbe0
[16] libQt6Widgets.so.6 + 0x41fc74
[17] libRebelPlugin.so + 0x7e5650
[18] libQt6Widgets.so.6 + 0x446670
[19] libQt6Widgets.so.6 + 0x1dfef4
[20] libQt6Widgets.so.6 + 0x25c66c
[21] libQt6Widgets.so.6 + 0x3f5020
[22] libQt6Core.so.6 + 0x128784
[23] libQt6Widgets.so.6 + 0x19828c
[24] libQt6Widgets.so.6 + 0x199b20
[25] libQt6Core.so.6 + 0x128468
[26] libQt6Widgets.so.6 + 0x1d2e54
[27] libQt6Widgets.so.6 + 0x1da794
[28] libQt6Widgets.so.6 + 0x1da684
[29] libQt6Widgets.so.6 + 0x1da684
[30] libQt6Widgets.so.6 + 0x1da684
[31] libQt6Widgets.so.6 + 0x1da684
[32] libQt6Widgets.so.6 + 0x1da684
[33] libQt6Widgets.so.6 + 0x1d3088
[34] libQt6Widgets.so.6 + 0x1da794
[35] libQt6Widgets.so.6 + 0x1d3088
[36] libQt6Widgets.so.6 + 0x1da794
[37] libQt6Widgets.so.6 + 0x1da684
[38] libQt6Widgets.so.6 + 0x1d3088
[39] libQt6Widgets.so.6 + 0x1da794
[40] libQt6Widgets.so.6 + 0x1d3088
[41] libQt6Widgets.so.6 + 0x1da794
[42] libQt6Widgets.so.6 + 0x1d3088
[43] libQt6Widgets.so.6 + 0x1da794
[44] libQt6Widgets.so.6 + 0x1da684
[45] libQt6Widgets.so.6 + 0x1da684
[46] libQt6Widgets.so.6 + 0x1da684
[47] libQt6Widgets.so.6 + 0x1da684
[48] libQt6Widgets.so.6 + 0x1da684
[49] libQt6Widgets.so.6 + 0x1da684
[50] libQt6Widgets.so.6 + 0x1da684
[51] libQt6Widgets.so.6 + 0x1da684
[52] libQt6Widgets.so.6 + 0x1da684
[53] libQt6Widgets.so.6 + 0x1da684
[54] libQt6Widgets.so.6 + 0x1da684
[55] libQt6Widgets.so.6 + 0x1da684
[56] libQt6Widgets.so.6 + 0x1d3088
[57] libQt6Widgets.so.6 + 0x1da794
[58] libQt6Widgets.so.6 + 0x1d3088
[59] libQt6Widgets.so.6 + 0x1da794
[60] libQt6Widgets.so.6 + 0x1d3088
[61] libQt6Widgets.so.6 + 0x1da794
[62] libQt6Widgets.so.6 + 0x1da684
[63] libQt6Widgets.so.6 + 0x1d3088
[64] libQt6Widgets.so.6 + 0x1da794
[65] libQt6Widgets.so.6 + 0x1d3088
[66] libQt6Widgets.so.6 + 0x1da794
[67] libQt6Widgets.so.6 + 0x1d3088
[68] libQt6Widgets.so.6 + 0x1da794
[69] libQt6Widgets.so.6 + 0x1d3088
[70] libQt6Widgets.so.6 + 0x1da794
[71] libQt6Widgets.so.6 + 0x1da684
[72] libQt6Widgets.so.6 + 0x1da684
[73] libQt6Widgets.so.6 + 0x1d3088
[74] libQt6Widgets.so.6 + 0x1da794
[75] libQt6Widgets.so.6 + 0x1d3088
[76] libQt6Widgets.so.6 + 0x1da794
[77] libQt6Widgets.so.6 + 0x1da684
[78] libQt6Widgets.so.6 + 0x1da684
[79] libQt6Widgets.so.6 + 0x1da684
[80] libQt6Widgets.so.6 + 0x1da684
[81] libQt6Widgets.so.6 + 0x1da684
[82] libQt6Widgets.so.6 + 0x1da684
[83] libQt6Widgets.so.6 + 0x1da684
[84] libQt6Widgets.so.6 + 0x1da684
[85] libQt6Widgets.so.6 + 0x1da684
[86] libQt6Widgets.so.6 + 0x1d3088
[87] libQt6Widgets.so.6 + 0x1ee028
[88] libQt6Widgets.so.6 + 0x1ee298
[89] libQt6Widgets.so.6 + 0x1e060c
[90] libQt6Widgets.so.6 + 0x2eeadc
[91] libQt6Widgets.so.6 + 0x1982ac
[92] libQt6Widgets.so.6 + 0x199b20
[93] libQt6Core.so.6 + 0x128468
[94] libQt6Core.so.6 + 0x1295bc
[95] libQt6Widgets.so.6 + 0x4bb2a4
[96] libQt6Core.so.6 + 0x16b2a4
[97] libQt6Widgets.so.6 + 0x4c448c
[98] libQt6Widgets.so.6 + 0x1982ac
[99] libQt6Widgets.so.6 + 0x1990d4
[100] libQt6Core.so.6 + 0x128468
[101] libQt6Core.so.6 + 0x1295bc
[102] libQt6Core.so.6 + 0x374ad0
[103] libglib-2.0.so.0 + 0x55a6c
[104] libglib-2.0.so.0 + 0xaaeb8
[105] libglib-2.0.so.0 + 0x52f10
[106] libQt6Core.so.6 + 0x374544
[107] libQt6Core.so.6 + 0x131650
[108] libQt6Core.so.6 + 0x128b54
[109] libAppLib.so + 0xbb680
[110] libc.so.6 + 0x274c8
[111] ncu-ui.bin + 0x6318

Thread 1
[0] libc.so.6 + 0xdbe38
[1] libc.so.6 + 0xdbe18
[2] libglib-2.0.so.0 + 0xaae54
[3] libglib-2.0.so.0 + 0x52f10
[4] libQt6Core.so.6 + 0x374544
[5] libQt6Core.so.6 + 0x131650
[6] libQt6Core.so.6 + 0x21e970
[7] libQt6DBus.so.6 + 0x30a40
[8] libQt6Core.so.6 + 0x28b674
[9] libc.so.6 + 0x7d5c4

Thread 2
[0] libc.so.6 + 0xdbe38
[1] libc.so.6 + 0xdbe18
[2] libxcb.so.1 + 0xc1ec
[3] libxcb.so.1 + 0xdbb4
[4] libQt6XcbQpa.so.6 + 0x51dfc
[5] libQt6Core.so.6 + 0x28b674
[6] libc.so.6 + 0x7d5c4

Thread 3
[0] libc.so.6 + 0x79df8
[1] libc.so.6 + 0x79dd4
[2] libc.so.6 + 0x7c8f8
[3] libTPSConnectionPlugin.so + 0xbf5b8
[4] libc.so.6 + 0x7d5c4

Thread 4
[0] libc.so.6 + 0xe607c
[1] libc.so.6 + 0xe6054
[2] libTPSConnectionPlugin.so + 0xc33d0
[3] libc.so.6 + 0x7d5c4

Thread 5
[0] libc.so.6 + 0x79df8
[1] libc.so.6 + 0x79dd4
[2] libc.so.6 + 0x7c8f8
[3] libRebelPlugin.so + 0x13bceec
[4] libRebelPlugin.so + 0x593578
[5] libc.so.6 + 0x7d5c4

Thread 6
[0] libc.so.6 + 0xdbe38
[1] libc.so.6 + 0xdbe18
[2] libglib-2.0.so.0 + 0xaae54
[3] libglib-2.0.so.0 + 0x52f10
[4] libQt6Core.so.6 + 0x374544
[5] libQt6Core.so.6 + 0x131650
[6] libQt6Core.so.6 + 0x21e970
[7] libQt6Core.so.6 + 0x28b674
[8] libc.so.6 + 0x7d5c4

Thread 7
[0] libc.so.6 + 0x79df8
[1] libc.so.6 + 0x79dd4
[2] libc.so.6 + 0x7cc0c
[3] libQt6Core.so.6 + 0x295714
[4] libQt6Core.so.6 + 0x29546c
[5] libQt6Core.so.6 + 0x28f0b4
[6] libQt6Core.so.6 + 0x28b674
[7] libc.so.6 + 0x7d5c4

Thread 8
[0] libc.so.6 + 0x79df8
[1] libc.so.6 + 0x79dd4
[2] libc.so.6 + 0x7cc0c
[3] libQt6Core.so.6 + 0x295714
[4] libQt6Core.so.6 + 0x29546c
[5] libQt6Core.so.6 + 0x28f0b4
[6] libQt6Core.so.6 + 0x28b674
[7] libc.so.6 + 0x7d5c4

Thread 9
[0] libc.so.6 + 0x79df8
[1] libc.so.6 + 0x79dd4
[2] libc.so.6 + 0x7cc0c
[3] libQt6Core.so.6 + 0x295714
[4] libQt6Core.so.6 + 0x29546c
[5] libQt6Core.so.6 + 0x28f0b4
[6] libQt6Core.so.6 + 0x28b674
[7] libc.so.6 + 0x7d5c4

Thread 10
[0] libc.so.6 + 0x79df8
[1] libc.so.6 + 0x79dd4
[2] libc.so.6 + 0x7cc0c
[3] libQt6Core.so.6 + 0x295714
[4] libQt6Core.so.6 + 0x29546c
[5] libQt6Core.so.6 + 0x28f0b4
[6] libQt6Core.so.6 + 0x28b674
[7] libc.so.6 + 0x7d5c4

Thread 11
[0] libc.so.6 + 0x79df8
[1] libc.so.6 + 0x79dd4
[2] libc.so.6 + 0x7cc0c
[3] libQt6Core.so.6 + 0x295714
[4] libQt6Core.so.6 + 0x29546c
[5] libQt6Core.so.6 + 0x28f0b4
[6] libQt6Core.so.6 + 0x28b674
[7] libc.so.6 + 0x7d5c4

Thread 12
[0] libc.so.6 + 0x79df8
[1] libc.so.6 + 0x79dd4
[2] libc.so.6 + 0x7cc0c
[3] libQt6Core.so.6 + 0x295714
[4] libQt6Core.so.6 + 0x29546c
[5] libQt6Core.so.6 + 0x28f0b4
[6] libQt6Core.so.6 + 0x28b674
[7] libc.so.6 + 0x7d5c4

Thread 13
[0] libc.so.6 + 0x79df8
[1] libc.so.6 + 0x79dd4
[2] libc.so.6 + 0x7cc0c
[3] libQt6Core.so.6 + 0x295714
[4] libQt6Core.so.6 + 0x29546c
[5] libQt6Core.so.6 + 0x28f0b4
[6] libQt6Core.so.6 + 0x28b674
[7] libc.so.6 + 0x7d5c4

Thread 14
[0] libc.so.6 + 0x79df8
[1] libc.so.6 + 0x79dd4
[2] libc.so.6 + 0x7cc0c
[3] libQt6Core.so.6 + 0x295714
[4] libQt6Core.so.6 + 0x29546c
[5] libQt6Core.so.6 + 0x28f0b4
[6] libQt6Core.so.6 + 0x28b674
[7] libc.so.6 + 0x7d5c4

Thread 15
[0] libc.so.6 + 0x79df8
[1] libc.so.6 + 0x79dd4
[2] libc.so.6 + 0x7cc0c
[3] libQt6Core.so.6 + 0x295714
[4] libQt6Core.so.6 + 0x29546c
[5] libQt6Core.so.6 + 0x28f0b4
[6] libQt6Core.so.6 + 0x28b674
[7] libc.so.6 + 0x7d5c4

Thread 16
[0] libc.so.6 + 0x79df8
[1] libc.so.6 + 0x79dd4
[2] libc.so.6 + 0x7cc0c
[3] libQt6Core.so.6 + 0x295714
[4] libQt6Core.so.6 + 0x29546c
[5] libQt6Core.so.6 + 0x28f0b4
[6] libQt6Core.so.6 + 0x28b674
[7] libc.so.6 + 0x7d5c4

Thread 17
[0] libc.so.6 + 0x79df8
[1] libc.so.6 + 0x79dd4
[2] libc.so.6 + 0x7cc0c
[3] libQt6Core.so.6 + 0x295714
[4] libQt6Core.so.6 + 0x29546c
[5] libQt6Core.so.6 + 0x28f0b4
[6] libQt6Core.so.6 + 0x28b674
[7] libc.so.6 + 0x7d5c4

Thread 18
[0] libc.so.6 + 0x79df8
[1] libc.so.6 + 0x79dd4
[2] libc.so.6 + 0x7cc0c
[3] libQt6Core.so.6 + 0x295714
[4] libQt6Core.so.6 + 0x29546c
[5] libQt6Core.so.6 + 0x28f0b4
[6] libQt6Core.so.6 + 0x28b674
[7] libc.so.6 + 0x7d5c4

Thread 19
[0] libc.so.6 + 0x79df8
[1] libc.so.6 + 0x79dd4
[2] libc.so.6 + 0x7cc0c
[3] libQt6Core.so.6 + 0x295714
[4] libQt6Core.so.6 + 0x29546c
[5] libQt6Core.so.6 + 0x28f0b4
[6] libQt6Core.so.6 + 0x28b674
[7] libc.so.6 + 0x7d5c4

Thread 20
[0] libc.so.6 + 0xdbe38
[1] libc.so.6 + 0xdbe18
[2] libglib-2.0.so.0 + 0xaae54
[3] libglib-2.0.so.0 + 0x52f10
[4] libQt6Core.so.6 + 0x374544
[5] libQt6Core.so.6 + 0x131650
[6] libQt6Core.so.6 + 0x21e970
[7] libQt6Core.so.6 + 0x28b674
[8] libc.so.6 + 0x7d5c4

veraj · November 8, 2024, 7:31am

Hi, @tnwilly

Sorry for the issue you met.

From what you paste, ncu-ui failed to launch directly.
What about ncu ? Can you profile directly with ncu ?

tnwilly · November 8, 2024, 7:59am

No, still crash.

I checked the path /home/dd/Documents/NVIDIA\ Nsight\ Compute/2024.3.1/Sections/
But what to do next?

$ sudo ncu ./xxx
==WARNING== Could not deploy stock section files to “/root/Documents/NVIDIA Nsight Compute/2021.3.1/Sections”.
Set the HOME environment variable to a writable directory.
==WARNING== Using “/usr/lib/aarch64-linux-gnu/nsight-compute/target/linux-desktop-t210-a64/…/…/sections” instead.
==WARNING== See 2. Kernel Profiling Guide — NsightCompute 12.6 documentation.
==WARNING== Failed to add default section search path /usr/lib/aarch64-linux-gnu/nsight-compute/target/linux-desktop-t210-a64/…/…/sections.
==ERROR== Failed to add rules search path /usr/lib/aarch64-linux-gnu/nsight-compute/target/linux-desktop-t210-a64/…/…/sections.

veraj · November 8, 2024, 9:49am

I checked with Orin test team. NCU-UI can launch successfully.
So maybe you need re-install your system with SDK Manager and try it again.

Regard to the failure when using ncu, this seems a permission issue.

tnwilly · November 9, 2024, 3:29pm

Actually, I re-installed nsight compute 2024.3.1 with SDK manager but still encountered the issue. (nsight system is ok.)
How to install specific version of nsight compute manually on AGX Orin? Ex: 2024.3.2

There’s no ARM version for AGX Orin here: Developer Tools Download Center | NVIDIA Developer

veraj · November 11, 2024, 9:35am

Hi, @tnwilly

Do you have a host machine, either Windows or Linux ?
Can you try remote profile from host machine ? So that we can know if the package installed from SDK has issue.

tnwilly · November 11, 2024, 10:02am

Hi Veraj,

Maybe not, I used ARM PL in the code base, so I can’t profile it on my host machine which is x86 arch.

I think reinstall with other version would be a solution.

veraj · November 11, 2024, 10:07am

I mean specify your Orin ip/usr/passwd here. And then do remote profile. The profile still happens on Orin side. (Note the usr need to be ‘root’ to get profiler permission)

tnwilly · November 11, 2024, 10:16am

How to exec it with root privileges? Any options I can select in the Nsight Compute GUI?

veraj · November 11, 2024, 10:19am

Which user are you using ? Not root ?
If not, you can execute “sudo usermode -aG ${your-user-name} debug” on Orin firstly, then collect with this user again.

tnwilly · November 11, 2024, 10:21am

I am root user of both host and device machine.

veraj · November 11, 2024, 10:26am

Then there must be permission issue in your env. From the result you pasted before.
Please try this command again, and figure out which folder you don’t have permission and resolve it.

$ sudo ncu ./xxx
==WARNING== Could not deploy stock section files to “/root/Documents/NVIDIA Nsight Compute/2021.3.1/Sections”.
Set the HOME environment variable to a writable directory.
==WARNING== Using “/usr/lib/aarch64-linux-gnu/nsight-compute/target/linux-desktop-t210-a64/…/…/sections” instead.
==WARNING== See 2. Kernel Profiling Guide — NsightCompute 12.6 documentation.
==WARNING== Failed to add default section search path /usr/lib/aarch64-linux-gnu/nsight-compute/target/linux-desktop-t210-a64/…/…/sections.
==ERROR== Failed to add rules search path /usr/lib/aarch64-linux-gnu/nsight-compute/target/linux-desktop-t210-a64/…/…/sections.

tnwilly · November 11, 2024, 10:30am

Suppose the path below needs to be redirected to somewhere instead of /root/xxx/...

==WARNING== Could not deploy stock section files to “/root/Documents/NVIDIA Nsight Compute/2021.3.1/Sections”.`

but how to do this?

I guess the path should be this one:

/opt/nvidia/nsight-compute/2024.3.1/sections

veraj · November 11, 2024, 10:50am

Please run “sudo /opt/nvidia/nsight-compute/2024.3.1/ncu” instead of sudo ncu

tnwilly · November 11, 2024, 10:59am

This works.

sudo /opt/nvidia/nsight-compute/2024.3.1/ncu

But use gui will crash.

sudo /opt/nvidia/nsight-compute/2024.3.1/ncu-ui

圖片887×137 31.5 KB

tnwilly · November 12, 2024, 3:26pm

Hi @veraj,
Is there any follow-up?

veraj · November 13, 2024, 2:55am

Hi, @tnwilly

As you can profile with “ncu” now, for workaround, you can use it to generate report, and then open it in another machine.

For “ncu-ui” can not launch issue, please provide the “ncu-ui”(should be a script) you used and generate a log for “sudo /opt/nvidia/nsight-compute/2024.3.1/ncu-ui”

The common usage for Orin is to remote profile from a host machine, but I don’t understand why you still got permission issue when connect with root user. This is strange.

tnwilly · November 13, 2024, 5:25am

I executed the nsight compute on the device and dumped the log directly.
Please take a look, thanks a lot!
ncu-ui-log.txt (59.6 KB)

veraj · November 13, 2024, 5:47am

What’s your operation ?
I assume you didn’t have GUI launched successfully, but you seems already get report generated.

==PROF== Attempting to connect to ncu-ui at 127.0.0.1:50152…
==PROF== Connected to ncu-ui at 127.0.0.1:50152.
==PROF== Connected to process 172595 (/home/ddjetson/Desktop/weiminc/cudadev/cpp/build/src/unittest/musicSim)
==PROF== Profiling “mdl(int *, const float *, int)” - 0: 0%…50%…100% - 8 passes
==PROF== Profiling “noise_extraction” - 1: 0%…50%…100% - 8 passes
==PROF== Profiling “opt2_scanning” - 2: 0%…50%…100% - 8 passes
==PROF== Profiling “spectrum_scanning” - 3: 0%…50%…100% - 8 passes
==PROF== Profiling “mdl(int *, const float *, int)” - 4: 0%…50%…100% - 8 passes
==PROF== Profiling “noise_extraction” - 5: 0%…50%…100% - 8 passes
==PROF== Profiling “opt2_scanning” - 6: 0%…50%…100% - 8 passes
==PROF== Profiling “spectrum_scanning” - 7: 0%…50%…100% - 8 passes
==PROF== Profiling “mdl(int *, const float *, int)” - 8: 0%…50%…100% - 8 passes
==PROF== Profiling “noise_extraction” - 9: 0%…50%…100% - 8 passes
==PROF== Profiling “opt2_scanning” - 10: 0%…50%…100% - 8 passes
==PROF== Profiling “spectrum_scanning” - 11: 0%…50%…100% - 8 passes
==PROF== Profiling “mdl(int *, const float *, int)” - 12: 0%…50%…100% - 8 passes
==PROF== Profiling “noise_extraction” - 13: 0%…50%…100% - 8 passes
==PROF== Profiling “opt2_scanning” - 14: 0%…50%…100% - 8 passes
==PROF== Profiling “spectrum_scanning” - 15: 0%…50%…100% - 8 passes
==PROF== Profiling “mdl(int *, const float *, int)” - 16: 0%…50%…100% - 8 passes
==PROF== Profiling “noise_extraction” - 17: 0%…50%…100% - 8 passes
==PROF== Profiling “opt2_scanning” - 18: 0%…50%…100% - 8 passes
==PROF== Profiling “spectrum_scanning” - 19: 0%…50%…100% - 8 passes
=========svd_t: 584.4
=========spectrum_t: 681504
=========noise_extract_t: 693998
=========padding_t: 0
=========maxFilter_t: 0
=========decideThreshold_t: 0
=========compareSpectrum_t: 0
=========mdl_t: 1.10621e+06
=========total_t: 3.17584e+06
==PROF== Disconnected from process 172595
==PROF== Report: /home/ddjetson/Desktop/weiminc/cudadev/cpp/build/src/unittest/test_log.ncu-rep

tnwilly · November 13, 2024, 5:52am

I connected to the device with ssh from the host, then executed the nsight compute GUI and dumped the log.
If I do the operation on the device directly, you will not see these GUI warnings in the log.

Topic		Replies	Views
Nsight Compute not detecting kernel launch Nsight Compute profiling	13	3036	May 6, 2021
Nsight compute - SSH login with root Jetson AGX Orin nsight	34	3621	September 21, 2022
Ncu does not detect kernels, ==ERROR== The application returned an error code (11) Nsight Compute kernel , profiling	6	1787	December 13, 2023
Nv-nsight-cu-cli segfault Nsight Compute cuda	6	1744	May 5, 2020
Can't Get NCU GUI To Import Properly Nsight Compute	8	1311	October 5, 2020
Option to profile only master process Nsight Compute cuda	23	3248	December 1, 2023
NSight Compute not finding kernels Nsight Compute	24	591	October 24, 2024
Many issues with nsight nvidia Jetson TX2 nsight	10	1193	October 18, 2021
Nsight systems doesn't seemed to work correctly Profiling Linux Targets cuda	19	2528	July 10, 2023
Fail to open Nsight Compute 2024.1.0(Ubuntu) Nsight Compute	8	1483	March 12, 2024

Nsight Compute crash with message: free(): invalid pointer

Related topics