Error: Target application terminated before first instrumented API call

Developer Tools CUDA Developer Tools Compute Sanitizer
1

Hi,
I am new to Cuda profiling. I got an illegal memory access from my code and wanted to use the compute-sanitizer to find the source of the problem. I tried the following command (as admin):

compute-sanitizer --tool memcheck RayTracing.exe

But I get:

========= COMPUTE-SANITIZER
========= Error: Target application terminated before first instrumented API call
========= Tracking kernels launched by child processes requires the --target-processes all option.

I couldn’t find a solution among the topics in this forum that refered to that error.
I am on Windows 11, using the latest cuda 12.3 and Visual Studio.
This might very well be my fault, but the error message does not give me enough information to solve this.
I am happy to add more information if needed.

Thanks in advance.
Kind regards.

2

Thanks for your post, is program RayTracing.exe running subprocesses? If that is the case, you need to use Sanitizer flag --target-processes all.

3

Thanks for the response.
Does not appear to be the case. Running with --target-processes all leads to the same message, but without the last line.

4

Thanks, can you please dump the output of command nvidia-smi and compute-sanitizer --version?

Also, can you double-check RayTracing.exe contains calls to the CUDA runtime API? Add a cudaFree(0); as the first line of your main function and verify whether or not the issue is addressed.

5

Here are the outputs:

nvidia-smi

Wed Feb 28 20:50:00 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 546.12                 Driver Version: 546.12       CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                     TCC/WDDM  | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1050 Ti   WDDM  | 00000000:09:00.0  On |                  N/A |
| 20%   59C    P0              N/A /  75W |   1321MiB /  4096MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      1208    C+G   ...les\Microsoft OneDrive\OneDrive.exe    N/A      |
|    0   N/A  N/A      2028    C+G   ...siveControlPanel\SystemSettings.exe    N/A      |
|    0   N/A  N/A      3956    C+G   C:\Windows\System32\dwm.exe               N/A      |
|    0   N/A  N/A      4756    C+G   ...5n1h2txyewy\ShellExperienceHost.exe    N/A      |
|    0   N/A  N/A      7800    C+G   C:\Windows\explorer.exe                   N/A      |
|    0   N/A  N/A      8160    C+G   ...\PowerToys\PowerToys.FancyZones.exe    N/A      |
|    0   N/A  N/A      9240    C+G   ...2txyewy\StartMenuExperienceHost.exe    N/A      |
|    0   N/A  N/A      9620    C+G   ...nt.CBS_cw5n1h2txyewy\SearchHost.exe    N/A      |
|    0   N/A  N/A      9984    C+G   ...GeForce Experience\NVIDIA Share.exe    N/A      |
|    0   N/A  N/A     12004    C+G   ...les\microsoft shared\ink\TabTip.exe    N/A      |
|    0   N/A  N/A     12880    C+G   ...werToys\PowerToys.ColorPickerUI.exe    N/A      |
|    0   N/A  N/A     12936    C+G   ...werToys\PowerToys.PowerLauncher.exe    N/A      |
|    0   N/A  N/A     14464    C+G   ...CBS_cw5n1h2txyewy\TextInputHost.exe    N/A      |
|    0   N/A  N/A     15252    C+G   ...ogram Files\Bitwarden\Bitwarden.exe    N/A      |
|    0   N/A  N/A     21756    C+G   ...m Files\Mozilla Firefox\firefox.exe    N/A      |
|    0   N/A  N/A     23564    C+G   ...__8wekyb3d8bbwe\WindowsTerminal.exe    N/A      |
+---------------------------------------------------------------------------------------+

compute-sanitizer --version

NVIDIA (R) Compute Sanitizer
Copyright (c) 2020-2023 NVIDIA Corporation
Version 2023.3.1.0 (build 33473865) (public-release)

Adding cudaFree(0); does not change the output. What I did not mention previous is that before the error message appears in the console a warning window appears:

Microsoft Visual C++ Runtime Library
Debug Error!
Program: …
abort() has been called

6

Thank you! Can you provide a minimal reproducer here so I can take a look at this locally?

7

I will try. But it is a bigger project and it might time reduce the size in a way that the program still works. So might take some time.

8

No worries, thank you for taking the time to look into this. In the meantime, I recommend you try using the debugger to debug your issue: Control GPU Execution :: NVIDIA Nsight VSE Documentation

9

Hi @aladram,
I figured it out.
As the error message says, memcheck did work until the application itself threw an error. Because of this error memcheck never reached a cuda function.

Turns out the problem was reading files using relative file paths. Visual Studio uses a different working directory compared to plain executing the app or running memcheck. That way I didn’t get any errors in Visual Studio, but the App crashed when using memcheck.

Thank you so much for your help!
Best regards.

10

Awesome thank you for letting me know, glad you resolved your issue!

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.