Compute-sanitizer error : Target application terminated before first instrumented API call

Development Tools CUDA Developer Tools Compute Sanitizer
1

Hi,

I am trying to use compute-sanitizer on my app (CUDA12.1 - windows 10 ):

.\compute-sanitizer.exe --launch-timeout 0 --target-processes all --print-level info --tool memcheck app.exe

That’s what has returned to me (… only that …) :

========= COMPUTE-SANITIZER
========= Target application terminated before first instrumented API call

I have no more information … no clue how to solve this problem.

Do you know how can i have more information ?

If it can help, i used cmake conf as:

...
target_link_libraries( app
    PRIVATE
        CUDA::cudart
        CUDA::cuda_driver
        CUDA::nvtx3
)

# Target options
set_target_properties( app
        PROPERTIES
            CXX_STANDARD 20
            CXX_STANDARD_REQUIRED ON
            CUDA_STANDARD 17
            CUDA_STANDARD_REQUIRED ON
            WINDOWS_EXPORT_ALL_SYMBOLS ON
            POSITION_INDEPENDENT_CODE ON #shared lib 
            CUDA_SEPARABLE_COMPILATION ON 
)


target_compile_options( app
    PRIVATE 
    $<$<AND:$<COMPILE_LANGUAGE:CXX>,$<CXX_COMPILER_ID:MSVC>>:
        /MP>
    $<$<AND:$<CONFIG:RELEASE>,$<COMPILE_LANGUAGE:CUDA>>:
        --generate-line-info
        --use_fast_math
        --diag-suppress=489,821
     #   --dlink-time-opt (-dlto)
        >
    $<$<AND:$<CONFIG:DEBUG>,$<COMPILE_LANGUAGE:CUDA>>:
        --device-debug
        >
)

target_compile_definitions(app
    PRIVATE
        NOMINMAX # enable use of std::min in windows context
 )
...

Thank you for your time and best Regards!

2

What GPU are you running on, and is WSL or any containers/virtualization involved at all? Thanks.

3

Hi,

The Gpu is Quadro RTX 5000. I don’t use any container or virtualisation.
I add some precisions:

  • the GPGPU compute part is a shared librart (compute_lib [but in the previous post i named it App, sorry for that] ).
  • I use ray tracing (optix 7.2) in another shared librart (optix_lib).
  • The executable StudiesScenario use compute_lib and optix_lib (and some other c++ libs)

During a normal launch of StudiesScenario in release mode , only sometimes (+/-1 times for 4 launchs) , after many iterations, i get some NAN as result.
In debug mode, no errors, the program run normaly, so i cannot use the debugger.

When i launch compute-sanitizer on StudiesScenario build in release mode i get as mentionned only :

========= COMPUTE-SANITIZER
========= Target application terminated before first instrumented API call

but in debug mode i get that:

image

I try compute-sanitizer on simple cuda addition exemple and there are no problems:

.\compute-sanitizer.exe --launch-timeout 100000 --target-processes all --print-level info --tool memcheck --require-cuda-init no E:\dev\prog\CudaSimple\x64\Debug\CudaSimple.exe
========= COMPUTE-SANITIZER
{1,2,3,4,5} + {10,20,30,40,50} = {11,22,33,44,55}
========= ERROR SUMMARY: 0 errors

Hope it can help.

Thanks a lot to help me.

4

It’s tough to say what’s going on here. When you run CS in release mode and it gives the message about “Target application terminated before first instrumented API call”, what does the application do? Does it run, crash, provide any useful output? It seems like the application may be calling abort() before CS can attach.

Are you able to provide any reproducer that the engineering team can use to try and debug this?

Have you tried any of the other tools besides memcheck? (initicheck/synccheck/racecheck)

As an aside - have you tried Nsight Compute? It has an API Stepper that allows you to step through the CUDA API calls and see their returns, etc… it may or may not shine some light on what APIs are being called and if anything unexpected is happening: