cuptiProfilerGetCounterAvailability Causes a Segmentation Fault with Cuda Toolkit 13.0.0 When Using Dynamic Shared Libraries

Developer Tools CUPTI – CUDA Profiler Tools Interface
1

Machine Information:

  • NVIDIA GPU: 1 * GH200
  • OS: RHEL 9.4
  • Driver version: 580.65.06

Background:

I have a workflow that loads the dynamic shared libraries for the Driver, Runtime, CUPTI, and PerfWorks Metrics libraries respectively via dlopen. Once opened, function pointers are created via dlsym. These function pointers are then used throughout the rest of the workflow to profile NVIDIA GPUs . Once the workflow has completed, dlcloseis called to close the open dynamic shared libraries and the function pointers are set equal to NULL.

Issue:

Starting in Cuda Toolkit 13.0.0 a sanity test began to have a segmentation fault that occurred for it. In previous versions such as Cuda Toolkit 12.8.1 this did not occur.

After some debugging, I found that the root cause of this segmentation fault beginning in Cuda Toolkit 13.0.0 resulted from the call in my workflow to cuptiProfilerGetCounterAvailability.

Reproducer:

To reproduce this issue I am attaching a tarball titled seg_fault_reproducers.tar.gz (3.3 KB) that contains the files:

  • Makefile
  • profiler_reproducer.c
  • reproducers.h
  • work_reproducer.cu

Once you unzip the tarball you must:

  1. Update the path assigned to the variable pathToDriver to the location of your driver dynamic shared library.
  2. Set CUDA_HOME to the root of your Cuda Toolkit installation, i.e. export CUDA_HOME=/packages/cuda/13.0.0.

After that is done, you can run make and ./work_reproducer to recreate the segmentation fault. I have verified this workflow on the same machine outlined in “Machine Information“. Please let me know if there are any issues here.

Notes:

  • This reproducer is a trimmed down version of the actual workflow and has been made to trigger the segmentation fault only.

Questions:

  • Solutions that I have found that work are (1) calling cuptiFinalizeand (2) moving the shutdownto after cuCtxDestroy. Are there any other recommended solutions?