CUDA Profiler Tools Interface (CUPTI) 13.4 is now available

Developer Tools CUPTI – CUDA Profiler Tools Interface
1

CUDA Profiler Tools Interface (CUPTI) 13.4 is now available for download in the NVIDIA Registered Developer Program.

The NVIDIA CUDA Profiler Tools Interface (CUPTI), distributed as part of the CUDA Toolkit, is a library that enables the creation of profiling and tracing tools that target CUDA applications. CUPTI provides a set of APIs targeted at ISVs creating profilers and other performance optimization tools:
  • the Activity API,
  • the Callback API,
  • the Host Profiling API,
  • the Range Profiling API,
  • the PC Sampling API,
  • the SASS Metric API,
  • the PM Sampling API,
  • the Checkpoint API,
  • the Profiling API,
  • the Python API (available separately)
Using these CUPTI APIs, independent software developers can create profiling tools that provide low and deterministic profiling overhead on the target system, while giving insight into the CPU and GPU behavior of CUDA applications.

Updates in CUDA Toolkit 13.4 Update 1

    Resolved Issues

    • Fixed unbounded CPU utilization growth during long-running activity tracing, improving performance stability for continuous GPU workload tracing.

    • Fixed a race condition in CUPTI HES mode that caused intermittent crashes during cudaGraphInstantiate by properly synchronizing related operations.

    • Fixed a memory leak when HES is enabled that caused approximately 100 bytes of host memory to leak per kernel activity record.

    • Fixed a system crash and intermittent zero-valued timestamps in HES mode during concurrent context destruction.

    • Fixed a deadlock in kernel latency timestamp collection that could occur when a device buffer was allocated during kernel launch.

    • Fixed stale kernel names in activity records after module reloads.

Updates in CUDA Toolkit 13.4

    New Features

    • Added support for the Rubin GPU architecture.

    • Fixed Clock Rate - Added APIs cuptiClockControlLock() and cuptiClockControlUnlock() with CUPTI_CLOCK_CONTROL_MODE_BASE or CUPTI_CLOCK_CONTROL_MODE_BOOST to lock GPU clocks and eliminate clock-rate variability during profiling. Refer to the section Fixed Clock Rate for more details.

    • Cache State Reset - Range Profiler can now reset L2 (or all) cache before each replay pass via the cacheControlMode field in CUpti_RangeProfiler_Enable_Params. Refer to the section Cache State Reset for more details.

    • Added sourceGraphId to provide the unique graph ID of the node from which the kernel node was instantiated or last updated and sourceGraphNodeId which provides unique graph node ID of the node from which the kernel node was instantiated or last updated. The activity record CUpti_ActivityKernel12 is deprecated and replaced by CUpti_ActivityKernel13.

    • To ensure tracing and profiling accuracy, live vGPU migration is blocked while CUPTI is active and resumes only after cuptiFinalize() is called or the process exits.

    • Added workqueueChannelCount and workqueueChannelIds fields to the green context activity record to report the number of HW channels assigned to a green context s work queue and their channel IDs, respectively. The activity record CUpti_ActivityGreenContext2 is deprecated and replaced by CUpti_ActivityGreenContext3.

    • CUPTI library installation path: For Windows, libraries are now placed under architecture-specific directories instead of the generic lib64 folder to support proper cross-compile support. For ex: on a x86_64 system, the CUPTI libraries are installed under C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA<version>\extras\CUPTI\lib\x64.

    Resolved Issues

    • Per-subscriber timestamp callback registration via CUPTI_ACTIVITY_ATTR_TIMESTAMP_CALLBACK using cuptiActivitySetAttribute_v2 is now supported in multi-subscriber mode, allowing tools to register their own callbacks rather than sharing a common callback.

    • Fixed issues that could occur after disabling user-defined records via the CUPTI_ACTIVITY_ATTR_USER_DEFINED_RECORDS attribute.

    • Fixed an issue that sometimes causes a brief (~2 s) CUPTI block when disabling the CUPTI_ACTIVITY_KIND_UNIFIED_MEMORY_COUNTER activity.

    • Fixed a host heap corruption issue that could occur when using CUPTI HES trace (cuptiActivityEnableHWTrace).

    • Fixed CUPTI HES activity so that graphNodeId field retains the full graphId (upper 32-bits) in CUDA-graph memory copy records.

Getting Started

Latest PRODUCT INFO