On a mixed CC machine ( 1 * A100 (Index 0), 1 * V100 (Index 1), 1 * P100 (Index 2) ) and using the sample scriptcupti_query.cpp. I am able to query Metrics with the Metrics API; however, I am unable to query Events with the Events API, i.e.:
./cupti_query -device 1 -getmetrics- Succeeds
./cupti_query -device 1 -getdomains- Fails
CUDA Device Id : 1
CUDA Device Name: Tesla V100-PCIE-16GB
Error: cupti_query.cpp:113: Function cuptiDeviceGetNumEventDomains(dev, &maxDomains) failed with error(27): CUPTI_ERROR_NOT_SUPPORTED.
The CUPTI_ERROR_NOT_SUPPORTED error is caused by an incompatibility with the 580 driver branch. While the CUDA 12.9 toolkit supports CUPTI Event and Metric APIs, you must use a driver from the 575 series or earlier to ensure functionality.
@mjain I understand and thank you for the information. How does the Metrics API seemingly continue to work when enumerating events with the 580 driver branch? I would think it would hit the same error as the Events API (CUPTI_ERROR_NOT_SUPPORTED).
I believe you won’t be able to profile events or metrics with the 580 driver branch. While the enumeration of metrics and underlying events works since the metric logic is fully integrated into the CUPTI layer, while querying domains and events requires support from the CUDA driver. I hope this helps in clarifying why some options from the sample cupti_query work.