profiling/debugging tools don't support CUDA dynamic parallelism on Volta and Turing?

Shiny new RTX 2080 card, shiny new CUDA 10 installed

this is nvprof

==5776== Warning: CDP tracing and profiling are not supported on devices with compute capability 7.0 and later.

this is cuda-memcheck (synccheck)

========= Internal Memcheck Error: CUDA Dynamic Parallelism is not supported

Really now? I upgrade all my hardware only to get a downgrade in the supported feature set of the toolkit.


Are you sure this points to a regression in the feature set of the software? To me it looks more like an indication that relevant hardware mechanisms may have been discontinued, which could also explain why the cublas_device library no longer exists.

If my interpretation holds, one wonders whether these features are gone for good because NVIDIA reclaimed the silicon real estate for other functionality that drives business today, or whether some of them may re-appear (or be retained) in future product offerings. Not that NVIDIA would comment on such plans publicly, they never do as a matter of policy.

The CUDA language and driver continue to support CUDA Dynamic Parrelism. The message is informing you that some of the developer tools do not support CUDA Dynamic Parallelism.

I don’t think anybody assumed otherwise. But tools like profilers and debuggers rely on hardware hooks, and those presumably went way. An alternative explanation would be that these features were dropped from the tools as they can no longer be maintained due to personnel resource constraints; quite unlikely giving the overall shape NVIDIA is in :-)

I don‘t feel inclined to use the dynamic parallelism feature if there is no good way to profile and debug such kernels.

Just like I would not drive a car with no seatbelts in it - even if it‘s a perfectly fine and tempting sports car.

Pure speculation: Maybe the maintainers of cublas_device felt the same way …

I am likewise just guessing that hardware changes are to blame for the reduced functionality. But I claim that this is a reasonably intelligent guess: as a software engineer whose entire career was spent working for hardware companies I recall all too well the instances where software folks got the blame for what were ultimately hardware issues / limitations.