cudaErrorNotSupported (code 801) while using cudaGraphAddChildGraphNode or cudaGraphClone() with CUBLAS nodes

[jy55]

You might be seeing this:

cuBLAS 12 graphs cannot be used as child graphs because of stream ordered memory allocation

cuBLAS12 (sometimes) inserts stream allocation nodes, which makes the graph incompatible for use as a child graph (according to the doc for cudaGraphAddMemAllocNode). I run into this in CUDA12 where a call to cudaGraphAddChildGraphNode returns 801 (operation not supported). The same code works in CUDA11 (I’m in the process of upgrading). I gather this allocation has to do with cuBLAS workspace management, but the docs are not entirely clear on the semantics or how to control it for graphs. I …