Using cuDNN Backend to Create a Fused Attention fprop Graph

[kiwimanshare]

@4imothy

L40S

Ok, according to this site NVIDIA L40S GPU the L40S is Ada Lovelace Architecture.

Since you are using CuDnn 9.1.1 → that could explain why you get diffrent error-messages. The newer versions of CuDNN have improved error-handling/logging. I.E. Version 9.3: " Error messages generated during retrieval of the CUDNN_ATTR_ENGINEHEUR_RESULTS attribute are accessible through the cudnnGetLastErrorString() function." from https://docs.nvidia.com/deeplearning/cudnn/latest/release-notes.html

@4imothy

In order to support MHA fprop in the general case would I have to create two operation graphs with separate engine configs and execution plans? One for scaling->matmul and another for softmax->matmul?

I am currently at this point, my best guess is: Yes. Maybe you want to join this discussion: Conceptual questions about how to use cuDNN v9