CUDNN_STATUS_NOT_SUPPORTED error when setting CUDNN_ATTR_ENGINE_GLOBAL_INDEX for Batch Normalization operation

AI & Data Science Deep Learning (Training & Inference) cuDNN
1

I’m encountering an issue while trying to implement a Batch Normalization operation using cuDNN’s backend API. When setting the CUDNN_ATTR_ENGINE_GLOBAL_INDEX attribute for the engine descriptor, I’m receiving a CUDNN_STATUS_NOT_SUPPORTED error. This occurs even when setting the index to 0, which should be valid according to the documentation.

Here’s a snippet of the relevant code:

cudnnBackendDescriptor_t engine;
CHECK_CUDNN(cudnnBackendCreateDescriptor(CUDNN_BACKEND_ENGINE_DESCRIPTOR, &engine));
CHECK_CUDNN(cudnnBackendSetAttribute(engine, CUDNN_ATTR_ENGINE_OPERATION_GRAPH,
                                    CUDNN_TYPE_BACKEND_DESCRIPTOR, 1, &opGraph));
int64_t gidx = 0;
CHECK_CUDNN(cudnnBackendSetAttribute(engine, CUDNN_ATTR_ENGINE_GLOBAL_INDEX,
                                    CUDNN_TYPE_INT64, 1, &gidx));
CHECK_CUDNN(cudnnBackendFinalize(engine));

The error occurs on the line where CUDNN_ATTR_ENGINE_GLOBAL_INDEX is set. I’ve verified that the operation graph is created successfully before this point.

Environment details:

  • cuDNN version: [9.2]
  • CUDA version: 12.4
  • GPU model: 2080TI
  • Operating System: ubuntu

I’ve reviewed the documentation and can’t find any indication that Batch Normalization operations should not support engine global indices. Has anyone encountered a similar issue or can provide insight into why this might be happening? Are there any known limitations or requirements for using engine global indices with Batch Normalization operations that I might be missing?

Any help or guidance would be greatly appreciated. Thank you in advance for your assistance.

2

Reported this with Engineering,
Will update you with the response.

Thanks