A plan of operation graph "convolution+biasadd+relu" cannot be finalized

[zuxian]

cudnn_cbr_sample.cpp (24.2 KB)
Makefile (6.5 KB)

Attachment is a demo code of an operation graph “convolution+biasadd+relu”. It has some error in finalizing the plan. But I have no idea about the mitigations through the cudnn log.

Can anyone help?

Hardware: NVIDIA GeForce RTX 3060
CUDA Version: 11.8
CUDNN Version: 8.6.0.163-1+cuda11.8
OS: Ubuntu 18.04

[AakankshaS]

Hi @zuxian ,
Can you please help us with the detailed error logs to debug it further.

Thanks

[zuxian]

Hi @AakankshaS ,

Here is the log with CUDNN_LOGINFO_DBG, CUDNN_LOGWARN_DBG, CUDNN_LOGERR_DBG opened:

log.txt (124.3 KB)

The error logging is:

E! CuDNN (v8500) function cudnnBackendFinalize() called:
e!     Info: Traceback contains 1 message(s)
e!         Error: CUDNN_STATUS_BAD_PARAM; Reason: finalize_internal()
e! Time: 2022-10-27T16:07:52.751669 (0d+0h+0m+0s since start)
e! Process=2822322; Thread=2822322; GPU=NULL; Handle=NULL; StreamId=NULL.

Please take a look. Thanks.

[zuxian]

Hi @AakankshaS . Is there any update?

[AakankshaS]

Hi @zuxian ,
We are checking on this. Please allow us some time.
Thank you for your patience.

[ryantong]

@AakankshaS i was also trying to do something similar to this and was wondering if there are updates on this

[ryantong]

@zuxian

CUDNN Version: 8

@zuxian did you ever solve this issue?

[zuxian]

Not yet. @ryantong

[zuxian]

@AakankshaS Any update on this?

[zuxian]

cudnn_cbr_sample.cpp (24.2 KB)

Finally, I got this pattern run-through. Some changes might be highlighted:

• The cross-correlation algorithm should be used;
• The first global engine requires NHWC tensor layout;
• CUDNN_ATTR_EXECUTION_PLAN_HANDLE is a required attribute in plan descriptor;

@ryantong FYI.

[zuxian]

Also, the validated platform is updated:

• Ubuntu 22.04, x86_64, 5.15.0-60-generic
• CUDA 12.0, driver version 525.85.12
• libcudnn8-8.8.0.121-1+cuda12.0
• gcc/g++ 10.4.0

[zhang662817]

niubility