A plan of operation graph "convolution+biasadd+relu" cannot be finalized

AI & Data Science Deep Learning (Training & Inference) cuDNN
1

cudnn_cbr_sample.cpp (24.2 KB)
Makefile (6.5 KB)

Attachment is a demo code of an operation graph “convolution+biasadd+relu”. It has some error in finalizing the plan. But I have no idea about the mitigations through the cudnn log.

Can anyone help?

Hardware: NVIDIA GeForce RTX 3060
CUDA Version: 11.8
CUDNN Version: 8.6.0.163-1+cuda11.8
OS: Ubuntu 18.04

2

Hi @zuxian ,
Can you please help us with the detailed error logs to debug it further.

Thanks

3

Hi @AakankshaS ,

Here is the log with CUDNN_LOGINFO_DBG, CUDNN_LOGWARN_DBG, CUDNN_LOGERR_DBG opened:

log.txt (124.3 KB)

The error logging is:

E! CuDNN (v8500) function cudnnBackendFinalize() called:
e!     Info: Traceback contains 1 message(s)
e!         Error: CUDNN_STATUS_BAD_PARAM; Reason: finalize_internal()
e! Time: 2022-10-27T16:07:52.751669 (0d+0h+0m+0s since start)
e! Process=2822322; Thread=2822322; GPU=NULL; Handle=NULL; StreamId=NULL.

Please take a look. Thanks.

4

Hi @AakankshaS . Is there any update?

5

Hi @zuxian ,
We are checking on this. Please allow us some time.
Thank you for your patience.

1 Like
6

@AakankshaS i was also trying to do something similar to this and was wondering if there are updates on this

7

@zuxian did you ever solve this issue?

8

Not yet. @ryantong

9

@AakankshaS Any update on this?

10

cudnn_cbr_sample.cpp (24.2 KB)

Finally, I got this pattern run-through. Some changes might be highlighted:

  • The cross-correlation algorithm should be used;
  • The first global engine requires NHWC tensor layout;
  • CUDNN_ATTR_EXECUTION_PLAN_HANDLE is a required attribute in plan descriptor;

@ryantong FYI.

1 Like
11

Also, the validated platform is updated:

  • Ubuntu 22.04, x86_64, 5.15.0-60-generic
  • CUDA 12.0, driver version 525.85.12
  • libcudnn8-8.8.0.121-1+cuda12.0
  • gcc/g++ 10.4.0
1 Like
12

niubility

1 Like
13

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.