Hi, I was adding some activation layers from CUDNN not currently present in dlib, in particular:
See PR details here.
While everything works fine in the CI tests for the CPU versions, on my local machine I have CUDA 11.2.0 and CUDNN 126.96.36.199, and the backward pass for the ELU activation doesn’t work (the gradient is not computed correctly and any training that uses that layer makes the loss go to -NaN).
I have implemented myself a CUDA backward pass in dlib and that works. However, I would like to know why the CUDNN implementation does not work.