Darknet slower using Jetpack 4.4 (cuDNN 8.0.0 / CUDA 10.2) than Jetpack 4.3 (cuDNN 7.6.3 / CUDA 10.0)

Anything updated in this topic?

Hi,

You can find some information in our cuDNN release notes.
https://docs.nvidia.com/deeplearning/cudnn/release-notes/rel_8.html#rel-800-Preview__section_f3r_df1_5kb

Known Issues

   ...
  • The performance of cudnnConvolutionBiasActivationForward() is slower than v7.6 in most cases. This is being actively worked on and performance optimizations will be available in the upcoming releases.

Thanks.

Hi,

I have the same problem and fps drops using jetpack 4.4 and cudnn.

Is there an updated version of cudnn available and how to install it on jetson?

Thank you.

Hi,

Currently, our latest software is JetPack 4.4 product release which includes cuDNN v8.0.0.
Thanks.

Is the bug fixed in cudnn 8.0.2 and if yes, can I update the cudnn on the jetson?

Thanks.

I installed old version of cudnn over jetpack 4.4 and performance is fine again:

1 Like

Hello.
my version is Jetpack 4.4
I have same issue(performance is slower than cudnn7).
So I want to down grade my cudnn version but I don’t know how to delete current cudnn package.
How do you reinstall cudnn7.6.5 without reflashing?

Cudnn 7.6.4 for arm: https://developer.nvidia.com/cuda-toolkit/arm
Installation: https://docs.nvidia.com/deeplearning/cudnn/install-guide/index.html

compiling darknet worked fine, I am just compiling opencv to check. [works]

Question: cudnn8.0.3 is released but I cant find a download for arm?

The ARM version for Jetson will be included in the next major JetPack release, please wait for our announcement.

Hey Lars, to install cudnn 7.6.3 on top of jetpack 4.4 was it sufficient to run apt-get remove libcudnn8 and then apt-get install libcudnn7 (And the dev and doc versions)? Is that how you did it ?

I downloaded the arm-version and installed as described in the link:

Basically extracting the compressed file and manually copy the files:


Procedure

  1. Navigate to your directory containing the cuDNN Tar file.
  2. Unzip the cuDNN package.

$ tar -xzvf cudnn-x.x-linux-x64-v8.x.x.x.tgz

or

$ tar -xzvf cudnn-x.x-linux-aarch64sbsa-v8.x.x.x.tgz

  1. Copy the following files into the CUDA Toolkit directory, and change the file permissions.

$ sudo cp cuda/include/cudnn*.h /usr/local/cuda/include

$ sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64

$ sudo chmod a+r /usr/local/cuda/include/cudnn*.h /usr/local/cuda/lib64/libcudnn*

Ah. I see it now. That’s the cudnn 8.xxx version.

I was under the impression that with the original pjreddie version of darknet that cudnn would result in faster times than compiling without it. However, this apparently is only true on the 7.xxx versions. That broke with 8.xxx.

I can’t find a downloadable arm version of the cudnn 7.XXX for the Jetson NX in order to get this speed improvement :-( Only 8.XXX versions.

Cheers,
Kim

Any updates on this topic?
Is the darknet version still slow with the newest jetpack and cudnn8 ?
Kind regards,
Nils

I read in other thread that it is not fixed in this JP update.

I tested with the latest version I could find about two weeks ago. It wasn’t fixed then. I’m just going to have to be patient. The software does work just slower. I would expect that it should eventually be fixed.

Kim

Test environment: t4 cuda10.1+cudnn7.6.5
Model: batch_size=1 MobilenetV3
time: 2.5ms
gpu utilization:50%-70%

Test environment: t4 cuda10.1+cudnn8.0
Model: batch_size=1 MobilenetV3
time: 10ms+
gpu utilization:20%-40%

It seems that cudnn8 has some problems on t4.

1 Like

Hi,

Is there an update on this subject?

In the jetpack 4.5.1 for nano and nano2G the problem still exists…

Thanks,
Luis

Problems are still here on both Jetpack & T4 (running cuDNN 8.2 on this one)
Any news from NVIDIA regarding this ?

Hi,

I wanted to reproduce JetPack 4.4, CUDA 10.2 with cuDNN 7.6.4 (which are compatible according to Support Matrix :: NVIDIA Deep Learning cuDNN Documentation)

how did you manage to get this version?
Thanks.