Darknet slower using Jetpack 4.4 (cuDNN 8.0.0 / CUDA 10.2) than Jetpack 4.3 (cuDNN 7.6.3 / CUDA 10.0)

Hi,

Just want to keep you updated.

This issue can be reproduced in our environment with Xavier.
We are going to check which cuDNN call causes the performance drop.

Thanks.

2 Likes

Good to know ! Good luck ! No problem for now I’m sticking with Jetpack 4.3 ;-)

2 Likes

Hi, I have plan to test darknet with Xavier NX. I have found that there is only 4.4 DP for for Xavier NX. Is it possible that this issue still to Xavier NX?

Hi,

We are working on this.
Will keep this topic updated once we solve this.

Thanks.

I am likewise experiencing ~40% performance drop using a different detection model, with efficientnet backbone and a customized Retina head.

Same problem on PyTorch 1.4
I am follow this page to install PyTorch:

I am testing the YOLOv5:

and found JetPack 4.4 inference time is about 0.25s
the JetPack 4.3 inference time is about 0.14s

Same to me, using our Mask_RCNN(tensorflow) based software on Jetson Xavier AGX.
With JetPack4.4 (not DP), the software do inference in 0.7FPS, while same software process in 1.6FPS with JetPack4.3.

Anything updated in this topic?

Hi,

You can find some information in our cuDNN release notes.
https://docs.nvidia.com/deeplearning/cudnn/release-notes/rel_8.html#rel-800-Preview__section_f3r_df1_5kb

Known Issues

   ...
  • The performance of cudnnConvolutionBiasActivationForward() is slower than v7.6 in most cases. This is being actively worked on and performance optimizations will be available in the upcoming releases.

Thanks.

Hi,

I have the same problem and fps drops using jetpack 4.4 and cudnn.

Is there an updated version of cudnn available and how to install it on jetson?

Thank you.

Hi,

Currently, our latest software is JetPack 4.4 product release which includes cuDNN v8.0.0.
Thanks.

Is the bug fixed in cudnn 8.0.2 and if yes, can I update the cudnn on the jetson?

Thanks.

I installed old version of cudnn over jetpack 4.4 and performance is fine again:

1 Like

Hello.
my version is Jetpack 4.4
I have same issue(performance is slower than cudnn7).
So I want to down grade my cudnn version but I don’t know how to delete current cudnn package.
How do you reinstall cudnn7.6.5 without reflashing?

Cudnn 7.6.4 for arm: https://developer.nvidia.com/cuda-toolkit/arm
Installation: https://docs.nvidia.com/deeplearning/cudnn/install-guide/index.html

compiling darknet worked fine, I am just compiling opencv to check. [works]

Question: cudnn8.0.3 is released but I cant find a download for arm?

The ARM version for Jetson will be included in the next major JetPack release, please wait for our announcement.

Hey Lars, to install cudnn 7.6.3 on top of jetpack 4.4 was it sufficient to run apt-get remove libcudnn8 and then apt-get install libcudnn7 (And the dev and doc versions)? Is that how you did it ?

I downloaded the arm-version and installed as described in the link:

Basically extracting the compressed file and manually copy the files:


Procedure

  1. Navigate to your directory containing the cuDNN Tar file.
  2. Unzip the cuDNN package.

$ tar -xzvf cudnn-x.x-linux-x64-v8.x.x.x.tgz

or

$ tar -xzvf cudnn-x.x-linux-aarch64sbsa-v8.x.x.x.tgz

  1. Copy the following files into the CUDA Toolkit directory, and change the file permissions.

$ sudo cp cuda/include/cudnn*.h /usr/local/cuda/include

$ sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64

$ sudo chmod a+r /usr/local/cuda/include/cudnn*.h /usr/local/cuda/lib64/libcudnn*

Ah. I see it now. That’s the cudnn 8.xxx version.

I was under the impression that with the original pjreddie version of darknet that cudnn would result in faster times than compiling without it. However, this apparently is only true on the 7.xxx versions. That broke with 8.xxx.

I can’t find a downloadable arm version of the cudnn 7.XXX for the Jetson NX in order to get this speed improvement :-( Only 8.XXX versions.

Cheers,
Kim