Issues Building Pytorch on Jetpack 4.4

[al3xh]

I’m attempting to build a pytorch v1.4 wheel using the instructions found here but am running into some issues that I believe is due to cuDNN being version 8.0 instead of 7.0. For example:

../caffe2/operators/conv_op_cudnn.cc:1173:11: error: ‘CUDNN_CONVOLUTION_BWD_FILTER_SPECIFY_WORKSPACE_LIMIT’ was not declared in this scope
           CUDNN_CONVOLUTION_BWD_FILTER_SPECIFY_WORKSPACE_LIMIT,

../caffe2/operators/conv_op_cudnn.cc:754:56: error: ‘cudnnGetConvolutionForwardAlgorithm’ was not declared in this scope
       CUDNN_ENFORCE(cudnnGetConvolutionForwardAlgorithm(
                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
           cudnn_wrapper_.inline_cudnn_handle(),
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           bottom_desc_,
           ~~~~~~~~~~~~~
           filter_desc_,
           ~~~~~~~~~~~~~
           conv_desc_,
           ~~~~~~~~~~~
           top_desc_,
           ~~~~~~~~~~
           CUDNN_CONVOLUTION_FWD_SPECIFY_WORKSPACE_LIMIT,
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           cudnn_ws_nbytes_limit_,
           ~~~~~~~~~~~~~~~~~~~~~~~
           &algo_));

....

../caffe2/operators/conv_op_cudnn.cc:1255:63: error: ‘cudnnGetConvolutionBackwardDataAlgorithm’ was not declared in this scope
         CUDNN_ENFORCE(cudnnGetConvolutionBackwardDataAlgorithm(
                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
             cudnn_wrapper_.inline_cudnn_handle(),
             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
             filter_desc_,
             ~~~~~~~~~~~~~
             top_desc_,
             ~~~~~~~~~~
             bwd_data_conv_desc_,
             ~~~~~~~~~~~~~~~~~~~~
             bottom_desc_,
             ~~~~~~~~~~~~~
             CUDNN_CONVOLUTION_BWD_DATA_SPECIFY_WORKSPACE_LIMIT,
             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
             cudnn_ws_nbytes_limit_,
             ~~~~~~~~~~~~~~~~~~~~~~~
             &bwd_data_algo_));

I know there are wheels successfully pre-built with those instructions. Am I missing something to be able to build this?

[dusty_nv]

@al3xh

I know there are wheels successfully pre-built with those instructions. Am I missing something to be able to build this?

Hi @al3xh, the wheels for PyTorch 1.4 appear to have been built for JetPack 4.2/4.3 and JetPack 4.4 Developer Preview (which as I recall, came with cuDNN 7 as opposed to cuDNN 8). I would instead try to build PyTorch 1.6 which has wheels for JetPack 4.4 onwards.

[al3xh]

If I needed to build pytorch 1.4 for compatibility reasons, is there a way to do it on Jetpack 4.4? Or is a downgrade to 4.4 Developer Preview required?

[dusty_nv]

@al3xh

If I needed to build pytorch 1.4 for compatibility reasons, is there a way to do it on Jetpack 4.4?

I’m not sure because it would seem like PyTorch 1.4 doesn’t support cuDNN 8, and JetPack 4.4 Developer Preview isn’t recommended to use anymore because it’s not a production release. Is it possible to update your code to be compatible with a newer version of PyTorch? I think that would be your best bet going forward, and for the ongoing maintenance of your application.

Also, I have no idea if this would work and it’s not officially supported, but you could try grabbing the cuDNN 7 files/packages off of JetPack 4.4 Developer Preview and see if you can get them to install/run on JetPack 4.4. As mentioned there may be issues that we’ll be unable to help with there, but if given no other option you could give it a try.