Jetson Nano Deep Stream - Could not initialize cudnn

ArgusRo · June 4, 2024, 2:30pm

Hello, I’m trying to run one of the samples from deepstream, but I’m getting the following error:

/opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app$ sudo deepstream-app -c source2_1080p_dec_infer-resnet_demux_int8.txt
WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
ERROR: [TRT]: 1: [executionResources.cpp::setTacticSources::156] Error Code 1: Cudnn (Could not initialize cudnn, please check cudnn installation.)
ERROR: [TRT]: 4: [runtime.cpp::deserializeCudaEngine::50] Error Code 4: Internal Error (Engine deserialization failed.)
ERROR: Deserialize engine failed from file: /opt/nvidia/deepstream/deepstream-6.0/samples/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_fp16.engine
0:00:04.073357256  8204      0x8ac3a90 WARN                 nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1889> [UID = 1]: deserialize engine from file :/opt/nvidia/deepstream/deepstream-6.0/samples/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_fp16.engine failed
0:00:04.074664226  8204      0x8ac3a90 WARN                 nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1996> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/deepstream/deepstream-6.0/samples/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_fp16.engine failed, try rebuild
0:00:04.074727874  8204      0x8ac3a90 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1914> [UID = 1]: Trying to create engine from model files
WARNING: INT8 not supported by platform. Trying FP16 mode.
ERROR: [TRT]: 1: [executionResources.cpp::setTacticSources::156] Error Code 1: Cudnn (Could not initialize cudnn, please check cudnn installation.)
ERROR: Build engine failed from config file
ERROR: failed to build trt engine.
0:00:04.670118768  8204      0x8ac3a90 ERROR                nvinfer gstnvinfer.cpp:632:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1934> [UID = 1]: build engine file failed
0:00:04.672103470  8204      0x8ac3a90 ERROR                nvinfer gstnvinfer.cpp:632:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2020> [UID = 1]: build backend context failed
0:00:04.672256027  8204      0x8ac3a90 ERROR                nvinfer gstnvinfer.cpp:632:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1257> [UID = 1]: generate backend failed, check config file settings
0:00:04.672326290  8204      0x8ac3a90 WARN                 nvinfer gstnvinfer.cpp:841:gst_nvinfer_start:<primary_gie> error: Failed to create NvDsInferContext instance
0:00:04.672394625  8204      0x8ac3a90 WARN                 nvinfer gstnvinfer.cpp:841:gst_nvinfer_start:<primary_gie> error: Config file path: /opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/config_infer_primary.txt, NvDsInfer Error: NVDSINFER_CONFIG_FAILED
** ERROR: <main:707>: Failed to set pipeline to PAUSED
Quitting
ERROR from primary_gie: Failed to create NvDsInferContext instance
Debug info: /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp(841): gst_nvinfer_start (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInfer:primary_gie:
Config file path: /opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/config_infer_primary.txt, NvDsInfer Error: NVDSINFER_CONFIG_FAILED
App run failed

I suppose that my issue is in this:
ERROR: [TRT]: 1: [executionResources.cpp::setTacticSources::156] Error Code 1: Cudnn (Could not initialize cudnn, please check cudnn installation.)

Here are some details about the system:

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Feb_28_22:34:44_PST_2021
Cuda compilation tools, release 10.2, V10.2.300
Build cuda_10.2_r440.TC440_70.29663091_0

uname -a
Linux ubuntu 4.9.299-tegra #1 SMP PREEMPT Tue Nov 22 09:24:39 PST 2022 aarch64 aarch64 aarch64 GNU/Linux

cat /etc/nv_tegra_release
# R32 (release), REVISION: 7.4, GCID: 33514132, BOARD: t210ref, EABI: aarch64, DATE: Fri Jun  9 04:25:08 UTC 2023

./deviceQuery
/usr/local/cuda/samples/1_Utilities/deviceQuery/./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "NVIDIA Tegra X1"
  CUDA Driver Version / Runtime Version          10.2 / 10.2
  CUDA Capability Major/Minor version number:    5.3
  Total amount of global memory:                 3964 MBytes (4156514304 bytes)
  ( 1) Multiprocessors, (128) CUDA Cores/MP:     128 CUDA Cores
  GPU Max Clock rate:                            922 MHz (0.92 GHz)
  Memory Clock rate:                             13 Mhz
  Memory Bus Width:                              64-bit
  L2 Cache Size:                                 262144 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
  Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 32768
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 1 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            Yes
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            No
  Supports Cooperative Kernel Launch:            No
  Supports MultiDevice Co-op Kernel Launch:      No
  Device PCI Domain ID / Bus ID / location ID:   0 / 0 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 10.2, CUDA Runtime Version = 10.2, NumDevs = 1
Result = PASS

Thanks

junshengy · June 5, 2024, 7:04am

Jetson nano can only support up to DS-6.0.x.

cudnn is part of Jetpack, you need to re-burn it, please refer to this link

Here is the compatibility table

https://docs.nvidia.com/metropolis/deepstream/6.2/dev-guide/text/DS_Quickstart.html#id5

ArgusRo · June 5, 2024, 7:14am

What do you mean, I’m running DeepStream 6.0, is that not correct?
Also I don’t have physical access to the unit, needs to be fixed remotely.

Thanks for the quick reply.

junshengy · June 5, 2024, 7:19am

You can use DS-6.0, but Jetson nano cannot run higher versions

Try this guide. If the jetpack is installed correctly, DS-6.0 will run normally.

https://docs.nvidia.com/jetson/archives/jetpack-archived/jetpack-461/install-jetpack/index.html

ArgusRo · June 5, 2024, 11:47am

Thanks for the reply, manage to fix using your’s link suggestion, uninstall everything and deployed the jetpack again using the sdk manager.

Thank you very much.

system · June 19, 2024, 11:48am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Could not initialize cudnn JETSON NANO DeepStream SDK cudnn	8	373	March 12, 2024
New installation Multiple Failues DeepStream SDK	18	1105	June 28, 2022
Error from deepstream sample apps DeepStream SDK	9	1007	October 12, 2021
Is bug ?deepstream docker image 's app run error:Error Code 1: Cudnn (Could not initialize cudnn, please check cudnn installation.) DeepStream SDK	4	1536	February 8, 2022
Deepstream occupancy analytics running issues DeepStream SDK	2	588	April 25, 2022
ERROR: [TRT]: 10: Could not find any implementation for node /0/model.24/Range DeepStream SDK	12	468	July 9, 2024
Failed to deploy efficientdet-tf1 in deepstream DeepStream SDK	2	366	November 6, 2023
There is a error when run deepstream-mrcnn-app DeepStream SDK	60	3078	October 12, 2021
Problems with deepstream docker on Jetson nano DeepStream SDK	13	1030	October 12, 2021
Issue with deepstream-3d-action-recognition sample app DeepStream SDK deepstream	5	140	July 12, 2024

Jetson Nano Deep Stream - Could not initialize cudnn

Related topics