Jetson Nano: cuDNN fails when booting from SD card but succeeds when booting from eMMC

guipav · January 13, 2025, 5:52am

Hello,

I have a Jetson Nano with 16 GB eMMC. It has the pre-installed Ubuntu-based OS.
I want to run the cuDNN conv_sample for testing the cuDNN features.
So I have installed nvidia-cuda, nvidia-tensorrt, nvidia-cudnn8, and nvidia-opencv packages through apt-get install command.
Then I tested the conv_sample from /usr/src/cudnn_samples_v8/conv_sample and it seems to be working fine, the tests passed.

However, the 16 GB storage from the eMMC is not enough to fulfill my needs. So I have copied the partition to the USB Flash Drive with 64 GB of storage by running a few commands:

sudo mkfs.ext4 /dev/sda
sudo mount /dev/sda /mnt
sudo cp -ax / /mnt
sudo umount /mnt/

and modified the value of root= in the /boot/extlinux/extlinux.conf file before rebooting.

It boots successfully from the SD card partition. However, when I try to run the same conv_sample, I start getting CUDNN_STATUS_EXECUTION_FAILED errors:

sudo bash run_conv_sample.sh
Executing: conv_sample -c2048 -h7 -w7 -k512 -r1 -s1 -pad_h0 -pad_w0 -u1 -v1
Using format CUDNN_TENSOR_NCHW (for INT8x4 and INT8x32 tests use CUDNN_TENSOR_NCHW_VECT_C)
Testing single precision
====USER DIMENSIONS====
input dims are 1, 2048, 7, 7
filter dims are 512, 2048, 1, 1
output dims are 1, 512, 7, 7
====PADDING DIMENSIONS====
padded input dims are 1, 2048, 7, 7
padded filter dims are 512, 2048, 1, 1
padded output dims are 1, 512, 7, 7
Testing conv
CUDNN error at conv_sample.cpp:721, code=8 (CUDNN_STATUS_EXECUTION_FAILED) in 'cudnnConvolutionForward(handle_, (void*)(&alpha), cudnnIdesc, devPtrI, cudnnFdesc, devPtrF, cudnnConvDesc, algo, workSpace, workSpaceSize, (void*)(&beta), cudnnOdesc, devPtrO)'
Test FAILED, num errors = 1

The CUDA toolkit itself seems to be working well, but the issue seems to be in the Convolutional Forward function.

It is important to mention that I have installed:
CUDA VERSION: 10.2
CUDNN VERSION: 8.2.1.32
Jetpack VERSION: 4.6.6
Python VERSION: 3.6.9

I have tried to reinstall all the nvidia-jetpack packages in the SD Card partition, but it did not solve the issue.

Does anyone have an insight about why it may be failing the tests in this case?

Thank you very much,
Guilherme

guipav · January 22, 2025, 9:22am

I noticed something curious about this behavior.
I have downloaded the libcudnn8_8.2.1.32-1+cuda10.2_arm64.deb package and reinstall it when booting from the SD card (/dev/sda) using: * sudo dpkg -i ~/libcudnn8_8.2.1.32-1+cuda10.2_arm64.deb*

Preparing to unpack .../libcudnn8_8.2.1.32-1+cuda10.2_arm64.deb ...
Unpacking libcudnn8 (8.2.1.32-1+cuda10.2) over (8.2.1.32-1+cuda10.2) ...
Setting up libcudnn8 (8.2.1.32-1+cuda10.2) ...
Processing triggers for libc-bin (2.27-3ubuntu1.6) ...

Then, when I run the bash run_conv_sample.sh script, the tests pass, it seems the cuDNN starts working properly:

jetson@jetson-desktop:/usr/src/cudnn_samples_v8/conv_sample$ bash run_conv_sample.sh
Executing: conv_sample -c2048 -h7 -w7 -k512 -r1 -s1 -pad_h0 -pad_w0 -u1 -v1
Using format CUDNN_TENSOR_NCHW (for INT8x4 and INT8x32 tests use CUDNN_TENSOR_NCHW_VECT_C)
Testing single precision
====USER DIMENSIONS====
input dims are 1, 2048, 7, 7
filter dims are 512, 2048, 1, 1
output dims are 1, 512, 7, 7
====PADDING DIMENSIONS====
padded input dims are 1, 2048, 7, 7
padded filter dims are 512, 2048, 1, 1
padded output dims are 1, 512, 7, 7
Testing conv
^^^^ CUDA : elapsed = 1.68304 sec,
Test PASSED
Testing half precision (math in single precision)
====USER DIMENSIONS====
input dims are 1, 2048, 7, 7
filter dims are 512, 2048, 1, 1
output dims are 1, 512, 7, 7
====PADDING DIMENSIONS====
padded input dims are 1, 2048, 7, 7
padded filter dims are 512, 2048, 1, 1
padded output dims are 1, 512, 7, 7
Testing conv
^^^^ CUDA : elapsed = 0.016382 sec,
Test PASSED
Executing: conv_sample -c512 -h28 -w28 -k128 -r1 -s1 -pad_h0 -pad_w0 -u1 -v1
Using format CUDNN_TENSOR_NCHW (for INT8x4 and INT8x32 tests use CUDNN_TENSOR_NCHW_VECT_C)
Testing single precision
====USER DIMENSIONS====
input dims are 1, 512, 28, 28
filter dims are 128, 512, 1, 1
output dims are 1, 128, 28, 28
====PADDING DIMENSIONS====
padded input dims are 1, 512, 28, 28
padded filter dims are 128, 512, 1, 1
padded output dims are 1, 128, 28, 28
Testing conv
^^^^ CUDA : elapsed = 1.65834 sec,
Test PASSED
Testing half precision (math in single precision)
====USER DIMENSIONS====
input dims are 1, 512, 28, 28
filter dims are 128, 512, 1, 1
output dims are 1, 128, 28, 28
====PADDING DIMENSIONS====
padded input dims are 1, 512, 28, 28
padded filter dims are 128, 512, 1, 1
padded output dims are 1, 128, 28, 28
Testing conv
^^^^ CUDA : elapsed = 0.0119801 sec,
Test PASSED
Executing: conv_sample -c512 -h28 -w28 -k1024 -r1 -s1 -pad_h0 -pad_w0 -u2 -v2
Using format CUDNN_TENSOR_NCHW (for INT8x4 and INT8x32 tests use CUDNN_TENSOR_NCHW_VECT_C)
Testing single precision
====USER DIMENSIONS====
input dims are 1, 512, 28, 28
filter dims are 1024, 512, 1, 1
output dims are 1, 1024, 14, 14
====PADDING DIMENSIONS====
padded input dims are 1, 512, 28, 28
padded filter dims are 1024, 512, 1, 1
padded output dims are 1, 1024, 14, 14
Testing conv
^^^^ CUDA : elapsed = 1.61455 sec,
Test PASSED
Testing half precision (math in single precision)
====USER DIMENSIONS====
input dims are 1, 512, 28, 28
filter dims are 1024, 512, 1, 1
output dims are 1, 1024, 14, 14
====PADDING DIMENSIONS====
padded input dims are 1, 512, 28, 28
padded filter dims are 1024, 512, 1, 1
padded output dims are 1, 1024, 14, 14
Testing conv
^^^^ CUDA : elapsed = 0.027364 sec,
Test PASSED
Executing: conv_sample -c512 -h28 -w28 -k256 -r1 -s1 -pad_h0 -pad_w0 -u2 -v2
Using format CUDNN_TENSOR_NCHW (for INT8x4 and INT8x32 tests use CUDNN_TENSOR_NCHW_VECT_C)
Testing single precision
====USER DIMENSIONS====
input dims are 1, 512, 28, 28
filter dims are 256, 512, 1, 1
output dims are 1, 256, 14, 14
====PADDING DIMENSIONS====
padded input dims are 1, 512, 28, 28
padded filter dims are 256, 512, 1, 1
padded output dims are 1, 256, 14, 14
Testing conv
^^^^ CUDA : elapsed = 1.65665 sec,
Test PASSED
Testing half precision (math in single precision)
====USER DIMENSIONS====
input dims are 1, 512, 28, 28
filter dims are 256, 512, 1, 1
output dims are 1, 256, 14, 14
====PADDING DIMENSIONS====
padded input dims are 1, 512, 28, 28
padded filter dims are 256, 512, 1, 1
padded output dims are 1, 256, 14, 14
Testing conv
^^^^ CUDA : elapsed = 0.00695896 sec,
Test PASSED
Executing: conv_sample -c256 -h14 -w14 -k256 -r3 -s3 -pad_h1 -pad_w1 -u1 -v1
Using format CUDNN_TENSOR_NCHW (for INT8x4 and INT8x32 tests use CUDNN_TENSOR_NCHW_VECT_C)
Testing single precision
====USER DIMENSIONS====
input dims are 1, 256, 14, 14
filter dims are 256, 256, 3, 3
output dims are 1, 256, 14, 14
====PADDING DIMENSIONS====
padded input dims are 1, 256, 14, 14
padded filter dims are 256, 256, 3, 3
padded output dims are 1, 256, 14, 14
Testing conv
^^^^ CUDA : elapsed = 1.67795 sec,
Test PASSED
Testing half precision (math in single precision)
====USER DIMENSIONS====
input dims are 1, 256, 14, 14
filter dims are 256, 256, 3, 3
output dims are 1, 256, 14, 14
====PADDING DIMENSIONS====
padded input dims are 1, 256, 14, 14
padded filter dims are 256, 256, 3, 3
padded output dims are 1, 256, 14, 14
Testing conv
^^^^ CUDA : elapsed = 0.019665 sec,
Test PASSED
Executing: conv_sample -c256 -h14 -w14 -k1024 -r1 -s1 -pad_h0 -pad_w0 -u1 -v1
Using format CUDNN_TENSOR_NCHW (for INT8x4 and INT8x32 tests use CUDNN_TENSOR_NCHW_VECT_C)
Testing single precision
====USER DIMENSIONS====
input dims are 1, 256, 14, 14
filter dims are 1024, 256, 1, 1
output dims are 1, 1024, 14, 14
====PADDING DIMENSIONS====
padded input dims are 1, 256, 14, 14
padded filter dims are 1024, 256, 1, 1
padded output dims are 1, 1024, 14, 14
Testing conv
^^^^ CUDA : elapsed = 1.64909 sec,
Test PASSED
Testing half precision (math in single precision)
====USER DIMENSIONS====
input dims are 1, 256, 14, 14
filter dims are 1024, 256, 1, 1
output dims are 1, 1024, 14, 14
====PADDING DIMENSIONS====
padded input dims are 1, 256, 14, 14
padded filter dims are 1024, 256, 1, 1
padded output dims are 1, 1024, 14, 14
Testing conv
^^^^ CUDA : elapsed = 0.014518 sec,
Test PASSED
Executing: conv_sample -c1024 -h14 -w14 -k256 -r1 -s1 -pad_h0 -pad_w0 -u1 -v1
Using format CUDNN_TENSOR_NCHW (for INT8x4 and INT8x32 tests use CUDNN_TENSOR_NCHW_VECT_C)
Testing single precision
====USER DIMENSIONS====
input dims are 1, 1024, 14, 14
filter dims are 256, 1024, 1, 1
output dims are 1, 256, 14, 14
====PADDING DIMENSIONS====
padded input dims are 1, 1024, 14, 14
padded filter dims are 256, 1024, 1, 1
padded output dims are 1, 256, 14, 14
Testing conv
^^^^ CUDA : elapsed = 1.63442 sec,
Test PASSED
Testing half precision (math in single precision)
====USER DIMENSIONS====
input dims are 1, 1024, 14, 14
filter dims are 256, 1024, 1, 1
output dims are 1, 256, 14, 14
====PADDING DIMENSIONS====
padded input dims are 1, 1024, 14, 14
padded filter dims are 256, 1024, 1, 1
padded output dims are 1, 256, 14, 14
Testing conv
^^^^ CUDA : elapsed = 0.0136962 sec,
Test PASSED
Executing: conv_sample -c1024 -h14 -w14 -k2048 -r1 -s1 -pad_h0 -pad_w0 -u2 -v2
Using format CUDNN_TENSOR_NCHW (for INT8x4 and INT8x32 tests use CUDNN_TENSOR_NCHW_VECT_C)
Testing single precision
====USER DIMENSIONS====
input dims are 1, 1024, 14, 14
filter dims are 2048, 1024, 1, 1
output dims are 1, 2048, 7, 7
====PADDING DIMENSIONS====
padded input dims are 1, 1024, 14, 14
padded filter dims are 2048, 1024, 1, 1
padded output dims are 1, 2048, 7, 7
Testing conv
^^^^ CUDA : elapsed = 1.68824 sec,
Test PASSED
Testing half precision (math in single precision)
====USER DIMENSIONS====
input dims are 1, 1024, 14, 14
filter dims are 2048, 1024, 1, 1
output dims are 1, 2048, 7, 7
====PADDING DIMENSIONS====
padded input dims are 1, 1024, 14, 14
padded filter dims are 2048, 1024, 1, 1
padded output dims are 1, 2048, 7, 7
Testing conv
^^^^ CUDA : elapsed = 0.0313749 sec,
Test PASSED
Executing: conv_sample -c1024 -h14 -w14 -k512 -r1 -s1 -pad_h0 -pad_w0 -u2 -v2
Using format CUDNN_TENSOR_NCHW (for INT8x4 and INT8x32 tests use CUDNN_TENSOR_NCHW_VECT_C)
Testing single precision
====USER DIMENSIONS====
input dims are 1, 1024, 14, 14
filter dims are 512, 1024, 1, 1
output dims are 1, 512, 7, 7
====PADDING DIMENSIONS====
padded input dims are 1, 1024, 14, 14
padded filter dims are 512, 1024, 1, 1
padded output dims are 1, 512, 7, 7
Testing conv
^^^^ CUDA : elapsed = 1.66685 sec,
Test PASSED
Testing half precision (math in single precision)
====USER DIMENSIONS====
input dims are 1, 1024, 14, 14
filter dims are 512, 1024, 1, 1
output dims are 1, 512, 7, 7
====PADDING DIMENSIONS====
padded input dims are 1, 1024, 14, 14
padded filter dims are 512, 1024, 1, 1
padded output dims are 1, 512, 7, 7
Testing conv
^^^^ CUDA : elapsed = 0.00818205 sec,
Test PASSED
Executing: conv_sample -c512 -h7 -w7 -k512 -r3 -s3 -pad_h1 -pad_w1 -u1 -v1
Using format CUDNN_TENSOR_NCHW (for INT8x4 and INT8x32 tests use CUDNN_TENSOR_NCHW_VECT_C)
Testing single precision
====USER DIMENSIONS====
input dims are 1, 512, 7, 7
filter dims are 512, 512, 3, 3
output dims are 1, 512, 7, 7
====PADDING DIMENSIONS====
padded input dims are 1, 512, 7, 7
padded filter dims are 512, 512, 3, 3
padded output dims are 1, 512, 7, 7
Testing conv
^^^^ CUDA : elapsed = 1.71264 sec,
Test PASSED
Testing half precision (math in single precision)
====USER DIMENSIONS====
input dims are 1, 512, 7, 7
filter dims are 512, 512, 3, 3
output dims are 1, 512, 7, 7
====PADDING DIMENSIONS====
padded input dims are 1, 512, 7, 7
padded filter dims are 512, 512, 3, 3
padded output dims are 1, 512, 7, 7
Testing conv
^^^^ CUDA : elapsed = 0.051836 sec,
Test PASSED
Executing: conv_sample -c512 -h7 -w7 -k2048 -r1 -s1 -pad_h0 -pad_w0 -u1 -v1
Using format CUDNN_TENSOR_NCHW (for INT8x4 and INT8x32 tests use CUDNN_TENSOR_NCHW_VECT_C)
Testing single precision
====USER DIMENSIONS====
input dims are 1, 512, 7, 7
filter dims are 2048, 512, 1, 1
output dims are 1, 2048, 7, 7
====PADDING DIMENSIONS====
padded input dims are 1, 512, 7, 7
padded filter dims are 2048, 512, 1, 1
padded output dims are 1, 2048, 7, 7
Testing conv
^^^^ CUDA : elapsed = 1.65965 sec,
Test PASSED
Testing half precision (math in single precision)
====USER DIMENSIONS====
input dims are 1, 512, 7, 7
filter dims are 2048, 512, 1, 1
output dims are 1, 2048, 7, 7
====PADDING DIMENSIONS====
padded input dims are 1, 512, 7, 7
padded filter dims are 2048, 512, 1, 1
padded output dims are 1, 2048, 7, 7
Testing conv
^^^^ CUDA : elapsed = 0.0163169 sec,
Test PASSED
Executing: conv_sample -c2048 -h7 -w7 -k512 -r1 -s1 -pad_h0 -pad_w0 -u1 -v1
Using format CUDNN_TENSOR_NCHW (for INT8x4 and INT8x32 tests use CUDNN_TENSOR_NCHW_VECT_C)
Testing single precision
====USER DIMENSIONS====
input dims are 1, 2048, 7, 7
filter dims are 512, 2048, 1, 1
output dims are 1, 512, 7, 7
====PADDING DIMENSIONS====
padded input dims are 1, 2048, 7, 7
padded filter dims are 512, 2048, 1, 1
padded output dims are 1, 512, 7, 7
Testing conv
^^^^ CUDA : elapsed = 1.66518 sec,
Test PASSED
Testing half precision (math in single precision)
====USER DIMENSIONS====
input dims are 1, 2048, 7, 7
filter dims are 512, 2048, 1, 1
output dims are 1, 512, 7, 7
====PADDING DIMENSIONS====
padded input dims are 1, 2048, 7, 7
padded filter dims are 512, 2048, 1, 1
padded output dims are 1, 512, 7, 7
Testing conv
^^^^ CUDA : elapsed = 0.017077 sec,
Test PASSED
Executing: conv_sample -mathType1 -filterFormat2 -dataType2 -n1 -c512 -h100 -w100 -k64 -r8 -s8 -pad_h0 -pad_w0 -u1 -v1 -b
Using format CUDNN_TENSOR_NCHW_VECT_C (for single and double precision tests use a different format)
Device version 53 does not support int8x4!
Skipping test, SM53 does not support int8x32
Executing: conv_sample -mathType1 -filterFormat2 -dataType2 -n1 -c4096 -h64 -w64 -k64 -r4 -s4 -pad_h1 -pad_w1 -u1 -v1 -b
Using format CUDNN_TENSOR_NCHW_VECT_C (for single and double precision tests use a different format)
Device version 53 does not support int8x4!
Skipping test, SM53 does not support int8x32
Executing: conv_sample -mathType1 -filterFormat2 -dataType2 -n1 -c512 -h100 -w100 -k64 -r8 -s8 -pad_h1 -pad_w1 -u1 -v1 -b
Using format CUDNN_TENSOR_NCHW_VECT_C (for single and double precision tests use a different format)
Device version 53 does not support int8x4!
Skipping test, SM53 does not support int8x32
Executing: conv_sample -mathType1 -filterFormat2 -dataType2 -n1 -c512 -h128 -w128 -k64 -r13 -s13 -pad_h1 -pad_w1 -u1 -v1 -b
Using format CUDNN_TENSOR_NCHW_VECT_C (for single and double precision tests use a different format)
Device version 53 does not support int8x4!
Skipping test, SM53 does not support int8x32
Executing: conv_sample -mathType1 -filterFormat2 -dataType3 -n1 -c512 -h100 -w100 -k64 -r8 -s8 -pad_h0 -pad_w0 -u1 -v1 -b
Using format CUDNN_TENSOR_NCHW_VECT_C (for single and double precision tests use a different format)
Device version 53 does not support int8x4!
Skipping test, SM53 does not support int8x32
Executing: conv_sample -mathType1 -filterFormat2 -dataType3 -n1 -c4096 -h64 -w64 -k64 -r4 -s4 -pad_h1 -pad_w1 -u1 -v1 -b
Using format CUDNN_TENSOR_NCHW_VECT_C (for single and double precision tests use a different format)
Device version 53 does not support int8x4!
Skipping test, SM53 does not support int8x32
Executing: conv_sample -mathType1 -filterFormat2 -dataType3 -n1 -c512 -h100 -w100 -k64 -r8 -s8 -pad_h1 -pad_w1 -u1 -v1 -b
Using format CUDNN_TENSOR_NCHW_VECT_C (for single and double precision tests use a different format)
Device version 53 does not support int8x4!
Skipping test, SM53 does not support int8x32
Executing: conv_sample -mathType1 -filterFormat2 -dataType3 -n1 -c512 -h128 -w128 -k64 -r13 -s13 -pad_h1 -pad_w1 -u1 -v1 -b
Using format CUDNN_TENSOR_NCHW_VECT_C (for single and double precision tests use a different format)
Device version 53 does not support int8x4!
Skipping test, SM53 does not support int8x32
Executing: conv_sample -mathType1 -filterFormat2 -dataType3 -n5 -c32 -h16 -w16 -k32 -r5 -s5 -pad_h0 -pad_w0 -u1 -v1 -b -transformFromNCHW
Using format CUDNN_TENSOR_NCHW_VECT_C (for single and double precision tests use a different format)
Device version 53 does not support int8x4!
Skipping test, SM53 does not support int8x32
Executing: conv_sample -dgrad -c1024 -h14 -w14 -k2048 -r1 -s1 -pad_h0 -pad_w0 -u2 -v2 -fold
Using format CUDNN_TENSOR_NCHW (for INT8x4 and INT8x32 tests use CUDNN_TENSOR_NCHW_VECT_C)
Testing single precision
====USER DIMENSIONS====
input dims are 1, 1024, 14, 14
filter dims are 2048, 1024, 1, 1
output dims are 1, 2048, 7, 7
====PADDING DIMENSIONS====
padded input dims are 1, 1024, 14, 14
padded filter dims are 2048, 1024, 1, 1
padded output dims are 1, 2048, 7, 7
Testing dgrad

 WORKSPACE = 1146699776
^^^^ CUDA : elapsed = 3.10011 sec,
Test PASSED
Testing half precision (math in single precision)
====USER DIMENSIONS====
input dims are 1, 1024, 14, 14
filter dims are 2048, 1024, 1, 1
output dims are 1, 2048, 7, 7
====PADDING DIMENSIONS====
padded input dims are 1, 1024, 14, 14
padded filter dims are 2048, 1024, 1, 1
padded output dims are 1, 2048, 7, 7
Testing dgrad

 WORKSPACE = 1146699776
^^^^ CUDA : elapsed = 1.51487 sec,
Test PASSED

However, this behavior does not persist. After I reboot the Nano, the old behavior reappears.

jetson@jetson-desktop:/usr/src/cudnn_samples_v8/conv_sample$ ./conv_sample
Executing: conv_sample
Using format CUDNN_TENSOR_NCHW (for INT8x4 and INT8x32 tests use CUDNN_TENSOR_NCHW_VECT_C)
Testing single precision
====USER DIMENSIONS====
input dims are 1, 32, 4, 4
filter dims are 32, 32, 1, 1
output dims are 1, 32, 4, 4
====PADDING DIMENSIONS====
padded input dims are 1, 32, 4, 4
padded filter dims are 32, 32, 1, 1
padded output dims are 1, 32, 4, 4
Testing conv
CUDNN error at conv_sample.cpp:721, code=8 (CUDNN_STATUS_EXECUTION_FAILED) in 'cudnnConvolutionForward(handle_, (void*)(&alpha), cudnnIdesc, devPtrI, cudnnFdesc, devPtrF, cudnnConvDesc, algo, workSpace, workSpaceSize, (void*)(&beta), cudnnOdesc, devPtrO)'
Test FAILED, num errors = 1
Testing half precision (math in single precision)
====USER DIMENSIONS====
input dims are 1, 32, 4, 4
filter dims are 32, 32, 1, 1
output dims are 1, 32, 4, 4
====PADDING DIMENSIONS====
padded input dims are 1, 32, 4, 4
padded filter dims are 32, 32, 1, 1
padded output dims are 1, 32, 4, 4
Testing conv
CUDNN error at conv_sample.cpp:721, code=8 (CUDNN_STATUS_EXECUTION_FAILED) in 'cudnnConvolutionForward(handle_, (void*)(&alpha), cudnnIdesc, devPtrI, cudnnFdesc, devPtrF, cudnnConvDesc, algo, workSpace, workSpaceSize, (void*)(&beta), cudnnOdesc, devPtrO)'
Test FAILED, num errors = 1

And then I need to reinstall the deb package again

guipav · January 23, 2025, 6:32am

By investigating further this behavior, I found that the library file /usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so.8.2.1 is the root cause of the failure. For some reason, its md5sum value changed from the expected e16679193b8161954f8dc91f1de4cbe4 to ffd5e716b6b88b434eddc3fbcfcadde9 every time I reboot the Nano.
The two files are identical up to byte 287793153, where the corruption starts.

Manually copying the correct file would temporarily solve the issue, but it sometimes change again the next time I boot the system.
I believe this issue is not caused by the cuDNN libraries, it is most likely a Linux filesystem issue.
So this issue can be closed.

sophwats · January 23, 2025, 10:35am

Thanks for letting us know - pleased you were able to get this resolved.

system · February 6, 2025, 10:36am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Cudnn 7.6.5 conv_sample running in half + nhwc + tensorcore + 2080ti, it gives out of bound error cuDNN	2	594	August 24, 2021
Failed cuDNN test (./mnistCUDNN) cuDNN	24	21088	June 15, 2023
.mtd files corrupted, please help nvidia Jetson Nano	19	2585	October 18, 2021
Failed to get convolution algorithm. This is probably because cuDNN failed to initialize cuDNN	29	51603	October 12, 2021
issues with dlib library Jetson Nano	47	17399	July 28, 2021
Converting a kernel from floats and ints to halfs is 6x slower CUDA Programming and Performance cuda	14	1026	October 16, 2023
Simple Audio Recognition on Jetson Nano Jetson Nano	6	4581	October 18, 2021
Fail to initialize CUDNN when running tensorflow: CUDNN_STATUS_INTERNAL_ERROR Jetson AGX Xavier tensorflow , cudnn	7	2819	October 18, 2021
All algo is CUDNN_STATUS_NOT_SUPPORTED cuDNN	7	1631	October 30, 2022
Caffe make faild Jetson Nano cuda , caffe	11	2043	December 22, 2021

Jetson Nano: cuDNN fails when booting from SD card but succeeds when booting from eMMC

Related topics