Cuda not detecting on SE70

Hi folks,
We have been trying to install cuda on se70 to enable GPU acceleration for our computer vision task. Unfortunately its not able to compile cuda with torch and therefore we are not able to get GPU acceleration.

torch.cuda.is_available() - False

JETPACK - 5.0.2
Ubuntu - 20
System - SE70
Torch - 2.0.1 (tried with torch 1.12,1.13)

May I know what SE70 is?

By SE70 I mean Jetson Xavier NX.

Hi,

Do you use our prebuilt package?
Third-party packages might not built with CUDA support.

You can install our prebuilt by following the below guide:
https://docs.nvidia.com/deeplearning/frameworks/install-pytorch-jetson-platform/index.html

Thanks

We are using prebuilt packages still the issue persist.

Hi,

It looks like the SE70 is a third-party custom board.
Could you check if CUDA can work correctly first?

$ cd /usr/local/cuda/samples/1_Utilities/deviceQuery
$ sudo make
$ ./deviceQuery

Thanks.

infy@infy-gpu:/usr/local/cuda/samples/1_Utilities/deviceQuery$ ./deviceQuery
./deviceQuery Starting…

CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: “Xavier”
CUDA Driver Version / Runtime Version 11.4 / 11.4
CUDA Capability Major/Minor version number: 7.2
Total amount of global memory: 14774 MBytes (15491137536 bytes)
(006) Multiprocessors, (064) CUDA Cores/MP: 384 CUDA Cores
GPU Max Clock rate: 1109 MHz (1.11 GHz)
Memory Clock rate: 1109 Mhz
Memory Bus Width: 256-bit
L2 Cache Size: 524288 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total shared memory per multiprocessor: 98304 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 1 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: Yes
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device supports Managed Memory: Yes
Device supports Compute Preemption: Yes
Supports Cooperative Kernel Launch: Yes
Supports MultiDevice Co-op Kernel Launch: Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 0 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 11.4, CUDA Runtime Version = 11.4, NumDevs = 1
Result = PASS

Code :
import torch
print(f’PyTorch version: {torch.version}‘)
print(f’CUDNN version: {torch.backends.cudnn.version()}’)
print(f’Available GPU devices: {torch.cuda.device_count()}')
Output:
“PyTorch version: 2.0.1
CUDNN version: None
Available GPU devices: 0”

This was output when i ran the above mentioned commands.

Hi,

How do you install PyTorch?
Do you install the packages from the below link?
https://developer.download.nvidia.com/compute/redist/jp/v502/pytorch/

Based on the log, it can not find or doesn’t built with cuDNN.
Have you installed all the JetPack components?

Thanks.

Hi,

Thanks for your reply.
For PyTorch we tried couple of methods.

  1. PyTorch for Jetson
  2. PyTorch Official website. → https://pytorch.org/

Yes, exactly as per the logs it fails to build cuDNN which is essential for GPU acceleration.

Can you elaborate what do you exactly mean by Jetpack Component?
Yes we tried with jetson xavier which was already shipped with JETPACK (component installed).

Thanks.

Hi,

It looks like your PyTorch fails to find cuDNN.
There are two possibilities: PyTorch doesn’t build with cuDNN or uses an incompatible version.

First, please check if cuDNN can work well in your environment.

$ cd /usr/src/cudnn_samples_v8/conv_sample/
$ sudo make
$ ./conv_sample
...
Test PASSED

Then make sure you have installed the PyTorch that is built on the same JetPack version.
For example, check the JetPack version of your device with the following command:

$ apt show nvidia-jetpack
Package: nvidia-jetpack
Version: 5.1.1-b56
Priority: standard

Then compare it to the package link that also contains the JetPack version.
https://developer.download.nvidia.cn/compute/redist/jp/v511/pytorch/torch-2.0.0+nv23.05-cp38-cp38-linux_aarch64.whl

Thanks.

Hi,

Thanks for your prompt reply.
We tried those command.


Refer the last command in the above image.
Its unable to locate the cudnn_samples folder. Let me know if i am missing anything

$ apt show nvidia-jetpack
Package: nvidia-jetpack 5.0.1b dev
Ubuntu: 20.04.6 LTS

The last link the above reply seems to be dead. Can you please check from your end.

Regards,
Sarib Zama

Hi,

It looks like you are using JetPack 5.0.1 rather than the 5.0.2 mentioned at the top of the topic.
Since 5.0.1 is a DP version, would you mind reflashing your device into JetPack 5.0.2 or newer?

For PyTorch, you can find the detailed instructions in the below doc:

Thanks.

Hi,

Thanks for the quick reply.
Earlier we tried downgrading to ubuntu 18 and upgrading the Jetpack to the latest version. Unfortunately there was some installation failure. Let us know if we are missing something.

Regards,
Sarib Zama

Hi,

Could you share more about the failure?

Do you fail to launch SDKmanager or SDKmanager fails to reflash the device?
Could you share the error log with us so we can know more about the issue?

In general, you can reflash and install Jetson with the instructions below:
https://docs.nvidia.com/sdk-manager/install-with-sdkm-jetson/index.html

Thanks.

Hi,

We tried downgrading ubuntu quite a while ago before reaching out to you guys. We don’t have logs handy but if I remeber well it was something related to dependecy failure(SDK fails to reflash the device).

Yes, we followed the above documentation while we were trying to reflash the device by using the target machine as se70 and host machine as our laptop.

Thanks & Regards,
Sarib Zama

Hi,

The Ubuntu you indicate is a host machine or Jetson?
For a non-supported host version, please try our SDKmanager container.

https://docs.nvidia.com/sdk-manager/1.2/docker-containers/index.html

To move forward, would you mind reflashing the device again and helping us collect the error if it fails?

Thanks.

Hi,

We have successfully downgraded to ubuntu 18.0. We couldn’t check whether cuda is detecting or not since we couldn’t import torch (Illegal instruction - core dumped). I am attaching the screenshots for better understanding.

Thanks & Regards.
Sarib Zama



Hi,

Please install NumPy 1.19.4 and try it again.

pip3 install numpy==1.19.4

Thanks.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.