YOLOv8 Python Script has really high inference time due unused GPU Memory

I am using YOLOv8 inference on Jetson Orin NX 16GB running Python. The GPU however does not take the load and instead the CPU is still utilized.

Screenshot from 2024-03-18 10-24-35

I imagine CUDA compatibility is an issue here.
Jetpack 6.0, but I manually installed CUDA12.2, cudnn9.0, PyTorch 2.2.1, Torchvision 0.17, CudaCompat-12.2
As per the instruction provided by NVIDIA and PyTorch.

wget https://developer.download.nvidia.com/compute/cudnn/9.0.0/local_installers/cudnn-local-tegra-repo-ubuntu2204-9.0.0_1.0-1_arm64.debsudo dpkg -i cudnn-local-tegra-repo-ubuntu2204-9.0.0_1.0-1_arm64.debsudo cp /var/cudnn-local-tegra-repo-ubuntu2204-9.0.0/cudnn-*-keyring.gpg /usr/share/keyrings/sudo apt-get updatesudo apt-get -y install cudnn

sudo apt-get -y install cudnn-cuda-12

The initilization in the script reads:

import cv2
import pandas as pd
import torch
from ultralytics import YOLO
from tracker import*
import paho.mqtt.client as mqtt
import json
import cuda

tracker = Tracker() # tracks objects and assigns IDs to them

model = YOLO(‘trial_29_yolov8_s_best.pt’)

I have read about specifically setting a target device for cuda but no torch.cuda device was detected.


Sample scripts I use to confirm the utilization of CUDA lead to negative results.

import torch

def print_gpu_properties():
if torch.cuda.is_available():
num_devices = torch.cuda.device_count()
print(f"Number of CUDA devices: {num_devices}“)
for i in range(num_devices):
device = torch.device(f"cuda:{i}”)
properties = torch.cuda.get_device_properties(device)
print(f"Device {i} properties:“)
print(f”\tName: {properties.name}“)
print(f”\tTotal memory: {properties.total_memory / (1024 ** 3):.2f} GB")
print(f"\tCompute capability: {properties.major}.{properties.minor}")
print(“CUDA is not available on this system.”)


I could not verify that Jetson Orin NX was even able to use CUDA12 so that might be the most banal answer. The product documentation specifically mentions CUDA10 cores but I have a Ubuntu 22.4 Hostmachine and I dread setting up a new one for Jetpack 5.0


Please install our prebuilt PyTorch package instead:

The standard way to setup PyTorch on Jetson is:

  1. Setup your system with the same JetPack (including OS, CUDA, cuDNN, … )
  2. Install the PyTorch with the above doc, please select the package according to your JetPack version.


Hi AsataLLL,
I see this Pytorch is just torch but no torchvision and torchaudio. How to install them for jetson platform?


TorchVision and TorchAudio need to be built from the source.
You can find the building instructions below:

Or you can use our container which has all pre-installed.


This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.