Hi, I am trying to write a script which uses jetson.utils videoSource and videoOutput to run inference with Yolov8. However, right now I am just focusing on getting the input stream, converting the cudaImages to Numpy and back (need this later for Yolo) and then stream it. When running it on the Jetson Orin Dev Kit, everything worked so far, but the predict function from Yolo didnt work anymore, because pytorch didnt have access to the GPU. So I switched to using the default jetson-inference container, installed a few more packages using pip and then commited this to a new image.
Now my current problem: when I run my script (below), I get these 2 errors:
[OpenGL] glDisplay – X screen 0 resolution: 1920x1080
[OpenGL] glDisplay – X window resolution: 1920x1080
[OpenGL] failed to create X11 Window.
AND
failed to find/open file /proc/device-tree/model
I ran video-viewer, which works fine and can create a X11 window, so I dont know whats the problem.
Script:
import sys
from jetson_utils import cudaImage, cudaToNumpy, cudaFromNumpy, cudaConvertColor, cudaAllocMapped, cudaResize, cudaDeviceSynchronize
from jetson_utils import videoSource, videoOutput
from ultralytics import YOLO
import numpy as np
import cv2
from torch import cuda
# create video sources & outputs
output = videoOutput("webrtc://@:8554/my_output")
input = videoSource("/dev/video0")
# load model
model = YOLO("models/yolov8n.onnx")
# capture frames until end-of-stream (or the user exits)
while True:
# get Frame from Source as cuda Image
cuda_img = input.Capture(format='rgb32f', timeout=1000)
if cuda_img is None: # if a timeout occurred
continue
# Resize the image to match YOLOv8's input shape (640x640)
cuda_resized = cudaAllocMapped(width=640, height=640, format=cuda_img.format)
cudaResize(cuda_img, cuda_resized)
# convert imgage to np.ndarray
np_img = cudaToNumpy(cuda_resized)
cudaDeviceSynchronize()
# convert type to float16
np_img = np_img.astype("float16")
# transpose to match yolov8 input shape (3, 640, 640)
np_img = np_img.transpose(2, 0, 1)
# add extra dimension to represent batch size (1, 3, 640, 640)
np_img = np_img[np.newaxis, :]
# inference
#preds = model.predict(np_img, iou=0.5, device=0)
# convert results to np.ndarray
# preds = preds.plot()
# convert back to cuda Image
# Revert the array to the original shape
np_img = np_img.squeeze() # Remove the first dimension (1)
np_img = np_img.transpose(1, 2, 0) #transpose back to 640,640,3
cudaDeviceSynchronize()
image = cudaFromNumpy(np_img)
# Resize possible, but decreases quality because no interpolation, image becomes more "cubic"
# image_resized = cudaAllocMapped(width=1280, height=720, format=image.format)
# cudaResize(image, image_resized)
# render frame with labels
output.Render(image)
# exit on input/output EOS
if not input.IsStreaming() or not output.IsStreaming():
break