OpenCV window not showing video source from Jetson Inference Object Detection

[aldhanekadev]

Hi, so after I looked up some threads and github discussion about how to grab frame from jetson inference, it turns out I need to convert it into numpy. What I’m trying to do is, to grab frame from jetson inference with the box detection included, like this thread Detectnet - object in frames - #3 by nerk , I need to convert captured camera to numpy

bgr_img = jetson.utils.cudaFromNumpy(cv_img, isBGR=True)

and also the sample from this thread is not enough so I look for another thread and it’s from Jetson Inference GitHub Issue, at the end of this github issue @dust-nv said to convert the camera captured to opencv, we need to add additional:

bgr_img = jetson.utils.cudaAllocMapped(width=rgb_img.width,height=rgb_img.height,format='bgr8')

and from those two threads, I ended up with this code

import numpy as np
import math
import jetson_inference
import jetson_utils
import cv2

from network import NetworkTable

# e = argparse.ArgumentParser
net = jetson_inference.detectNet(argv=['--model=models/roboflow.ai/ssd-mobilenet-4.onnx', '--labels=models/roboflow.ai/labels.txt', '--input-blob=input_0', '--output-cvg=scores', '--threshold=0.4', '--output-bbox=boxes'])
camera = jetson_utils.videoSource("v4l2:///dev/video0", ['--input-width=640', '--input-height=640'])

speed=0.0

if __name__ == '__main__':
    
    bgr_img = None
    cv_img = None
    networkTable = NetworkTable()

    # print("camera.IsStreaming()", camera.IsStreaming())
    while True:
        img = camera.Capture()
        detections = net.Detect(img, width=img.width, height=img.height, overlay='none')
	#cv_img = jetson.utils.cudaToNumpy(img)
        
        if bgr_img is None:
            bgr_img = jetson_utils.cudaAllocMapped(width=img.width, height=img.height, format="bgr8")
            jetson_utils.cudaConvertColor(img, bgr_img)
            cv_img = jetson_utils.cudaToNumpy(bgr_img)

        jetson_utils.cudaDeviceSynchronize()

        #get the object with the highest weight
        currentFocusObj = None

        angleSolver = AngleSolver()
        classID = ""
        for detection in detections:
            x1 = detection.Left
            x2 = detection.Right
            y1 = detection.Top
            y2 = detection.Bottom

            
            print("x1: ", x1)
            print("x2: ", x2)
            print("y1: ", y1)
            print("y2: ", y2)
            
            # cv2.rectangle(cv_img, (x1, y1), (x2, y2), (0, 0, 255), 2)


            classID = net.GetClassDesc(detection.ClassID)
            object = Object([x1,y1,x2,y2], classID)

            if object.compareWeight(currentFocusObj):
                currentFocusObj = object
            #info.OverlayText(cv_img, 5, 5,"speed:{:.2f}".format(speed),int(detection.Left)+5, int(detection.Top)+35, info.White, info.Gray40)


        
        cv2.namedWindow("Output", cv2.WINDOW_NORMAL)
        cv2.imshow("Output", cv_img)
        #new_img = jetson_utils.cudaFromNumpy(cv_img, isBGR=True)
        
        #display.Render(new_img)
        #display.SetStatus("Object Detection | Network {:.0f} FPS".format(net.GetNetworkFPS()))
        
        #print(f"Angle to obj: {}")
        if currentFocusObj == None:
            continue
        angleToObject = angleSolver(currentFocusObj)
        
        print(f"Angle to obj {angleToObject} \n ClassID {classID}")
        vals = [angleToObject, currentFocusObj.classID]

        networkTable.putData(angleToObject, classID)


    cv2.destroyAllWindows()

When I run the code, it runs without any error, however the output window doesn’t give any thing, it just transparent window, I expect the window should be captured frame from camera

[aldhanekadev]

my python version is 3.7 on Jetson Nano

[Honey_Patouceul]

You need to call waitKey() after imshow():

    cv2.imshow("Output", cv_img)
    cv2.waitKey(1)

[aldhanekadev]

@Honey_Patouceul

    cv2.waitKey(1)

what does waitKey mean?

[aldhanekadev]

ok theres a frame in the output window, but it freeze, hm how can I fix this?

[aldhanekadev]

The frame freezed it turns out because of this line

if bgr_img is None:
            bgr_img = jetson_utils.cudaAllocMapped(width=img.width, height=img.height, format="bgr8")
            jetson_utils.cudaConvertColor(img, bgr_img)
            cv_img = jetson_utils.cudaToNumpy(bgr_img)

        jetson_utils.cudaDeviceSynchronize()

so it converts Cuda to numpy only once, while we have to convert this everytime. So the solution is;

        bgr_img = jetson_utils.cudaAllocMapped(width=img.width, height=img.height, format="bgr8")
        jetson_utils.cudaConvertColor(img, bgr_img)
        cv_img = jetson_utils.cudaToNumpy(bgr_img)

        jetson_utils.cudaDeviceSynchronize()

I was wrapped those lines with if statement because on this Github Issue, @dust-nv said that " For performance reasons, you don’t want to call jetson.utils.cudaAllocMapped() every frame. If you only call it once, you only need to call cudaToNumpy() once, because it is a persistent mapping (and any changes CUDA make, OpenCV will see). You can do something like this:" , but it seems like it prevent opencv to update the frame.

Anyway, thanks for @Honey_Patouceul to help me out! I really appreciate that.

[Honey_Patouceul]

Not sure I correctly understand what you’re trying, but if this can help:

You would do allocation of CUDA buffer once in initialisation (with CV Window creation):

bgr_img = jetson_utils.cudaAllocMapped(width=img.width, height=img.height, format="bgr8")
cv2.namedWindow("Output", cv2.WINDOW_NORMAL)

and in the loop you would read, convert, copy and display:

    while True:
        img = camera.Capture()
        jetson_utils.cudaConvertColor(img, bgr_img)
        cv_img = jetson_utils.cudaToNumpy(bgr_img)
        jetson_utils.cudaDeviceSynchronize()

        cv2.imshow("Output", cv_img)
        cv2.waitKey(1)