Blank window when trying to view numpy array converted from CUDA capsule

Hello,

I am trying to convert a CUDA capsule to numpy array using the cudaToArray function. When I try to view the numpy array, the cv2 window just returns a blank screen. Please see the script below:

import jetson.inference
import jetson.utils
import ctypes
import sys
import cv2
from imutils.video import VideoStream
import imutils
import time
import os
import numpy as np

# Parameters
video_file = 'toronto.mp4'
network = 'fcn-resnet18-cityscapes-1024x512'
print('Done setting the parameters...')
width = 1024 # Desired output width
height = 512 # Desired output height

# Setting the network
net = jetson.inference.segNet(network)
print('Finished loading the pretrained network...')

# Defining the video stream
vs = cv2.VideoCapture(os.path.join(os.getcwd(), video_file))
print('Started the defined video...')

# Creating allocated memory for output image
img_overlay = jetson.utils.cudaAllocMapped(width*height*4*ctypes.sizeof(ctypes.c_float))

time.sleep(2.0)

while True:
    grabbed, frame = vs.read()

    if grabbed is False:
        print('Didn\'t capture any frame')
        break

    # Creating a copy of the original frame
    out_frame = frame.copy()
    out_frame = imutils.resize(out_frame, width = width)

    # Resizing the frame
    frame = cv2.resize(frame, (width, height)) 

    # Converting to RGBA format for Jetson Inference
    frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    frame = cv2.cvtColor(frame, cv2.COLOR_RGB2RGBA)

    # Converting to jetson capsule for optimized performance
    cuda_frame = jetson.utils.cudaFromNumpy(frame)

    # Running the inference model on the capsule
    net.Process(cuda_frame, width, height)

    # Converting the output to numpy array
    out_frame = jetson.utils.cudaToNumpy(img_overlay, width, height, 4)
    out_frame = (out_frame*255).astype(np.uint8)

    cv2.imshow('Output', out_frame)
    key = cv2.waitKey(1) & 0xFF

    if key == ord('q'):
        break

vs.release()
cv2.destroyAllWindows()

Could anyone point out if there is anything wrong with the script?

Hi @satyaakhilreddyb, you never generate the overlay after the initial processing - try adding this beneath net.Process()

    net.Overlay(img_overlay, width, height, filter_mode='linear')
1 Like

Thank you for the input @dusty_nv. I was able to solve the problem with the above step, but I have run into another problem. The output is something like

Is it because of the cudaToArray conversion? I checked the input frames and they seem to be correct.

I have tried changing the code over several iterations. This was final output I could come up with:

The script is as follows:

import jetson.inference
import jetson.utils
import ctypes
import sys
import cv2
from imutils.video import VideoStream
import imutils
import time
import os
import numpy as np

# Parameters
video_file = 'toronto.mp4'
network = 'fcn-resnet18-cityscapes-1024x512'
print('Done setting the parameters...')
width = 512 # Desired output width
height = 256 # Desired output height

# Setting the network
net = jetson.inference.segNet(network)
print('Finished loading the pretrained network...')

# Defining the video stream
vs = cv2.VideoCapture(os.path.join(os.getcwd(), video_file))
print('Started the defined video...')

# Creating allocated memory for output image
img_overlay = jetson.utils.cudaAllocMapped(width*height*4*ctypes.sizeof(ctypes.c_float))

time.sleep(2.0)

while True:
    grabbed, frame = vs.read()

    if grabbed is False:
        print('Didn\'t capture any frame')
        break

    # Creating a copy of the original frame
    #out_frame = frame.copy()
    #out_frame = imutils.resize(out_frame, width = width)

    # Resizing the frame
    frame = cv2.resize(frame, (width, height))
    cv2.imshow('Input', frame) 

    # Converting to RGBA format for Jetson Inference
    frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    frame = cv2.cvtColor(frame, cv2.COLOR_RGB2RGBA)

    # Converting to jetson capsule for optimized performance
    cuda_frame = jetson.utils.cudaFromNumpy(frame)

    # Running the inference model on the capsule
    net.Process(cuda_frame, width, height)
    net.SetOverlayAlpha(50.0)
    net.Overlay(img_overlay, width, height, filter_mode = 'point')

    # Converting the output to numpy array
    out_frame = jetson.utils.cudaToNumpy(img_overlay, width, height, 4)
    out_frame = out_frame.astype(np.uint8)

    cv2.imshow('Output', out_frame)
    key = cv2.waitKey(1) & 0xFF

    if key == ord('q'):
        break

vs.release()
cv2.destroyAllWindows()

Why is the segmentation output coming as huge blocks?

Glad you were able to get it displaying - try changing to filter_mode='linear' in your call to net.Overlay()

1 Like

@dusty_nv, I really appreciate your help. There’s a slight improvement in the segmentation output. Is there anything I can improve to make it look better, apart from using a higher resolution model?

Also, the colors given in the model definition are not showing up exactly in the image. For eg, car was supposed to be segmented in blue it’s showing up in red color, sky was supposed to be blue but it’s yellow in the picture. Any idea how to resolve this?

Hi @satyaakhilreddyb, using higher-res model would be the primary way, because it will also have a high-res segmentation grid (which in turn, produces the overlay).

When the segNet model is loaded, it will print out the class names and then the class colors that it loads. Can you confirm that it is loading those correctly?

1 Like

@dusty_nv, hope you has a good weekend!! Yes, I looked and confirmed the colors and classes before posting this issue on Friday. Am I missing any point while converting the image from capsule to numpy? Like RGBA to RGB or minute stuff like that?

Ah, ok - so if you are converting back to numpy/cv2, you would need to convert from RGB->BGR (because OpenCV expects BGR). I think that would explain why the sky is red when it should be blue (because OpenCV thinks the blue channel is actually the red channel).

Can you try adding something like this?

out_frame = cv2.cvtColor(out_frame, cv2.COLOR_RGB2BGR)

If that doesn’t fix it, I would add a debug call to jetson.utils.saveImageRGBA() after your net.Overlay() call to make sure that it looks correct on-disk.

@dusty_nv Changing from RGB to BGR worked!! I really appreciate your help with this issue… Thank you…

No problem, glad you got it working - FWIW, in my dev branch of jetson-inference, I have added support for video playback and colorspace conversion with CUDA. When released, it could eliminate the extra processing and hoops you had to jump through using OpenCV. Keep an eye when it is merged into master with documentation.

That would be really helpful!! Will keep a lookout… Thanks @dusty_nv