Visual error with Segnet output using OpenCV

[sarobando]

Hello,
I’m working on a program running Segnet which reads a webcam video stream using OpenCV, processes the image with Segnet, converts the output back to OpenCV, and displays an output. I’m getting a visual error where only a tiny portion of the Segnet overlay displays on the image. Below is an example, with a few small lines of the Segnet overlay being displayed, mainly in the top left:

I’m using a code that differs significantly from the example segnet.py, as segnet.py has trouble reading in my video frames (something to do with size mismatch in GStreamer). I have tested segnet.py with single images, and produced the expected result, so I think it must be an issue with my code. The processing loop is as follows:

 while camera.isOpened():
    _,image = camera.read()

    img = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    img = cv2.cvtColor(img, cv2.COLOR_RGB2RGBA).astype(np.float32)

    img_input = jetson.utils.cudaFromNumpy(img)

    net.Process(img_input)

    net.Overlay(img_input)

    img_cv2 = jetson.utils.cudaToNumpy(img_input)

    img_cv2 = cv2.cvtColor(img_cv2, cv2.COLOR_RGBA2BGR).astype(np.float32)

    cv2.imshow('OpenCV Output', img_cv2/255)
    if cv2.waitKey(1) & 0xFF == ord('q'):
            break # CTRL + Q to stop

What could be causing this issue?

[DaneLLL]

Hi,
We would suggest run the application and dump image, img, img_input, img_cv2. So that you can inspect the frame data through YUV viewer.

image is read from camera and it looks to be in BGR. Should be able to call cv2.imshow(‘OpenCV Output’, image) and check if it good first.

[sarobando]

@sarobando

:

Surprisingly, the output looks fine when I display a set of images (I displayed image, img, img_np (new), and img_cv2 as below:

    _,image = camera.read()
    cv2.imshow('image',image)

    img = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    img = cv2.cvtColor(img, cv2.COLOR_RGB2RGBA).astype(np.float32)
    cv2.imshow('img',img)

    img_input = jetson.utils.cudaFromNumpy(img)

    net.Process(img_input)

    net.Overlay(img_input)

    img_np = jetson.utils.cudaToNumpy(img_input)
    cv2.imshow('img_np',img_np)

    img_np = cv2.cvtColor(img_np, cv2.COLOR_RGBA2BGR).astype(np.float32)
    img_cv2 = img_np.copy()

    cv2.imshow('OpenCV Output', img_cv2/255)

Bizarrely, it seems that just having all the images shown with cv2.imshow() makes the output appear as desired, with the full overlay applied. Actually, as long as I also imshow() img_np, the correct output is displayed by img_cv2.

I thought it might be an issue with values getting overwritten too fast for the program to display the original (so the img_cv2 values were getting rewritten with regular RGB frames for some reason), but I didn’t see the correct outcome when doing a time.sleep() to delay the loop.

Any insight you could give would be much appreciated.

[dusty_nv]

@sarobando

I thought it might be an issue with values getting overwritten too fast for the program to display the original

Hi @sarobando, can you try adding a call to jetson.utils.cudaDeviceSynchronize() after the net.Overlay() call?

@sarobando

segnet.py has trouble reading in my video frames (something to do with size mismatch in GStreamer)

Can you try re-building jetson-inference with ENABLE_NVMM=off flag like so?

cd /path/to/your/jetson-inference/build
cmake -DENABLE_NVMM=off ../
make
sudo make install