Cylindrical Projection OpenCV

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
,
1

Hi, I’m currently working on a project trying to stitch 2 video inputs together to form a wide view covering 180 degrees. An alternative to stitching, which I assume would take a lot of processing power for real time stitching, could be cylindrical projection of the images. Currently, I have two cameras set up next to each other, and concatinating them into 1 window using OpenCV.

I’m showing 2 camera inputs using the code below. It runs fine, but straight lines that transition from one image to the next become ‘bent’. I was wondering if some sort of projection would make the two images look nicer. Best example I could fine:


Any ideas how to achieve the above? Any help appriciated!

import cv2
import numpy as np



cap1 = cv2.VideoCapture("v4l2src device=/dev/video0 ! video/x-raw, format=BGRx, width=2064, height=1544 ! videoconvert ! video/x-raw, format=BGR ! appsink")
cap2 = cv2.VideoCapture("v4l2src device=/dev/video1 ! video/x-raw, format=BGRx, width=2064, height=1544 ! videoconvert ! video/x-raw, format=BGR ! appsink")


while True:
    _, frame1 = cap1.read()
    _, frame2 = cap2.read()

    frame1 = cv2.resize(frame1, (720,520))
    frame2 = cv2.resize(frame2, (720,520))

    
    sbs = cv2.hconcat([frame2, frame1])
    cv2.imshow('sbs', sbs)

    if cv2.waitKey(1) == ord('q'):
        break

sbs.release() 
cv2.destroyAllWindows()
3

Hi,
We have VPI that supports computer vision fuctions and is executed on hardware engines. Please check the document and see if it can be applied to your use-case:
VPI - Vision Programming Interface: Main Page

OpenCV mainly uses CPU and if you can switch to VPI, can shift the loading from CPU to hardware engines like VIC, GPU.

4

Thanks Dane, I’ll look into VPI. As of now, I’m not too concerned about latency. The problem is that I’ve no idea how to go about image projection in general. But down the line, I will need to put the load on GPU for sure, so I will have a look!

5

I only just realized that VPI might not be a solution at this moment because I’m working with JetPack 4.6?

6

Hi,
It is VPI 1.1 in Jetpack 4.6. Please refer to
VPI - Vision Programming Interface: Main Page

7

Hm any idea why I’d get this error in VS code?

Python 3.6.9
JetPack version: 4.6-b197

import vpi
import numpy as np

warp = vpi.WarpMap(vpi.WarpGrid(input.size))

AttributeError: ‘module’ object has no attribute ‘WarpMap’

8

Hi,

Could you try if you can run our fisheye sample first?
It calls the vpi.WarpMap as well and it should work on JetPack 4.6.

/opt/nvidia/vpi1/samples/11-fisheye/

Thanks.

9

Hi Aasta, thanks! Do I just run this from the command line? In VS Code, I’m getting “too few arguments”. Seems like I need some checkerboard images too?

EDIT:

So that seemed to work.

Screenshot from 2022-07-29 12-27-16
Screenshot from 2022-07-29 12-27-161005×185 87.1 KB

10

Hi,

Could you also try it with VS?
In the sample, it calls vpiWarpMap as well.

Thanks.

11

This is probably a stupid question but I’m getting “too few arguments” when running in VS Code. How do I test this with the needed arguments like I did with the above example from the terminal?

12

There is also a terminal within VS code, and you may have noticed that when you hit the run button it just pastes the command to run the file into this terminal. so you could just copy and paste (ctrl + shift + c/v) that command with your arguments added to it.
(make sure the vscode terminal is in the right directory also).

There are also other ways like a launch.json file, but i think the terminal is the easiest way.

13

that seemed to work. But isn’t that just like running it in a linux terminal? I mean, would this properly test if it works in VS Code?

command:
python3 main.py -c 10,7 -s 22 …/assets/fisheye/*.jpg

EDIT

Anyway, problem seems fixed as this code seem to run fine.

import vpi
import numpy as np
import cv2
import PIL
from PIL import Image

img = cv2.imread(‘input.jpeg’)

vpi_image = vpi.asimage(np.asarray(img))

grid = vpi.WarpGrid((2064,1544))

sensorWidth = 7.12
focallength = 3.5

f = focallength * (2064 / sensorWidth)

K = [[f, 0, 2064/2],
[0, f, 1544/2]]

X = np.eye(3,4)

warp = vpi.WarpMap.fisheye_correction(grid, K=K, X=X,
mapping=vpi.FisheyeMapping.EQUIDISTANT,
coeffs=[-0.01, 0.2])

with vpi.Backend.CUDA:
output = vpi_image.remap(warp, interp=vpi.Interp.CATMULL_ROM, border=vpi.Border.ZERO)

with output.rlock():
output = Image.fromarray(output.cpu()).save(‘output.jpeg’)

pil_image = PIL.Image.open(‘output.jpeg’).convert(‘RGB’)
cv2_image = np.array(pil_image)
cv2_image = cv2_image[:, :, ::-1].copy()

cv2_image = cv2.resize(cv2_image, (920,590))
img = cv2.resize(img, (920, 590))

sbs = cv2.hconcat([img, cv2_image])
cv2.imshow(“sbs”, sbs)
cv2.waitKey(0)

Going back to my original question, I wonder if any of the VPI methods are useful for projecting an image to a cylinder. Maybe I will play around with the parameters above and see. So far this is only running on an image but I’d like it to run in a loop for video input.

Thanks for the replies so far!

14

Follow up:

Can the above code be run on a video in a loop?

Any help appriciated.

15

Anyone have ideers to why “cv2.VideoCapture(0)” would cause "Segmentation fault (core dumped) error?

The following code runs fine, until I uncomment ‘cap = cv2.VideoCapture(0)’.

import vpi
import cv2
import numpy as np
from PIL import Image

# cap = cv2.VideoCapture(0)


img = cv2.imread('input.jpeg')
vpi_image = vpi.asimage(np.asarray(img))

xform = [[ 0.5386, 0.1419, -74    ],
         [ -0.4399, 0.8662, 291.5 ],
         [ -0.0005, 0.0003, 1     ]]

with vpi.Backend.CUDA:
    output = vpi_image.perspwarp(xform)

with output.rlock():
    output = Image.fromarray(output.cpu()).save('output.jpeg')

Thanks!