Perspective distortion dewrapping

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
Xavier
• DeepStream Version
6.0
• JetPack Version (valid for Jetson only)
4.6
• TensorRT Version
TRT 8.0.1
• NVIDIA GPU Driver Version (valid for GPU only)
10.2
I am testing dewrapping app for fisheye camera perspective undistortion.
The cv2 API was used to calibrate the fisheye camera.

The API produces K and D. D has only four K values.

How do I set K and D into distortion?
CV2 calibration has only k1,k2,k3,k4.

Discussion here shows distortion needs

distortion=-0.3721;0.1023;-0.00203;0;0
`distortion=k_0,;k_1,;k_2;k_3;k_4`
If not provided, all coefficients are set to 0 (no distortion)

Config file distortion needs k0 to k4.
How can I set distortion values?

Used cv2’s outputs k1,k2,k3,k4 to
distortion configuration as
distortion = k1,k2,k3,k4,0
But image is not dewrapped.
dewrap

I guess you are using “NVDS_META_SURFACE_PERSPECTIVE_PERSPECTIVE” mode. This mode supports polynomial distortion correction for a perspective source and not a fisheye distortion correction.

To get the correct distortion coefficients you need to use the following calibration method of opencv :

https://docs.opencv.org/2.4.13.7/doc/tutorials/calib3d/camera_calibration/camera_calibration.html

This will give you focal length(fx, fy), principal point(cx, cy) and 5 distortion coefficients : (k1, k2, p1, p2, k3)
You can set them in the dewarper plugin config file as follows :
focal-length=fx/fy/avg(fx,fy)
src-x0=cx
src-y0=cy
distortion=k1;k2;k3;p1;p2 (please note the reordering here)

Thanks Fiona, if my camera is fisheye lens with 180FPV, which calibration method should I use to run with dewrapper app?

For fisheye, I need the following infos. How do I know top-angle and bottom-angle? pitch, yaw, roll are all 0 for 180FOV fisheye camera?

projection-type=5 
surface-index=0
#dewarped surface parameters
width=3800
height=2900
top-angle=90
bottom-angle=-90
pitch=0
yaw=0
roll=0
focal-length=350
rot-axes=2
src-fov=180
#control=1.0

For 360 camera, I need the following infos. Where can I get pitch, yaw and roll?

pitch=90
yaw=150
roll=0
focal-length=850
rot-axes=3

“PerspectivePerspective” is enough for both cases. The correct sample is in /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-dewarper-test/config_dewarper_perspective.txt

To get the correct parameters needed by config_dewarper_perspective.txt, the openCV calibrateCamera can be used to calculate the parameters.

Camera Calibration and 3D Reconstruction — OpenCV 2.4.13.7 documentation

For theory, the following material may be helpful. OpenCV: Camera Calibration

Understanding Lens Distortion | LearnOpenCV #

Thanks Fiona for reply.
I did calibration using the cv2 api cv2.calibrateCamera.
Tested K and D using cv2’s sample program as follow.

import cv2
import numpy as np

K=np.asarray([[408.60670389,   0.0,         671.36058779],
 [  0.0 ,        393.36314377, 348.39577702],
 [  0.0,          0.0 ,          1.0       ],])


D=np.asarray([[-0.36489552,  0.1945094, -0.06888981,  -0.00040946,  0.00174717]])
cap = cv2.VideoCapture('fisheye.mp4')
if (cap.isOpened()== False):
   print("Error opening video stream or file")

while(cap.isOpened()):
   ret, img = cap.read()
   if ret == True:
      h,  w = img.shape[:2]
      newcameramtx, roi = cv2.getOptimalNewCameraMatrix(K, D, (w,h), 1, (w,h))
      #mapx, mapy = cv2.initUndistortRectifyMap(K, D, None, newcameramtx, (w,h), 5)
      #dst = cv2.remap(img, mapx, mapy, cv2.INTER_LINEAR)
      dst = cv2.undistort(img, K, D, None, newcameramtx)
      # crop the image
      #x, y, w, h = roi
      #dst = dst[y:y+h, x:x+w]
      cv2.imshow('dst',dst)
      cv2.waitKey(1)

The undistorted image using cv2 test program is as follow.
Not perfect undistortion, but still can see the view.

Using deepstream’s dewarper-test app, the config file is as follow. You can see that all parameters are same as in cv2 test program.

# DEALINGS IN THE SOFTWARE.
################################################################################

[property]
#dewarp-dump-frames=10
output-width=1920
output-height=1080
num-batch-buffers=1

#########################################
# Note - Max 4 surfaces are supported
#########################################

[surface0]
# 1=PushBroom, 2=VertRadCyl, 3=PerspectivePerspective
projection-type=3
width=1280
height=720
focal-length=400
#src-fov=180
#top-angle=30
#bottom-angle=-30
distortion=-0.36489552;0.1945094;-0.06888981;-0.00040946;0.00174717
src-x0=639
src-y0=359
# 0=cudaAddressModeClamp, 1=cudaAddressModeBorder
#cuda-address-mode=0
#src-fov=180

The undistorted view is not working.

The original video is https://www.dropbox.com/s/39dn72v53h5l2hb/fisheye.mp4?dl=0

What could be wrong?

Thanks it worked ald

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.