Crop in nvdewarper plugin using gst pipeline

[dimitar.yankov]

Hi,
is it possible to zoom out a little bit in nvdewarper plugin?
When undistort via OpenCV we can see black bars on the image (top & bottom), but when we use nvdewarper plugin looks like the image is cropped.
So my question is: How to prevent that and is it possible at all? We’d like to see as much as possible.

Please take a look in the attached images:

Camera is e-CAM21_CUNX from Econ Systems Ltd link

• Focal Length (mm) - 2.8mm
• Aperture - 1.2
• FOV - 132(D), 113(H), 61(V)

This is our Nvdewarp Plugin Configuration:

[property]
output-width=1920
output-height=1080

[surface0]
width=1920
height=1080

cuda-address-mode=1
projection-type=3
surface-index=0
src-fov=61
distortion=-0.34960287,0.13656695,-0.00045631,-0.00074422,-0.02579545;
pitch=0
yaw=0
roll=0
focal-length=1030.384

Our gst pipeline is:

    gst-launch-1.0 nvarguscamerasrc bufapi-version=1 sensor-mode=1 timeout=30 ! 'video/x-raw(memory:NVMM)', width=1920, height=1080, format=NV12, framerate=30/1 !  nvvideoconvert ! nvdewarper config-file=nvdewarp_config.txt source-id=0 ! nvvideoconvert ! nvv4l2h265enc bitrate=12240000 bufapi-version=1 ! h265parse ! matroskamux ! filesink location=/tmp/test.mkv

• Hardware Platform (Jetson NX)
• DeepStream Version (5.0)
• JetPack Version (4.4)
• TensorRT Version (7.1.3)

[DaneLLL]

Hi,
Could you share a method for doing comparison between the two cases? Maybe like:

1. Encode the source into a file // please attach test.mkv

$ gst-launch-1.0 nvarguscamerasrc num-buffers=150 bufapi-version=1 sensor-mode=1 timeout=30 ! 'video/x-raw(memory:NVMM)', width=1920, height=1080, format=NV12, framerate=30/1 ! nvvideoconvert ! nvv4l2h265enc bitrate=12240000 bufapi-version=1 ! h265parse ! matroskamux ! filesink location=/tmp/test.mkv

2. Run with nvdewarper:

$ gst-launch-1.0 uridecodebin uri=file:///tmp/test.mkv !  nvvideoconvert ! nvdewarper config-file=nvdewarp_config.txt source-id=0 ! nvvideoconvert ! nvv4l2h265enc bitrate=12240000 bufapi-version=1 ! h265parse ! matroskamux ! filesink location=/tmp/test_nv.mkv

3. Run with OpenCV // please share steps for it

So that we can do local replication and investigation further.

[dimitar.yankov]

Hi,
Thank you for the answer!

• This is the result video for point 1:
  test.mkv (6.1 MB)
• This is the result video for point 2:
  test_nv.mkv (7.3 MB)
  • We use the following config file for this video for nvdewarper plugin:
    nvdwarp_config.txt (289 Bytes)
• This is the result video for point 3:
  test_cv.mkv (4.8 MB)

We use common way via OpenCV to undistort the video and the camera matrix & distortion coeffs are calculated with OpenCV function calibrateCamera during calibration:

mtx = np.array([
    [513.4724301735596, 0.0, 493.63116061059867],
    [0, 510.06875366794804,  262.7858053697421],
    [0.0, 0.0, 1.0]]
)
dist = np.array([
    -0.33595095,
    0.11662713,
    0.00143557,
    -0.00251063,
    -0.01901783
])

 h, w = frame.shape[:2]
 new_camera_mtx, roi = cv2.getOptimalNewCameraMatrix(mtx, dist, (w, h), 0.5, (w, h))
 dst = cv2.undistort(frame, mtx, dist, None, new_camera_mtx)
 dst = cv2.resize(dst, (w, h), fx=1, fy=1, interpolation=cv2.INTER_CUBIC)

[DaneLLL]

Hi,
Please try to set width, height larger than original resolution. The test.mkv is 960x540 and please try

[surface0]
#dewarped surface parameters
width=1280
height=720

[dimitar.yankov]

Hi,
We have tried to increase the output resolution to 1280x720 and we are able to zoom out a little bit but then the image is not undistorted correctly on the sides. Maybe we have wrong configuration values for nvdewarp plugin because with opencv we can undistort the image, e.g calculation of the focal length we use the following formula:

convert degrees of vertical FOV to radians = 61deg = 1.06465084
pixels  per radian  = 540px (vertical resolution) / 1.06465084rad (vertical fov in radians)

test_nv_extended_resolution.mkv (7.3 MB) - result video with increased resolution

[DaneLLL]

Hi,
Please try to adjust:

distortion=-0.33595095;0.11662713;0.00143557;-0.00251063;-0.01901783;

1. Set last two coefficients to 0
2. Try to set third coefficient half or double, and check the effect. If it has effect, please tweak it until the lines straighten out. If it becomes a little wiggly, you may need to tweak the second coefficient too.

[dimitar.yankov]

Hi,
We have tried your suggestion to adjust coefficients but without any success.

1. Set last two coefficients to 0 - test_nv_last_two_coeffs_0.mkv (7.2 MB)
2. Try to set third coefficient half or double, and check the effect:

• Last two coefficients set to 0 & doubled third coefficient test_nv_last_two_coeffs_0_third_doubled.mkv (7.1 MB)
• Last two coefficients set to 0 & half of third coefficient - test_nv_last_two_coeffs_0_and_third_halfed.mkv (7.2 MB)
  There is no big difference among the three videos and they are all not undistorted correctly on the sides.

Do we know the meaning of each of these coefficients, because probably there is a difference in implementations between nvdewarp plugin and opencv undistortion?
Thank you in advance!

[DaneLLL]

Hi,
The algorithm should be similar to OpenCV. For perspective sources, these are 3 radial(k0, k1, k2):

s = f * r * (1 + k0 * r^2 + k1 * r^4 + k2 * r^6)

plus 2 tangential distortion coefficients(k3, k4).

Please also try to adjust first and second coefficients(k0, k1). See if tuning the coefficients helps.

[dimitar.yankov]

Hi,
When we did some changes on all distortion coefficients, we have something similar to OpenCV results (it is not perfect, good combination of undistortion and visibility on the sides).

Original Image:

Undistorted Image:

Original distortion coefficients from OpenCV:

distortion=-0.33595095;0.11662713;0.00143557;-0.00251063;-0.01901783;

Changed distortion coefficients:

distortion=-0.230095;0.0435713;-0.0067;-0.012;0;