Remove distortion by deepstream-app

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hi all,

I use Jetson Xavier platform.

Camera : AR0231 (120 degree)

OS : JatPack 4.4

TensorRT 7.1.3

CUDA 10.2

I want to remove the distortion of AR0231 (120 degree) when use deepstream-app(v5.0).

Could you give me the suggestion?

Best regards

-Jason

3

Hi,
The case is not supported in default deepstream-app. You would need to refer to

/opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream-dewarper-test

Check if you can apply your source to the sample and get expected result, and then apply it to deepstream-app.

5

Hi DaneDLL,

I following your suggestion.
I tried to use deepstream-dewarper-test.
But I got error when run deepstream-dewarper-test.

<>
[property]
#dewarp-dump-frames=10
#output-width=3680
output-width=1280
#output-height=2428
output-height=720
num-batch-buffers=1

#########################################

Note - Max 4 surfaces are supported

#########################################

[surface0]

1=PushBroom, 2=VertRadCyl, 3=PerspectivePerspective

projection-type=3
#width=3680
width=1280
#height=2428
height=720
#focal-length=1721.592
#src-fov=180
src-fov=120
#top-angle=30
#bottom-angle=-30
#distortion=-0.09787;0.00574;-0.00013;0;0
distortion=-0.299505;0.066764;-0.001612;0.000428;0.000000
#src-x0=1861.88
src-x0=640
#src-y0=1216.04
src-y0=360

0=cudaAddressModeClamp, 1=cudaAddressModeBorder

#cuda-address-mode=0

<>
nvidia@nvidia-desktop:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-dewarper-test$ ./deepstream-dewarper-app file:///home/nvidia/deepstream-distortion/fov120_distortion.mp4 0
Now playing: file:///home/nvidia/deepstream-distortion/fov120_distortion.mp4,

Using winsys: x11
Decodebin child added: source
Decodebin child added: decodebin0
Running

Decodebin child added: matroskademux0
Decodebin child added: multiqueue0
Decodebin child added: h264parse0
Decodebin child added: capsfilter1
Decodebin child added: nvv4l2decoder0
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
In cb_newpad
SpotCSV: Camera entry NOT found. CameraId: 0
AisleCSV: Camera entry NOT found. CameraId: 0
gst_nvdewarper_generate_output: 847 NvBufSurfTransform failed
CUDA runtime error 37 at line 909 in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvdewarper/gstnvdewarper.cpp
ERROR from element stream-muxer: Input buffer number of surfaces (0) must be equal to mux->num_surfaces_per_frame (4)
Set nvstreammux property num-surfaces-per-frame appropriately

Error details: /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvmultistream/gstnvstreammux.c(354): gst_nvstreammux_chain (): /GstPipeline:dewarper-app-pipeline/GstNvStreamMux:stream-muxer
Returned, stopping playback
Deleting pipeline

Could you give me suggestion?

Best regards

-Jason

6

Hi DaneDLL,

Sorry, I am not familiar with typography.
Some fonts in my last email are incorrectly typeset.

Best regards

-Jason

1 Like
8

Hi,
The sample command is

$ ./deepstream-dewarper-app file:///opt/nvidia/deepstream/deepstream-5.1/samples/streams/sample_cam6.mp4 6

You can run

./deepstream-dewarper-app file:///home/nvidia/deepstream-distortion/fov120_distortion.mp4 1

And tune the parameters for your use-case.

9

Hi DaneDLL,

Thank you about your reply.
It work about your last mail command.
But deepstream-dewarper-app is for 360 degree.
My camera is 120 degree.
The deepstream-dewarper-app don’t match my request.

I refer

https://forums.developer.nvidia.com/t/rtsp-server-with-gpu-accelerating-pushing-opencv-mats/149105

https://forums.developer.nvidia.com/t/how-to-perform-fish-eye-lens-distortion-correction-in-gstreamer-pipeline-hfov-150/82808/24

I wrote two programs.
Following are my test step.

Step 1:
build test_v2
g++ test_v2.cpp -o test_v2 pkg-config --cflags --libs opencv4 gstreamer-1.0 gstreamer-rtsp-server-1.0

build rtspServer_v2
gcc rtspServer_v2.c -o rstpServer_v2 $(pkg-config --cflags --libs gstreamer-1.0 gstreamer-rtsp-server-1.0)

Step 2:
run test_v2

2.1 read mp4 file (fov120_distortion_1min.mp4)
2.2 remove distortion
2.3 send data to upd port 5000

Step 3:
run rtspServer_v2

3.1 read data from udp port 5000
3.2 send data out with rtsp server

Step 4:
view rtsp data by smplayer
smplayer rtsp://127.0.0.1:8554/test

It work when use smplayer.

Step 5:
deepstream-app read data from RTSP server.

./deepstream-app -c deepstream_app_config_yoloV3_camera2_test.txt

It run fail.
Following are the log.

nvidia@nvidia-desktop:/opt/nvidia/deepstream/deepstream-5.0/sources/objectDetector_Yolo$ ./deepstream-app -c deepstream_app_config_yoloV3_camera2_test.txt
Unknown or legacy key specified ‘is-classifier’ for group [property]
Warn: ‘threshold’ parameter has been deprecated. Use ‘pre-cluster-threshold’ instead.

Using winsys: x11
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_mot_klt.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
gstnvtracker: Past frame output is OFF
Deserialize yoloLayerV3 plugin: yolo_83
Deserialize yoloLayerV3 plugin: yolo_95
Deserialize yoloLayerV3 plugin: yolo_107
0:00:03.330646253 20059 0x5576c3fb50 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1701> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-5.0/sources/objectDetector_Yolo/model_b1_gpu0_int8.engine
INFO: [Implicit Engine Info]: layers num: 4
0 INPUT kFLOAT data 3x608x608
1 OUTPUT kFLOAT yolo_83 255x19x19
2 OUTPUT kFLOAT yolo_95 255x38x38
3 OUTPUT kFLOAT yolo_107 255x76x76

0:00:03.330935547 20059 0x5576c3fb50 INFO nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1805> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-5.0/sources/objectDetector_Yolo/model_b1_gpu0_int8.engine
0:00:03.335012126 20059 0x5576c3fb50 INFO nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-5.0/sources/objectDetector_Yolo/config_infer_primary_yoloV3.txt sucessfully

Runtime commands:
h: Print this help
q: Quit

p: Pause
r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
To go back to the tiled display, right-click anywhere on the window.

** INFO: <bus_callback:181>: Pipeline ready

PERF: FPS 0 (Avg)
PERF: 0.00 (0.00)
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
NVMEDIA: NVMEDIABufferProcessing: 1504: NvMediaParserParse Unsupported Codec
[NvMediaParserParse:357] Video parser parse failed: 0**PERF: 0.00 (0.00)

Question:
The smplayer run fine.
But deepstream-app run fail.
Could you give me suggestion?

Best regards

-Jason

rtspServer_v2.c (1.9 KB)
test_v2.cpp (4.0 KB)
deepstream_app_config_yoloV3_camera2_test.txt (3.7 KB)

10

Hi,
By default the parameters are for 360 degree. If you cannot adjust it to have a acceptable result for 120 degree, you would need to check if there is 3rdparty functions which can be used. There seems to have certain functions in OpenCV. This would need other users to share experience.

11

Hi DaneLLL,

Thank you about your suggestion.
Because “NvMediaParserParse Unsupported Codec” is the main question of my last mail.
I think, I better create new title to discuss the question.

Best regards

-Jason