Merging dewraper config into deepstream-app

[edit_or]

• Hardware Platform (Jetson / GPU)
Xavier
• DeepStream Version
6.0
• JetPack Version (valid for Jetson only)
4.6
• TensorRT Version
TRT 8.0.1
• NVIDIA GPU Driver Version (valid for GPU only)
10.2

I have tested dewrapper app and it is working.

I like to merge dewraper config into deepstream-app.
My deepstream-app has main config file mrcnn.txt.
mrcnn.txt (7.8 KB)
mrcnn.txt has config_infer_mrcnn.txt.
config_infer_mrcnn.txt (3.9 KB)

Dewraper property [surface0] is added into config_infer_mrcnn.txt.

[surface0]
# 1=PushBroom, 2=VertRadCyl
projection-type=2
surface-index=0
#dewarped surface parameters
width=1920
height=1080
top-angle=90
bottom-angle=-10
pitch=-38
#pitch=0.657064
yaw=10
#yaw=-0012782
roll=-5
#roll=0.036893
focal-length=396.19545827
rot-axes=2
src-fov=180

When the deepstream-app is run with command ./deepstream-app -c ../../../../samples/configs/deepstream-app/mrcnn.txt, dewrapping does not work.
How to merge dewrapping into deepstream-app?

[yuweiw]

You can try to add the [dewarper] tag in your mrcnn.txt and do the correct config file with it.

[edit_or]

So the format is to put
[dewarper] tag in mrcnn.txt
Then put [surface0] and properties below, right?

[yuweiw]

No, you can refer the link below to see which paras can be add.
https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvdewarper.html

[edit_or]

[dewarper] is put inside mrcnn.txt as follows. But dewrapping still doesn’t happen.

[dewarper]
# 1=PushBroom, 2=VertRadCyl
projection-type=2
surface-index=0
#dewarped surface parameters
width=1920
height=1080
top-angle=90
bottom-angle=-10
pitch=-38
#pitch=0.657064
yaw=10
#yaw=-0012782
roll=-5
#roll=0.036893
focal-length=396.19545827
rot-axes=2
src-fov=180

Actual mrcn.txt is as
mrcnn.txt (8.1 KB)

[edit_or]

Any suggestion?

[edit_or]

Added

if (!g_strcmp0 (*group, CONFIG_GROUP_DEWARPER)) {
      parse_err = !parse_dewarper (&config->dewarper_config, cfg_file, cfg_file_path);
    }

into deepstream_app_config_parser.c.
I have some warnings

** WARN: <parse_dewarper:1050>: Unknown key 'projection-type' for group [dewarper]
** WARN: <parse_dewarper:1050>: Unknown key 'surface-index' for group [dewarper]
** WARN: <parse_dewarper:1050>: Unknown key 'width' for group [dewarper]
** WARN: <parse_dewarper:1050>: Unknown key 'height' for group [dewarper]
** WARN: <parse_dewarper:1050>: Unknown key 'top-angle' for group [dewarper]
** WARN: <parse_dewarper:1050>: Unknown key 'bottom-angle' for group [dewarper]
** WARN: <parse_dewarper:1050>: Unknown key 'pitch' for group [dewarper]
** WARN: <parse_dewarper:1050>: Unknown key 'yaw' for group [dewarper]
** WARN: <parse_dewarper:1050>: Unknown key 'roll' for group [dewarper]
** WARN: <parse_dewarper:1050>: Unknown key 'focal-length' for group [dewarper]
** WARN: <parse_dewarper:1050>: Unknown key 'rot-axes' for group [dewarper]
** WARN: <parse_dewarper:1050>: Unknown key 'src-fov' for group [dewarper]

Using winsys: x11 
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-6.0/lib/libnvds_nvmultiobjecttracker.so
gstnvtracker: Batch processing is ON
gstnvtracker: Past frame output is ON
[NvMultiObjectTracker] Initialized
q0:00:03.538130714 14745   0x55a54b5ef0 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1900> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/../../models/maskrcnn/10_8_2022/maskrcnn.etlt_b1_gpu0_fp16.engine
INFO: [Implicit Engine Info]: layers num: 3
0   INPUT  kFLOAT Input           3x576x640       
1   OUTPUT kFLOAT generate_detections 100x6           
2   OUTPUT kFLOAT mask_fcn_logits/BiasAdd 100x26x28x28    

0:00:03.538370473 14745   0x55a54b5ef0 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2004> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/../../models/maskrcnn/10_8_2022/maskrcnn.etlt_b1_gpu0_fp16.engine
0:00:03.549529335 14745   0x55a54b5ef0 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/config_infer_mrcnn.txt sucessfully

dewrapping stil doesn’t work.

[yuweiw]

You can refer the Gst Properties from the user guide that I attached, you only can add the paras below in the config file :

config-file
gpu-id
source-id
num-batch-buffers
nvbuf-memory-type

[edit_or]

Thanks. Let me try.
Yeah may be not using right properties.

[edit_or]

Yes you are right. The config file accepts only gpu-id and config-file.
We can see in deepstream_config_file_parser.c
So my config for dewarper is changed as follows in mrcnn.txt

[dewarper]
enable=1
gpu-id=0
config-file=config_dewarper.txt

But suspect that your code inside deepstream_source_bin.c has something not right.

Before create_dewarper_bin() in deepstream_app.c, config->dewarper_config.enable is 1.

But when checked in deepstream_source_bin.c inside create_multi_source_bin() , configs->dewarper_config.enable becomes 0. NEED TO CHECK

Tried to set configs->dewarper_config.enable = 1;
But got error at create_dewarper_bin.

[edit_or]

Proceed debugging into deepstream_dewarper_bin.c, found out that config->config_file is NULL. Then set manually
config->config_file = "/opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/config_dewarper.txt";

Then in deepstream_dewarper_bin.c

g_object_set (G_OBJECT (bin->nvdewarper), "config-file",
      config->config_file, NULL); 

doesn’t have source _id.

In deepstream_dewarper_test.c, it has source_id

g_object_set (G_OBJECT (nvdewarper),
      "config-file", "./config_dewarper.txt",
      "source-id", source_id,
      NULL);

The actual error is
nvdewarper0: Failed to parse config file /opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/config_dewarper.txt
But it can be parsed in deepstream_dewarper_test.c,
Parsing failed at deepstream-app

Failed at this line

g_object_set (G_OBJECT (bin->nvdewarper),
      "config-file", "/opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/config_dewarper.txt",
      "source-id", 1,
      NULL);

in deepstram_dewarper_bin.c

[edit_or]

Now it works.
deepstream-app is not tested well for dewarper config.
Inside deepstream_config_file_parser.c, can see dewarper config file path is NULL.
So need to set manually.
Then even though configs->dewarper_config.enable is 1 in deepsteam-app.c, when checked in deepstram_source_bin.c, it is 0.
So need to set manually.
Now it works.