How to use opencv in deepstream-app sample

[Leozih]

Please provide complete information as applicable to your setup.

**• Hardware Platform (Jetson / GPU)**Jetson
• DeepStream Version6.1.1
**• JetPack Version (valid for Jetson only)**5.1.1
**• Issue Type( questions, new requirements, bugs)**questions
Hi,guys,I am trying to add extra function in deepstream-app sample to realize edge detection.To realize that,I need to use opencv.A suggestion I get is to add the function in deepstream_app.c/gie_processing_done_buf_prob function,and I add something like:

  NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta(buf);
  for (NvDsMetaList * l_frame = batch_meta->frame_meta_list; l_frame != NULL;
      l_frame = l_frame->next) {
      //NvDsFrameMeta *frame_meta = gst_meta_get_nvds_frame_meta(batch_meta, buf);
      NvDsFrameMeta *frame_meta = l_frame->data;
      for (NvDsMetaList * l_obj = frame_meta->obj_meta_list; l_obj != NULL;
        l_obj = l_obj->next) {
        NvDsObjectMeta *obj = (NvDsObjectMeta *) l_obj->data;
        cv::Rect bbox(obj->rect_params.left, obj->rect_params.top,
                      obj->rect_params.width, obj->rect_params.height);
        GstMapInfo map;
        gst_buffer_map(buf, &map, GST_MAP_READ);
        cv::Mat frame(cv::Size(frame_meta->source_frame_width, frame_meta->source_frame_height),
                      CV_8UC3, (uchar*)map.data, cv::Mat::AUTO_STEP);
        gst_buffer_unmap(buf, &map);
        cv::Mat object_img = frame(bbox);
        float angle;
        Ellipse_feature_extraction2(object_img,angle);

Ellipse_feature_extraction2 is my costum edge detection function to get an angle from each detected object defined in the deepstream_app_config_parser_yaml.cpp because it involves some opencv functions.

My opinion is to define the codes involving opencv in the deepstream_app_config_parser_yaml.cpp and give it a statement in deepstream_app.h.Then it maybe can be used in the deepstream_app.c just like the

gboolean
parse_config_file_yaml (NvDsConfig *config, gchar *cfg_file_path)

defined in the original deepstream_app_config_parser_yaml.cpp.
So my question is:

1. Is my opinion practical?
2. If so,are there other changes I need to make?

[junshengy]

@Leozih

Is my opinion practical?

This method should work

@Leozih

If so,are there other changes I need to make?

As you know, gst_buffer contains nvbufsurface, which is a handle to the gpu memory, so if you want to use opencv for processing, you need to copy the memory to the cpu

Refer this sample code.

Deepstream sample code snippet

Simple description for your code a. Creating nv12 opencv mat from NvMM memory, b. Work on opencv nv12 mat, do conversion to other opencv RGBA mat, rgba mat. c. Create another scratch RGBA NvBufSurface and do opencv conversion to rotate RGBA in rotate mat. d. Transform rotated RGBA mat to NV12 memory in original input surface e. So now NvMM NV12 memory from original buffer is also rotated. Which version are you based on a. DS 5.0 DP Which platform have you verified for your code a. T4 --…

[Leozih]

Thank you for your apply,but there must be some changes in makefile to generate new deepstream-app application.What should I do?

[junshengy]

@Leozih

but there must be some changes in makefile to generate new deepstream-app application.

Are you having any problems? Is it that the header file of opencv cannot be found?

CFLAGS+=  -I /usr/include/opencv4
PKGS+= opencv4

CFLAGS+=$(shell pkg-config --cflags $(PKGS))
LIBS+=$(shell pkg-config --libs $(PKGS))

[Leozih]

I have tried to compile the code,the statement in deepstream_app.h is :

#include <opencv2/opencv.hpp>
using namespace cv
void Ellipse_feature_extraction2(const Mat& img,float& angle);

But it seems that the deepstream_app.h file originates from deepstream_app.c so it can’t cope with the opencv.hpp

[junshengy]

@junshengy

CFLAGS+=  -I /usr/include/opencv4
PKGS+= opencv4

CFLAGS+=$(shell pkg-config --cflags $(PKGS))
LIBS+=$(shell pkg-config --libs $(PKGS))

Refer to my answer above, you need to use pkg-config to manage your dependencies

[Leozih]

As you say,the makefile:

CUDA_VER?=11.4
ifeq ($(CUDA_VER),)
  $(error "CUDA_VER is not set")
endif

APP:= deepstream-app

TARGET_DEVICE = $(shell gcc -dumpmachine | cut -f1 -d -)

NVDS_VERSION:=6.1

LIB_INSTALL_DIR?=/opt/nvidia/deepstream/deepstream-$(NVDS_VERSION)/lib/
APP_INSTALL_DIR?=/opt/nvidia/deepstream/deepstream-$(NVDS_VERSION)/bin/

ifeq ($(TARGET_DEVICE),aarch64)
  CFLAGS:= -DPLATFORM_TEGRA
endif

SRCS:= $(wildcard *.c) $(wildcard *.cpp)
SRCS+= $(wildcard ../../apps-common/src/*.c)
SRCS+= $(wildcard ../../apps-common/src/deepstream-yaml/*.cpp)

INCS:= $(wildcard *.h)

PKGS:= gstreamer-1.0 gstreamer-video-1.0 x11 json-glib-1.0

OBJS:= $(SRCS:.c=.o)
OBJS:= $(OBJS:.cpp=.o)

CFLAGS+= -I./ -I../../apps-common/includes \
		 -I../../../includes -DDS_VERSION_MINOR=1 -DDS_VERSION_MAJOR=5 \
		 -I /usr/local/cuda-$(CUDA_VER)/include\
		 -I /usr/include/opencv4
PKGS+=opencv4

LIBS:= -L/usr/local/cuda-$(CUDA_VER)/lib64/ -lcudart

LIBS+= -L$(LIB_INSTALL_DIR) -lnvdsgst_meta -lnvds_meta -lnvdsgst_helper \
	  -lnvdsgst_smartrecord -lnvds_utils -lnvds_msgbroker -lm -lyaml-cpp \
    -lcuda -lgstrtspserver-1.0 -ldl -Wl,-rpath,$(LIB_INSTALL_DIR)

CFLAGS+= $(shell pkg-config --cflags $(PKGS))

LIBS+= $(shell pkg-config --libs $(PKGS))

all: $(APP)

%.o: %.c $(INCS) Makefile
	$(CC) -c -o $@ $(CFLAGS) $<

%.o: %.cpp $(INCS) Makefile
	$(CXX) -c -o $@ $(CFLAGS) $<

$(APP): $(OBJS) Makefile
	$(CXX) -o $(APP) $(OBJS) $(LIBS)

install: $(APP)
	cp -rv $(APP) $(APP_INSTALL_DIR)

clean:
	rm -rf $(OBJS) $(APP)

By the way ,the error is like:
/usr/include/opencv4/opencv2/core.hpp:49:4: error: #error core.hpp header must be compiled as C++
So maybe I should include something like opencv2/imgproc/imgproc.h ?

[junshengy]

Rename the code to xxxx.cpp

[Leozih]

Oh do you mean that I should rename the deepstream_app.c to deepstream_app.cpp?
The structure is:

So maybe I can’t just change it to cpp file easily.
The opinion I want to state my custom function in deepstream_app.h is because I see the function in deepstream_app_config_parser_yaml.cpp stated in the deepstream_app.h so that the function can be used in deepstream_app_main.c.
So I imitate the opinion and define the void Ellipse_feature_extraction2 in deepstream_app_config_parser_yaml.cpp and then give it a statement in deepstream_app.h.

The statement I add is :

#include <opencv2/opencv.hpp>
using namespace cv

void Ellipse_feature_extraction2(const Mat& img,float& angle);

Just like I mentioned before.But here

const Mat

infers to cv::Mat which do not belongs to C files,so maybe that’s why the error occurs.

In that case,I say that:
So maybe I should include something like **opencv2/imgproc/imgproc.h**
Because opencv2/imgproc/imgproc.h includes the C API of opencv,do you think I should do this?

[junshengy]

@Leozih

opencv2/imgproc/imgproc.h

If want to use c language, please include imgproc_c.h in your source code .imgproc.hpp is just used for cpp.
You can google get more information

[yingliu]

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks