The data seems to be compressed and the display in ros has been completely changed

[autodrive2022]

• deepstream-app version 6.1.0
• DeepStreamSDK 6.1.0
• CUDA Driver Version: 11.4
• CUDA Runtime Version: 11.0
• TensorRT Version: 8.2
• cuDNN Version: 8.4
• libNVWarp360 Version: 2.0.1d3
• on A6000
  1, I added ros publishing to the ds-example plugin to publish the cv::mat video data in it, but the data seems to be compressed and the display in ros has been completely changed.
  2, My test: adding a temporary image to the publish is fine, but to publish the mat from the ds-example is problematic.
  3, Here is the relevant code and the effect image in the gst_dsexample_transform_ip function.

#ifdef WITH_OPENCV
      cv::Mat in_mat;
      if (dsexample->blur_objects) {
        /* Map the buffer so that it can be accessed by CPU */
        if (surface->surfaceList[frame_meta->batch_id].mappedAddr.addr[0] == NULL){
          if (NvBufSurfaceMap (surface, frame_meta->batch_id, 0, NVBUF_MAP_READ_WRITE) != 0){
            GST_ELEMENT_ERROR (dsexample, STREAM, FAILED,
                ("%s:buffer map to be accessed by CPU failed", __func__), (NULL));
            return GST_FLOW_ERROR;
          }
        }
        /* Cache the mapped data for CPU access */
        if(dsexample->inter_buf->memType == NVBUF_MEM_SURFACE_ARRAY)
          NvBufSurfaceSyncForCpu (surface, frame_meta->batch_id, 0);
        in_mat =
            cv::Mat (surface->surfaceList[frame_meta->batch_id].planeParams.height[0],
            surface->surfaceList[frame_meta->batch_id].planeParams.width[0], CV_8UC4,
            surface->surfaceList[frame_meta->batch_id].mappedAddr.addr[0],
            surface->surfaceList[frame_meta->batch_id].planeParams.pitch[0]);
      }
#endif
      for (l_obj = frame_meta->obj_meta_list; l_obj != NULL;
          l_obj = l_obj->next)
      {
        obj_meta = (NvDsObjectMeta *) (l_obj->data);

        if (dsexample->blur_objects) {
          /* gaussian blur the detected objects using opencv */
#ifdef WITH_OPENCV
          /* if (blur_objects (dsexample, frame_meta->batch_id,
            &obj_meta->rect_params, in_mat) != GST_FLOW_OK) {         
            GST_ELEMENT_ERROR (dsexample, STREAM, FAILED,
            ("blurring the object failed"), (NULL));
            if (NvBufSurfaceUnMap (surface, frame_meta->batch_id, 0)){
              GST_ELEMENT_ERROR (dsexample, STREAM, FAILED,
                ("%s:buffer unmap to be accessed by CPU failed", __func__), (NULL));
            }
            return GST_FLOW_ERROR;
             */
           g_print  ("%s %d\n","frame_meta:id=",frame_meta->source_id);
           
           if(frame_meta->source_id==0){
           if(blur_objects (dsexample, frame_meta->batch_id,
            &obj_meta->rect_params, in_mat) == GST_FLOW_OK){
               //cv::Mat img=cv::imread("/home/fast-cnn-train/CODES/车辆数据/车辆数据集/data8/test/images/bigcar_002.jpg");
               sensor_msgs::ImagePtr msg = cv_bridge::CvImage(std_msgs::Header(), "bgr8", in_mat).toImageMsg(); //cvmat格式转ros ImagePtr格式
               pub.publish(msg);//ros发布
            }
            else{
             GST_ELEMENT_ERROR (dsexample, STREAM, FAILED,
            ("blurring the object failed"), (NULL));
            if (NvBufSurfaceUnMap (surface, frame_meta->batch_id, 0)){
              GST_ELEMENT_ERROR (dsexample, STREAM, FAILED,
                ("%s:buffer unmap to be accessed by CPU failed", __func__), (NULL));
            }
            return GST_FLOW_ERROR;
            } 
          }

[yingliu]

If you simply save the image from “in_mat” instead of publishing to ros, how about the image?

[autodrive2022]

This is the image I saved, and it looks like this.

[autodrive2022]

cv::imwrite("./img_res/"+std::to_string(index_img)+".jpg", *dsexample->cvmat);

I use this code to save the image directly and the saved image is correct.
But sending it to ros is not correct, as you can see I have added the code for opencv to ros image, but still not correct, what is the problem?

sensor_msgs::ImagePtr msg = cv_bridge::CvImage(std_msgs::Header(), "bgr8", in_mat).toImageMsg(); //cvmat to ros ImagePtr
pub.publish(msg);//ros pub

[yingliu]

Maybe you need isolate/reproduce this issue in a simple ROS environment and check with ROS forum(https://answers.ros.org/).

[autodrive2022]

I think it’s this cvmat to rosImage data conversion problem, what do you think?

[fanzh]

from your code, dsexample->cvmat is used to save jpg, in_mat is used to convert to msg , but dsexample->cvmat and in_mat are not the same.
please save in_mat to check.

[fanzh]

Sorry for the late reply, Is this still an DeepStream issue to support? Thanks
what is the whole media pipeline? please check NvBufSurfaceParams’s colorFormat first, then create cv::Mat, it is because colorFormat might be rgba or nv12.

[autodrive2022]

I tried to save the image data is normal, inferred that it was a problem with the data format conversion parameters I sent ros, finally modified it and it worked. Thanks for your help!