How to crop the image and save

that’s very helpful

Hello ,

Presently I am able to save all the images which are given as input to the system using get_converted_mat() function. I am in need of only those images which have detected objects.

In gstdsexample.cpp file ,

gst_dsexample_transform_ip()
{

if (frame[i].num_obj_meta > 0 )
{
if (get_converted_mat (dsexample, surface, i, &rect_params,
scale_ratio, dsexample->video_info.width,
dsexample->video_info.height) != GST_FLOW_OK) {
goto error;
}

}

}

Using this if loop I am able to get only few , detected images . However not every detected images. Would be glad if you can give me some inputs on the same ?

hi
if you are using object detection, you can use class id (obj_meta->class_id) to extract specific object or all object that being detected.

obj_meta = (NvDsObjectMeta *) (l_obj->data)

Hello ,

Thanks for the input.

If i use the above set up in full-frame = 0 mode , it is working fine. However presently I am facing issues in full-frame = 1 mode.

This is code snippet. which i have modified
if (dsexample->process_full_frame) {
for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
l_frame = l_frame->next)
{
frame_meta = (NvDsFrameMeta *) (l_frame->data);
NvOSD_RectParams rect_params;

  // Scale the entire frame to processing resolution
  rect_params.left = 0;
  rect_params.top = 0;
  rect_params.width = dsexample->video_info.width;
  rect_params.height = dsexample->video_info.height;



  // Scale and convert the frame
  /*
  if (get_converted_mat (dsexample, surface, i, &rect_params,
        scale_ratio, dsexample->video_info.width,
        dsexample->video_info.height) != GST_FLOW_OK) {
    goto error;
  }
  */

  // Process to get the output
  output =
      DsExampleProcess (dsexample->dsexamplelib_ctx,
      dsexample->cvmat->data);
  // Attach the metadata for the full frame


  // Modification 
  if (frame_meta[i].num_obj_meta > 0 ){

        if (get_converted_mat (dsexample, surface, i, &rect_params,
                scale_ratio, dsexample->video_info.width,
                dsexample->video_info.height) != GST_FLOW_OK) {
                goto error;
         }

  }
  attach_metadata_full_frame (dsexample, frame_meta, scale_ratio, output, i);
  i++;
  free (output);
}

}

@Danelll It creates a NvBufSurface for each object and saves it to a jpg file.

I am using exact same snippet as provided by you. It works perfectly fine on single source and I get segmentation fault while running it for multiple sources. Any pointers to resolve it ?

Hi hrsht.sarma,
Please look at
https://devtalk.nvidia.com/default/topic/1063771/deepstream-sdk/uridecodebin-with-filesink/post/5404139/#5404139
You would need to configure correct batch-size.

Hey @DaneLLL Thanks for quick response. As per other post on the forum I already tried setting up batch-size as 2 as my number of sources are 2 , also changed it to 1 . But still I receive segmentation fault. I also tried freeing up extra memory in the end. Not sure what is wrong here.

static GstFlowReturn
 get_converted_mat (GstDsExample * dsexample, NvBufSurface *input_buf, gint idx,
     NvOSD_RectParams * crop_rect_params, gdouble & ratio, gint input_width,
     gint input_height, int object_id, char *obj_label, gint source_id_info)
{
  NvBufSurfTransform_Error err;
  NvBufSurfTransformConfigParams transform_config_params;
  NvBufSurfTransformParams transform_params;
  NvBufSurfTransformRect src_rect;
  NvBufSurfTransformRect dst_rect;
  NvBufSurface ip_surf;
  cv::Mat in_mat, out_mat;
  ip_surf = *input_buf;

  ip_surf.numFilled = ip_surf.batchSize = 1;
  ip_surf.surfaceList = &(input_buf->surfaceList[idx]);

  gint src_left = GST_ROUND_UP_2(crop_rect_params->left);
  gint src_top = GST_ROUND_UP_2(crop_rect_params->top);
  gint src_width = GST_ROUND_DOWN_2(crop_rect_params->width);
  gint src_height = GST_ROUND_DOWN_2(crop_rect_params->height);
  //g_print("ltwh = %d %d %d %d \n", src_left, src_top, src_width, src_height);

  guint dest_width, dest_height;
  dest_width = src_width;
  dest_height = src_height;

  NvBufSurface *nvbuf;
  NvBufSurfaceCreateParams create_params;
  create_params.gpuId  = dsexample->gpu_id;
  create_params.width  = dest_width;
  create_params.height = dest_height;
  create_params.size = 0;
  create_params.colorFormat = NVBUF_COLOR_FORMAT_RGBA;
  create_params.layout = NVBUF_LAYOUT_PITCH;
#ifdef __aarch64__
  create_params.memType = NVBUF_MEM_DEFAULT;
#else
  create_params.memType = NVBUF_MEM_CUDA_UNIFIED;
#endif
  NvBufSurfaceCreate (&nvbuf, 1, &create_params);

  // Configure transform session parameters for the transformation
  transform_config_params.compute_mode = NvBufSurfTransformCompute_Default;
  transform_config_params.gpu_id = dsexample->gpu_id;
  transform_config_params.cuda_stream = dsexample->cuda_stream;

  // Set the transform session parameters for the conversions executed in this
  // thread.
  err = NvBufSurfTransformSetSessionParams (&transform_config_params);
  if (err != NvBufSurfTransformError_Success) {
    GST_ELEMENT_ERROR (dsexample, STREAM, FAILED,
        ("NvBufSurfTransformSetSessionParams failed with error %d", err), (NULL));
    goto error;
  }

  // Calculate scaling ratio while maintaining aspect ratio
  ratio = MIN (1.0 * dest_width/ src_width, 1.0 * dest_height / src_height);

  if ((crop_rect_params->width == 0) || (crop_rect_params->height == 0)) {
    GST_ELEMENT_ERROR (dsexample, STREAM, FAILED,
        ("%s:crop_rect_params dimensions are zero",__func__), (NULL));
    goto error;
  }

#ifdef __aarch64__
  if (ratio <= 1.0 / 16 || ratio >= 16.0) {
    // Currently cannot scale by ratio > 16 or < 1/16 for Jetson
    goto error;
  }
#endif
  // Set the transform ROIs for source and destination
  src_rect = {(guint)src_top, (guint)src_left, (guint)src_width, (guint)src_height};
  dst_rect = {0, 0, (guint)dest_width, (guint)dest_height};

  // Set the transform parameters
  transform_params.src_rect = &src_rect;
  transform_params.dst_rect = &dst_rect;
  transform_params.transform_flag =
    NVBUFSURF_TRANSFORM_FILTER | NVBUFSURF_TRANSFORM_CROP_SRC |
      NVBUFSURF_TRANSFORM_CROP_DST;
  transform_params.transform_filter = NvBufSurfTransformInter_Default;

  //Memset the memory
  NvBufSurfaceMemSet (nvbuf, 0, 0, 0);

  GST_DEBUG_OBJECT (dsexample, "Scaling and converting input buffer\n");

  // Transformation scaling+format conversion if any.
  err = NvBufSurfTransform (&ip_surf, nvbuf, &transform_params);
  if (err != NvBufSurfTransformError_Success) {
    GST_ELEMENT_ERROR (dsexample, STREAM, FAILED,
        ("NvBufSurfTransform failed with error %d while converting buffer", err),
        (NULL));
    goto error;
  }
  // Map the buffer so that it can be accessed by CPU
  if (NvBufSurfaceMap (nvbuf, 0, 0, NVBUF_MAP_READ) != 0){
    goto error;
  }

  // Cache the mapped data for CPU access
  NvBufSurfaceSyncForCpu (nvbuf, 0, 0);

  // Use openCV to remove padding and convert RGBA to BGR. Can be skipped if
  // algorithm can handle padded RGBA data.
  in_mat =
      cv::Mat (dest_height, dest_width,
      CV_8UC4, nvbuf->surfaceList[0].mappedAddr.addr[0],
      nvbuf->surfaceList[0].pitch);
  out_mat =
      cv::Mat (cv::Size(dest_width, dest_height), CV_8UC3);

  cv::cvtColor (in_mat, out_mat, CV_RGBA2BGR);
  static gint dump = 0;
  if (dump < 150) {
      char filename[64];
      snprintf(filename, 64, "/root/detected_obj/%s_%d_%s.jpg",source_id_info,object_id,obj_label);
      //snprintf(filename, 64, "/home/nvidia/image%03d.jpg", dump);
      cv::imwrite(filename, out_mat);
      dump++;
  }

  if (NvBufSurfaceUnMap (nvbuf, 0, 0)){
    goto error;
  }
  NvBufSurfaceDestroy(nvbuf);

#ifdef __aarch64__
  // To use the converted buffer in CUDA, create an EGLImage and then use
  // CUDA-EGL interop APIs
  if (USE_EGLIMAGE) {
    if (NvBufSurfaceMapEglImage (dsexample->inter_buf, 0) !=0 ) {
      goto error;
    }

    // dsexample->inter_buf->surfaceList[0].mappedAddr.eglImage
    // Use interop APIs cuGraphicsEGLRegisterImage and
    // cuGraphicsResourceGetMappedEglFrame to access the buffer in CUDA

    // Destroy the EGLImage
    NvBufSurfaceUnMapEglImage (dsexample->inter_buf, 0);
  }
#endif

  /* We will first convert only the Region of Interest (the entire frame or the
   * object bounding box) to RGB and then scale the converted RGB frame to
   * processing resolution. */
  return GST_FLOW_OK;

error:
  return GST_FLOW_ERROR;
}

Here is my config:

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=../../../kitti/
kitti-track-output-dir=../../../kitti_track/

[tiled-display]
enable=0
rows=1
columns=1
width=1280
height=720
gpu-id=0
#(0): nvbuf-mem-default - Default memory allocated, specific to particular platform
#(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory, applicable for Tesla
#(2): nvbuf-mem-cuda-device - Allocate Device cuda memory, applicable for Tesla
#(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory, applicable for Tesla
#(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
nvbuf-memory-type=0

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
#uri=file://../../../streams/sample_1080p_h264.mp4
uri=file://../../../../../harshit_sample/harshit_sample_vantagedorm_1.mp4
#uri=rtsp://admin:@192.168.1.10:554/mode=real&idc=1&ids=1
num-sources=1
gpu-id=0
#drop-frame-interval=5
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[source1]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
uri=file://../../../streams/sample_1080p_h264.mp4
num-sources=1
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File
type=1
sync=0
source-id=0
gpu-id=0
nvbuf-memory-type=0

[sink1]
enable=0
type=3
#1=mp4 2=mkv
container=1
#1=h264 2=h265
codec=3
sync=0
bitrate=2000000
output-file=out.mp4
source-id=0
gpu-id=0

[sink2]
enable=0
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=h264 2=h265
codec=1
sync=0
bitrate=4000000
# set below properties in case of RTSPStreaming
rtsp-port=8554
udp-port=5400
gpu-id=0

[osd]
enable=1
gpu-id=0
border-width=1
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=0
batch-size=2
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
## Set muxer output width and height
width=1920
height=1080
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0

# config-file property is mandatory for any gie section.
# Other properties are optional and if set will override the properties set in
# the infer config file.
[primary-gie]
enable=1
gpu-id=0
model-engine-file=../../../models/Primary_Detector/resnet10.caffemodel_b4_int8.engine
batch-size=2
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=0
gie-unique-id=1
nvbuf-memory-type=0
config-file=config_infer_primary.txt

[tracker]
enable=1
tracker-width=640
tracker-height=368
#ll-lib-file=/opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_mot_iou.so
ll-lib-file=/opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_nvdcf.so
#ll-lib-file=/opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_mot_klt.so
#ll-config-file required for DCF/IOU only
ll-config-file=tracker_config.yml
#ll-config-file=iou_config.txt
gpu-id=0
#enable-batch-process applicable to DCF only
enable-batch-process=1

[secondary-gie]
enable=1
model-engine-file=../../../models/Secondary_FaceDetect/fd_lpd_model/fd_lpd.caffemodel_b1_fp32.engine
gpu-id=0
batch-size=2
gie-unique-id=4
operate-on-gie-id=1
#operate-on-class-ids=0;
config-file=config_infer_secondary.txt

[ds-example]
enable=1
processing-width=640
processing-height=480
full-frame=0
unique-id=15
gpu-id=0

[tests]
file-loop=0

hi,
There is reference code in #13. It demonstrates how to get NvBufSurface at index 1 and sve to a JPG. Please take a look and investigate further.

Thanks for the pointer, let me look into this and update

Hi DaneLLL

I’ve made the changes in “/opt/nvidia/deepstream/deepstream-4.0/sources/gst-plugins/gst-dsexample/gstdsexample.cpp” as you have suggested in #2. I’ve also changed the path for storing an image to home folder in ubuntu. Ive compiled ds-example plugin using opencv4 successfully and added ds-example plugin in “dstest3_pgie_config.txt”. I didn’t make any changes inside “deepstream_test3_app.c” file. Deepstream app run successfully on rtsp camera but cropped images are not getting saved at specified location.

am i doing something wrong in configuration?

Is there a C example for this instead of Cpp ? I tried modifying the deepstream_app.c to save JPEG, but I get an image with white pixels(see attached). What am I doing wrong here ?

int write_jpeg_file( char *filename, unsigned char* rgb_image , int width, int height, int bytes_per_pixel, J_COLOR_SPACE color_space )
{

	struct jpeg_compress_struct cinfo;
	struct jpeg_error_mgr jerr;
	
	JSAMPROW row_pointer[1];
	FILE *outfile = fopen( filename, "wb" );
	
	if ( !outfile )
	{
		printf("Error opening output jpeg file %s\n!", filename );
		return -1;
	}
	cinfo.err = jpeg_std_error( &jerr );
	jpeg_create_compress(&cinfo);
	jpeg_stdio_dest(&cinfo, outfile);

	cinfo.image_width = width;	
	cinfo.image_height = height;
	cinfo.input_components = bytes_per_pixel;
	cinfo.in_color_space = color_space; //JCS_RGB

	jpeg_set_defaults( &cinfo );

	jpeg_start_compress( &cinfo, TRUE );

	while( cinfo.next_scanline < cinfo.image_height )
	{
		row_pointer[0] = &rgb_image[ cinfo.next_scanline * cinfo.image_width *  cinfo.input_components];
		jpeg_write_scanlines( &cinfo, row_pointer, 1 );
	}

	jpeg_finish_compress( &cinfo );
	jpeg_destroy_compress( &cinfo );
	fclose( outfile );

	return 1;
}

/**
 * Function which processes the inferred buffer and its metadata.
 * It also gives opportunity to attach application specific
 * metadata (e.g. clock, analytics output etc.).
 */

static void
process_buffer (GstBuffer * buf, AppCtx * appCtx, guint index)
{
  NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta (buf);

  NvDsMetaList *l_frame = NULL;
  NvDsMetaList *l_obj = NULL;
  NvDsMetaList * l_user_meta = NULL;
  NvDsDisplayMeta *display_meta = NULL;
  NvDsObjectMeta *obj_meta = NULL;

  //g_print("getting original raw data");
  // Get original raw data
    GstMapInfo in_map_info;
    char* src_data = NULL;
    memset (&in_map_info, 0, sizeof (in_map_info));
    if (!gst_buffer_map (buf, &in_map_info, GST_MAP_READ)) {
        g_print ("Error: Failed to map gst buffer\n");
        gst_buffer_unmap (buf, &in_map_info);
        return;
    }
    NvBufSurface *surface = (NvBufSurface *)in_map_info.data;

    int imgpix=0;
    int txtstep;
    unsigned char *totalpix;
    char *totalpix1;
    unsigned int *totalpix2;
    long unsigned int *totalpix3;

    //g_print("loaded surface");

if (!batch_meta) {
    NVGSTDS_WARN_MSG_V ("Batch meta not found for buffer %p", buf);
    return;
  }
  process_meta (appCtx, batch_meta);
  NvDsInstanceData *data = &appCtx->instance_data[index];
  guint i;

    data->frame_num++;

for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
       l_frame = l_frame->next)
  {
    NvDsFrameMeta *frame_meta = (NvDsFrameMeta *)(l_frame->data);
    NvDsUserMeta *user_meta = NULL;

    //call this every X frames
    if(data->frame_num % 90 == 0) {

      g_print("saving JPEG now.");
        src_data = (char*) malloc(surface->surfaceList[frame_meta->batch_id].dataSize);
        if(src_data == NULL) {
            g_print("Error: failed to malloc src_data \n");
            continue;
        }

#ifdef PLATFORM_TEGRA
        NvBufSurfaceMap (surface, -1, -1, NVBUF_MAP_READ);
        NvBufSurfacePlaneParams *pParams = &surface->surfaceList[frame_meta->batch_id].planeParams;
        unsigned int offset = 0;
        for(unsigned int num_planes=0; num_planes < pParams->num_planes; num_planes++){
            if(num_planes>0)
                offset += pParams->height[num_planes-1]*(pParams->bytesPerPix[num_planes-1]*pParams->width[num_planes-1]);
            for (unsigned int h = 0; h < pParams->height[num_planes]; h++) {
             memcpy((void *)(src_data+offset+h*pParams->bytesPerPix[num_planes]*pParams->width[num_planes]),
                    (void *)((char *)surface->surfaceList[frame_meta->batch_id].mappedAddr.addr[num_planes]+h*pParams->pitch[num_planes]),
                    pParams->bytesPerPix[num_planes]*pParams->width[num_planes]
                    );
            }
        }
        NvBufSurfaceSyncForDevice (surface, -1, -1);
        NvBufSurfaceUnMap (surface, -1, -1);
#else
        cudaMemcpy((void*)src_data,
                   (void*)surface->surfaceList[frame_meta->batch_id].dataPtr,
                   surface->surfaceList[frame_meta->batch_id].dataSize,
                   cudaMemcpyDeviceToHost);
#endif  

        char filename[200];
        uint32_t frame_width  =  surface->surfaceList[frame_meta->batch_id].width;
	      uint32_t frame_height =  surface->surfaceList[frame_meta->batch_id].height;
      	uint32_t Y_stride     =  surface->surfaceList[frame_meta->batch_id].pitch;
	      uint32_t buffer_size = surface->surfaceList[frame_meta->batch_id].dataSize;
	      uint32_t est_size = frame_width*frame_height;

        sprintf(filename,"file_y_%dx%d_%ld.jpg",frame_width,frame_height, data->frame_num);
        printf("Buffer size : %d,%d\n", buffer_size,Y_stride);
        printf("Estimated size : %d\n", est_size);
	      printf("frame_width : %d\n", frame_width);
	      printf("frame_height : %d\n", frame_height);
        printf("dataPtr : %p\n", surface->surfaceList[frame_meta->batch_id].dataPtr);
        printf("src_size:%ld\n",sizeof(char));
        printf("totalpix1:%ld\n",sizeof(unsigned char));
        printf("totalpix2:%ld\n",sizeof(unsigned int));
        printf("totalpix3:%ld\n",sizeof(long unsigned int));

        totalpix=src_data;
        printf("!%ln!",(long unsigned int*) src_data);
        printf("-%ln-",(long unsigned int*) src_data);
        printf("=%s=",(char*) src_data);
        sprintf(filename,"file_y_%dx%d_%d.jpg",frame_width,frame_height,outnumber);

        write_jpeg_file( filename, src_data , frame_width, frame_height, 1, JCS_GRAYSCALE );

        if(src_data != NULL) {
            free(src_data);
            src_data = NULL;
        }
    }
    
  }
  
  gst_buffer_unmap (buf, &in_map_info);

/* Opportunity to modify the processed metadata or do analytics based on
   * type of object e.g. maintaining count of particular type of car.
   */
  if (appCtx->all_bbox_generated_cb) {
    appCtx->all_bbox_generated_cb (appCtx, buf, batch_meta, index);
  }
  //data->bbox_list_size = 0;

  /*
   * callback to attach application specific additional metadata.
   */
  if (appCtx->overlay_graphics_cb) {
    appCtx->overlay_graphics_cb (appCtx, buf, batch_meta, index);
  }
}

Hi bestestt
You may start a new post with more information. The code in #31 seems not related to DeepStream SDK. Not easy to give suggestion with limited information.

Hi.
I would like to ask how to classify and save the target image
Can you give me a demo of the code?

Hi,
The sample code is in #2
Please enable dsexample and apply the patch for a try.

Hi,
Thank you for your reply!
I’ve run this example, but the effect is that the detected target objects are all in one place.
How to save different category objects in different places?

Hi,
You can specify different path in

  if (dump < 150) {
      char filename[64];
      snprintf(filename, 64, "/home/nvidia/image%03d.jpg", dump);
      cv::imwrite(filename, out_mat);
      dump++;
  }

Hi,
How do I judge the different categories of objects?

Hi,

You can look at the logic of extracting metadata in gst_dsexample_transform_ip().

@DaneLLL
How do I start if my objective is to crop/save Deepsteam facedetectIR detections?
Does it require to add code to config file? Or to build a separate app that will read from RTSP outputs?Both? Neither of the two?

Hi,
We have a implementation of encoding detected objects into JPEGs. Please check the header file:

/opt/nvidia/deepstream/deepstream-5.0/sources/includes/nvds_obj_encode.h

And the sample:

/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-image-meta-test

Check if it can be applied to your usecase.