Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU) Jetson • DeepStream Version 7.1 • JetPack Version (valid for Jetson only) 6.2 • TensorRT Version 10.3.0 • NVIDIA GPU Driver Version (valid for GPU only) 540.4.0 • Issue Type( questions, new requirements, bugs) Bug • How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing) • Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)
Hi,
I found that GPU Mem keeps on increasing and after running for around 24 hours 64GB memory of Orin AGX is consumed and board freezes. Please see attached screenshots for start and after 24 hours
I have applied the fix but I am still seeing the issue. The other fix you have mentioned is for Yolo Parser and it is not applicable here. If you recall this is reason why I have transitioned to DS7.1 but still facing the same issue. Please look at this old ticket where you suggested to go to DS7.1
If I uncomment below code then RAM usage is stable. This code is adopted from Deepstream sample code snippet - #3 by bcao Do you see anything wrong in below code implemented in gie_processing_done_buf_prob()
NvBufSurface *inter_buf = nullptr;
NvBufSurfaceCreateParams create_params;
create_params.gpuId = in_surf->gpuId;
create_params.width = width;
create_params.height = height;
create_params.size = 0;
create_params.colorFormat = NVBUF_COLOR_FORMAT_RGBA;
create_params.layout = NVBUF_LAYOUT_PITCH;
create_params.memType = NVBUF_MEM_DEFAULT;
//Create another scratch RGBA NvBufSurface
if (NvBufSurfaceCreate (&inter_buf, 1, &create_params) != 0) {
GST_ERROR ("Error: Could not allocate internal buffer ");
return GST_PAD_PROBE_OK;
}
inter_buf->numFilled = 1;
if(NvBufSurfaceMap (inter_buf, 0, -1, NVBUF_MAP_READ_WRITE) != 0)
std::cout << "map error" << std::endl;
if(NvBufSurfaceSyncForCpu (inter_buf, 0, 0) != 0) {
printf("Sync CPU failed for in_surf\n");
}
cv::Mat inter_mat = cv::Mat(height, width, CV_8UC4, inter_buf->surfaceList[0].mappedAddr.addr[0],
inter_buf->surfaceList[0].pitch);
cv::cvtColor(in_frame_bgr, inter_mat, cv::COLOR_BGR2RGBA);
NvBufSurfaceSyncForDevice(inter_buf, 0, 0);
NvBufSurfTransformConfigParams transform_config_params;
NvBufSurfTransformParams transform_params;
NvBufSurfTransformRect src_rect;
NvBufSurfTransformRect dst_rect;
cudaStream_t cuda_stream;
CHECK_CUDA_STATUS (cudaStreamCreate (&cuda_stream), "Could not create cuda stream");
transform_config_params.compute_mode = NvBufSurfTransformCompute_Default;
transform_config_params.gpu_id = in_surf->gpuId;
transform_config_params.cuda_stream = cuda_stream;
// Set the transform session parameters for the conversions executed in this thread.
if (NvBufSurfTransformSetSessionParams (&transform_config_params) != NvBufSurfTransformError_Success) {
std::cerr << "NvBufSurfTransformSetSessionParams failed with error" << std::endl;
return GST_PAD_PROBE_OK;
}
// Set the transform ROIs for source and destination, only do the color format conversion
src_rect = {0, 0, width, height};
dst_rect = {0, 0, width, height};
// Set the transform parameters
transform_params.src_rect = &src_rect;
transform_params.dst_rect = &dst_rect;
transform_params.transform_flag = NVBUFSURF_TRANSFORM_FILTER | NVBUFSURF_TRANSFORM_CROP_SRC |
NVBUFSURF_TRANSFORM_CROP_DST;
transform_params.transform_filter = NvBufSurfTransformInter_Default;
// Transformation format conversion, Transform rotated RGBA mat to NV12 memory in original input surface
if (NvBufSurfTransform (inter_buf, in_surf, &transform_params) != NvBufSurfTransformError_Success) {
std::cerr << "NvBufSurfTransform failed with error %d while converting buffer" << std::endl;
return GST_PAD_PROBE_OK;
}
// unmap surface after writing
if(NvBufSurfaceUnMap(inter_buf, 0, -1) != 0) {
std::cerr << "Error: Failed to unmap NvBufSurface in_surf" << std::endl;
gst_buffer_unmap(buf, &in_map_info);
return GST_PAD_PROBE_OK;
}
NvBufSurfaceDestroy(inter_buf);
// unmap surface after writing
if(NvBufSurfaceUnMap(in_surf, 0, -1) != 0) {
std::cerr << "Error: Failed to unmap NvBufSurface after writing" << std::endl;
gst_buffer_unmap(buf, &in_map_info);
return GST_PAD_PROBE_OK;
}
I found that the stream was created but not destroyed and causing GPU Mem Leak. I request you to update the reference code. Also review the code I have shared and let me know if you have any comments.
