Hello,
I use cuda to resize/crop the EGLImage, but need to put/apply the cropped/resized image to EGLImage. How can i do it?
Thanks!
// Create EGLImage from dmabuf fd
ctx->egl_image = NvEGLImageFromFd(ctx->egl_display, buffer->planes[0].fd);
if (ctx->egl_image == NULL)
{
fprintf(stderr, "Error while mapping dmabuf fd (0x%X) to EGLImage\n",buffer->planes[0].fd);
return false;
}
// use cuda to resize/crop the egl_image buffer and put the cropped image to egl_image
CropEGLImage(&ctx->egl_image);
// Destroy EGLImage
NvDestroyEGLImage(ctx->egl_display, ctx->egl_image);
ctx->egl_image = NULL;
I know i can use gstreamer-nvvidconv to scale/crop the video, but my requirement is crop several regions in the video image and merge the cropped images and finally output to the display.
Hi,
You need to create buffers through
/**
* Allocates a HW buffer.
* @param[out] dmabuf_fd Returns `dmabuf_fd` of hardware buffer.
* @param[in] input_params Input parameters for hardware buffer creation.
*
* @returns 0 for success, -1 for failure
*/
int NvBufferCreateEx (int *dmabuf_fd, NvBufferCreateParams *input_params);
So that NvEGLImageFromFd() can be called to create EGLImage.
Hi DaneLLL,
Thanks for your replay,I am not familiar with the Multimedia API. I want to do the above process by using nvivafilter in nvsample_cudaprocess.so.But it seems that it’s impossible to put the new image buffer(the cropped image buffer by cuda) to EGLimage.
I’m preparing to implement one gstreamer plugin sink that can receive the NVMM buffer from the decode_bin sink. and do cuda process(crop several regions in the video image and merge the cropped images),then output the processed image to the next sink.Is it possible？ Thank you!
Hi @117365946
You may found interesting the following information about the GstCUDA framework, I think that is exactly what you are looking for. Below you will find a more detailed description, but in summary, it consists of a framework that allows to easily and optimally interface GStreamer with CUDA, guaranteeing zero memory copies. It also supports several inputs.
GstCUDA is a RidgeRun developed GStreamer plug-in enabling easy CUDA algorithm integration into GStreamer pipelines. GstCUDA offers a framework that allows users to develop custom GStreamer elements that execute any CUDA algorithm. The GstCUDA framework is a series of base classes abstracting the complexity of both CUDA and GStreamer. With GstCUDA, developers avoid writing elements from scratch, allowing the developer to focus on the algorithm logic, thus accelerating time to market.
GstCUDA offers a GStreamer plugin that contains a set of elements, that are ideal for GStreamer/CUDA quick prototyping. Those elements consist in a set of filters with different input/output pads combinations, that are run-time loadable with an external custom CUDA library that contains the algorithm to be executed on the GPU on each video frame that passes through the pipeline. GstCUDA plugin allows users to develop their own CUDA processing library, pass the library into the GstCUDA filter element that best adapts to the algorithm requirements, executes the library on the GPU, passing upstream frames from the GStreamer pipeline to the GPU and passing the modified frames downstream to the next element in the GStreamer pipeline. Those elements were created with the CUDA algorithm developer in mind - supporting quick prototyping and abstracting all GStreamer concepts. The elements are fully adaptable to different project needs, making GstCUDA a powerful tool that is essential for CUDA/GStreamer project development.
One remarkable feature of GstCUDA is that it provides a zero memory copy interface between CUDA and GStreamer on Jetson TX1/TX2 platforms. This enables heavy algorithms and large amounts of data (up to 2x 4K 60fps streams) to be processed on CUDA without the performance caused by copies or memory conversions. GstCUDA provides the necessary APIs to directly handle NVMM buffers to achieve the best possible performance on Jetson TX1/TX2 platforms. It provides a series of base classes and utilities that abstract the complexity of handle memory interface between GStreamer and CUDA, so the developer can focus on what actually gives value to the end product. GstCuda ensures an optimal performance for GStreamer/CUDA applications on Jetson platforms.
You can find detailed information about GstCUDA on the following link:
I hope this information can be useful to you.
Best regards,
-Daniel
Hi DaneLLL,
I refer the test code and try to use NvBuffer APIs to get fd in appsink.
but the frameType of EGLImgage getting by NvEGLImageFromFd is block linear format(CU_EGL_FRAME_TYPE_ARRAY) that it seems I can’t do CUDA operations directly on it.
How can i convert the buffer layout from block linear format to pitch linear format. Thank you!
static GstFlowReturn new_buffer(GstAppSink *appsink, gpointer user_data)
{
GstSample *sample = NULL;
g_signal_emit_by_name (appsink, "pull-sample", &sample,NULL);
if (sample)
{
GstBuffer *buffer = NULL;
GstCaps *caps = NULL;
GstMapInfo map = {0};
int dmabuf_fd = 0;
caps = gst_sample_get_caps (sample);
if (!caps)
{
printf("could not get snapshot format\n");
}
gst_caps_get_structure (caps, 0);
buffer = gst_sample_get_buffer (sample);
gst_buffer_map (buffer, &map, GST_MAP_READ);
ExtractFdFromNvBuffer((void *)map.data, &dmabuf_fd);
printf("dmabuf_fd %d \n", dmabuf_fd);
// Create EGLImage from dmabuf fd
egl_image = NvEGLImageFromFd(egl_display, dmabuf_fd);
if (egl_image == NULL)
{
fprintf(stderr, "Error while mapping dmabuf fd (0x%X) to EGLImage\n",
dmabuf_fd);
return GST_FLOW_OK;
}
HandleEGLImage(egl_image);
// Destroy EGLImage
NvDestroyEGLImage(egl_display, egl_image);
//renderer->render(dmabuf_fd);
//NvReleaseFd(dmabuf_fd);
frame_count++;
gst_buffer_unmap(buffer, &map);
gst_sample_unref (sample);
}
else
{
g_print ("could not make snapshot\n");
}
return GST_FLOW_OK;
}
The pipeline i used as follows:
location=Bourne_Trailer.mp4 ! decodebin ! nvvidconv ! "video/x-raw(memory:NVMM), format=NV12 ! appsink
Hi,
It outputs in pitch-linear if it is ‘format=I420’.
location=Bourne_Trailer.mp4 ! decodebin ! nvvidconv ! "video/x-raw(memory:NVMM), <b>format=I420</b> ! appsink
Hi DaneLLL,
I used the following pipeline to test,But the format of EGLImgage is still CU_EGL_FRAME_TYPE_ARRAY.
location=Bourne_Trailer.mp4 ! decodebin ! nvvidconv ! "video/x-raw(memory:NVMM), format=I420 ! appsink
I tried to add a conversion to convert the fd format from block linear to pitch linear.It works well.Here is the code.
static GstFlowReturn new_buffer(GstAppSink *appsink, gpointer user_data)
{
GstSample *sample = NULL;
g_signal_emit_by_name (appsink, "pull-sample", &sample,NULL);
if (sample)
{
GstBuffer *buffer = NULL;
GstCaps *caps = NULL;
GstMapInfo map = {0};
int dmabuf_fd = 0;
caps = gst_sample_get_caps (sample);
if (!caps)
{
printf("could not get snapshot format\n");
}
gst_caps_get_structure (caps, 0);
buffer = gst_sample_get_buffer (sample);
gst_buffer_map (buffer, &map, GST_MAP_READ);
ExtractFdFromNvBuffer((void *)map.data, &dmabuf_fd);
printf("dmabuf_fd %d \n", dmabuf_fd);
int fd_use = 0;
#if 1
// convert BlockLinear format to Pitch
int dst_fd = 0;
NvBufferCreateParams input_params = {0};
NvBufferParams params;
NvBufferGetParams(dmabuf_fd,¶ms);
input_params.payloadType = NvBufferPayload_SurfArray;
input_params.layout = NvBufferLayout_Pitch;
input_params.width = 800;
input_params.height = 480;
input_params.colorFormat = NvBufferColorFormat_ABGR32;
input_params.nvbuf_tag = NvBufferTag_NONE;
NvBufferCreateEx(&dst_fd,&input_params);
NvBufferTransformParams transParams ={0};
memset(&transParams, 0, sizeof(transParams));
transParams.transform_flag = NVBUFFER_TRANSFORM_FILTER;
transParams.transform_filter = NvBufferTransform_Filter_Smart;
NvBufferTransform(dst_fd,dmabuf_fd,&transParams);
fd_use = dst_fd;
#else
fd_use = dmabuf_fd;
#endif
// Create EGLImage from dmabuf fd
egl_image = NvEGLImageFromFd(egl_display, fd_use);
if (egl_image == NULL)
{
fprintf(stderr, "Error while mapping dmabuf fd (0x%X) to EGLImage\n",
dmabuf_fd);
return GST_FLOW_OK;
}
HandleEGLImage(egl_image);
// Destroy EGLImage
NvDestroyEGLImage(egl_display, egl_image);
//renderer->render(dmabuf_fd);
//NvReleaseFd(dmabuf_fd);
frame_count++;
gst_buffer_unmap(buffer, &map);
gst_sample_unref (sample);
}
else
{
g_print ("could not make snapshot\n");
}
return GST_FLOW_OK;
}
Hi,
Using NvBuffertransform() is a good solution.
Hi DaneLLL，
I watched the CPU Usage, it seems strange that the NvBuffertransform conversion takes too much CPU consumption in thread of omxh264dec-omxh. Without the conversion, the omxh264dec-omxh thread take a little CPU usage.
Hi DaneLLL，
I have fond the reason! The high CPU consumption comes from the NvBufferCreateEx in the callback of new_buffer function. After I set dst_fd as a global variable and just call NvBufferCreateEx to initialize it at the beginning，the CPU usage is down. Thank you!