Nvv4l2h264enc performance problem

hello, I get YUY2 data from camera,and would like to encode h264 data for 1080p 30 fps. Finally I will get the h264 data from appsink. There only 3 cameras,but fps is lower than 30fps. How can I fix this problem? This is my pipeline.

appsrc name=appsrc ! video/x-raw,format=YUY2,width=1920,height=1080,framerate=30/1 ! nvvideoconvert ! video/x-raw, format=NV12, width=1920, height=1080,framerate=(fraction)30/1 ! nvv4l2h264enc control-rate=constant_bitrate bitrate=245760000 iframeinterval=0 profile=0 maxperf-enable=true insert-sps-pps=true insert-vui=true ! appsink name=appsink

Hi,
You may try to allocate NvBuffer in appsrc so that you can put data into the buffers directly, and then run:

appsrc name=appsrc ! video/x-raw(memory:NVMM),format=YUY2,width=1920,height=1080,framerate=30/1 ! nvvideoconvert ! video/x-raw(memory:NVMM), format=NV12, width=1920, height=1080,framerate=(fraction)30/1 ! ...

Please refer to the sample:
Creating a GStreamer source that publishes to NVMM - #7 by DaneLLL

Hi,
Thanks for your reply. I try to allocate NvBuffer and push into appsrc. But It failed to encode.

            cam_buffer.buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
            cam_buffer.buf.memory = V4L2_MEMORY_DMABUF;
            if (-1 == xioctl(m_videoFd, VIDIOC_DQBUF, &cam_buffer.buf)) {
                printf("xioctl VIDIOC_DQBUF failed,dev=%s,error:%d:%s\n",
                       m_strDevName.c_str(), errno, strerror(errno));
                return -1;
            }

            NvBufferMemSyncForDevice(
                m_pNvbuff[cam_buffer.buf.index].dmabuff_fd, 0,
                (void **)&m_pNvbuff[cam_buffer.buf.index].start);
            cam_buffer.data = m_pNvbuff[cam_buffer.buf.index].start;
            cam_buffer.data_len = m_pNvbuff[cam_buffer.buf.index].size;

this is data source, and I push cam_buffer’s data into appsrc.

bool GStreamerEncoder::Encode(const unsigned char *image_buffer,
                              const size_t &image_buffer_size,
                              unsigned char **encoded_buffer,
                              size_t *encoded_buffer_size) {
    if ((nullptr == image_buffer) || (nullptr == encoded_buffer) ||
        (nullptr == encoded_buffer_size)) {
        std::cerr << "Nullptr!\n";
        return false;
    }
    GstClockTime duration, timestamp;
    duration = gst_util_uint64_scale_int(
        1, GST_SECOND,
        fps_);  // static_cast<GstClockTime>((1.0 / fps_) * GST_SECOND);
    timestamp = num_frames_ * duration;
    /**
     * gst_app_src_push_buffer takes ownership of the buffer, so we need to
     *   supply it a copy
     */
    GstBuffer *buffer =
        gst_buffer_new_allocate(nullptr, image_buffer_size, nullptr);
    GstMapInfo info;
    gst_buffer_map(buffer, &info, (GstMapFlags)GST_MAP_READ);
    memcpy(info.data, (guint8 *)image_buffer, image_buffer_size);
    gst_buffer_unmap(buffer, &info);
    GST_BUFFER_DURATION(buffer) = duration;
    GST_BUFFER_PTS(buffer) = timestamp;
    GST_BUFFER_DTS(buffer) = timestamp;
    // set the current number in the frame
    GST_BUFFER_OFFSET(buffer) = num_frames_;
    GstFlowReturn status =
        gst_app_src_push_buffer(GST_APP_SRC(source_), buffer);
    if (status != GST_FLOW_OK) {
        std::cerr << "Error pushing buffer to GStreamer pipeline!\n";
        gst_safe_release(&buffer);
        return false;
    }
    static bool is_sink_ready = false;
    GstSample *sample = nullptr;
    if (is_sink_ready) {
        // pull_sample will block until sample is ready.
        // In our case "sample" is an encoded picture.
        sample = gst_app_sink_pull_sample(GST_APP_SINK(sink_));
    } else {
        // push buffer until AppSink ready, 10ms is a parameter after testing
        GstClockTime timeout_10ms = 500 * 1000000;
        sample =
            gst_app_sink_try_pull_sample(GST_APP_SINK(sink_), timeout_10ms);
        is_sink_ready = true;
    }
    if (nullptr == sample) {
        std::cerr << "[Warning!!!] Error pulling sample from AppSink!\n";

        bool is_eos =
            static_cast<bool>(gst_app_sink_is_eos(GST_APP_SINK(sink_)));
        if (is_eos) {
            std::cerr << "[Warning!!!] AppSink is eos!\n";
        } else {
            std::cerr << "[Warning!!!] Timeout to get AppSink!\n";
        }

        return false;
    }
    GstBuffer *recv_buffer = gst_sample_get_buffer(sample);
    if (!recv_buffer) {
        std::cerr << "Error samping buffer from GStreamer pipeline!\n";
        return false;
    }
    gst_buffer_map(recv_buffer, &info, (GstMapFlags)GST_MAP_READ);
    *encoded_buffer_size = info.size;
    *encoded_buffer = (unsigned char *)malloc(*encoded_buffer_size);
    memcpy(*encoded_buffer, info.data, info.size);
    gst_buffer_unmap(recv_buffer, &info);
    // we don't need the appsink sample anymore
    gst_sample_unref(sample);
    num_frames_++;
    return true;
}

and this is Nvbuffer create

bool CV4l2Cam::initDma() {
    m_pNvbuff = new nv_buffer[V4L2_BUFFER_LENGHT];
    NvBufferCreateParams input_params = {0};
    input_params.payloadType = NvBufferPayload_SurfArray;
    input_params.width = m_dwWidth;
    input_params.height = m_dwHeight;
    input_params.layout = NvBufferLayout_Pitch;

    /* Create buffer and provide it with camera */
    for (unsigned int index = 0; index < V4L2_BUFFER_LENGHT; index++) {
        int fd;
        // NvBufferParams params = {0};
        input_params.colorFormat = get_nvbuff_color_fmt(V4L2_VIDEO_FORMAT);
        input_params.nvbuf_tag = NvBufferTag_CAMERA;
        if (-1 == NvBufferCreateEx(&fd, &input_params)) {
            printf("NvBufferCreateEx failed,devname=%s,errno: %d, %s\n",
                   m_strDevName.c_str(), errno, strerror(errno));
            return false;
        }
        m_pNvbuff[index].dmabuff_fd = fd;
        if (-1 == NvBufferMemMap(m_pNvbuff[index].dmabuff_fd, 0,
                                 NvBufferMem_Read_Write,
                                 (void **)&m_pNvbuff[index].start)) {
            printf("NvBufferMemMap failed,devname=%s,errno: %d, %s\n",
                   m_strDevName.c_str(), errno, strerror(errno));
            return false;
        }
    }

    bool bRet = true;
    struct v4l2_requestbuffers req;
    memset(&req, 0, sizeof(struct v4l2_requestbuffers));
    req.count =
        V4L2_BUFFER_LENGHT;  //缓存数量,根据图像占用空间大小申请的缓存区个数
    req.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;  //数据流类型,视频捕获模式.
    req.memory = V4L2_MEMORY_DMABUF;         // DMA使用方式

    //请求buf
    if (-1 == xioctl(m_videoFd, VIDIOC_REQBUFS, &req)) {
        printf("xioctl  VIDIOC_REQBUFS failed,devname=%s,errno: %d, %s\n",
               m_strDevName.c_str(), errno, strerror(errno));
        return false;
    }

    for (unsigned int index = 0; index < V4L2_BUFFER_LENGHT; index++) {
        struct v4l2_buffer buf;

        /* Query camera v4l2 buf length */
        memset(&buf, 0, sizeof buf);
        buf.index = index;
        buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
        buf.memory = V4L2_MEMORY_DMABUF;

        if (ioctl(m_videoFd, VIDIOC_QUERYBUF, &buf) < 0) {
            bRet = false;
            printf("xioctl  VIDIOC_QUERYBUF failed,devname=%s,errno: %d, %s\n",
                   m_strDevName.c_str(), errno, strerror(errno));
            break;
        }

        buf.m.fd = (unsigned long)m_pNvbuff[index].dmabuff_fd;
        if (buf.length != m_pNvbuff[index].size) {
            // TZTEK_CLASSLOG(TZTEK_WARNING_LEV,"Camera v4l2 buf length is not
            // expected,devname=%s,errno: %d, %s\n", m_strDevName.c_str(),
            // errno, strerror(errno));
            m_pNvbuff[index].size = buf.length;
        }
    }

    if (bRet == false) {
        if (m_pNvbuff != NULL) {
            for (unsigned i = 0; i < V4L2_BUFFER_LENGHT; i++) {
                if (m_pNvbuff[i].dmabuff_fd) {
                    NvBufferDestroy(m_pNvbuff[i].dmabuff_fd);
                }
            }
            delete[] m_pNvbuff;
            m_pNvbuff = NULL;
        }
    }
    return bRet;
}

FYI Nvv4l2h264enc performance problem - #4 by zyukyunman

Hi,
You ay construct the pipeline step by step:

appsrc name=appsrc ! video/x-raw(memory:NVMM),format=YUY2,width=1920,height=1080,framerate=30/1 ! fakesink
appsrc name=appsrc ! video/x-raw(memory:NVMM),format=YUY2,width=1920,height=1080,framerate=30/1 ! nvvideoconvert ! video/x-raw(memory:NVMM), format=NV12, width=1920, height=1080,framerate=(fraction)30/1 ! fakesink

To clarify which plugin triggers the failure.

Hi,
I can pull sample from the first pipeline. But not from the second pipelin.So I think nvvideoconvert triggers the failure.

Why can’t I send my message by replying to you every time?

Hi,

We would need some time to check and provide suggestion. If you see issue with nvvideoconvert plugin, please try nvvidconv plugin. You may also try the reference sample to confirm it works first.

Hi,
I have tried nvvidconv plugin.nvvidconv’s performance is better than nvvideoconvert.But it also failed to run this pipeline.

appsrc name=appsrc ! video/x-raw(memory:NVMM),format=YUY2,width=1920,height=1080,framerate=30/1 ! nvvidconv ! video/x-raw(memory:NVMM), format=NV12, width=1920, height=1080,framerate=(fraction)30/1 ! fakesink

what is the difference between nvvideoconvert and nvvidconv?
Finally,thanks for your support.And looking forward to your reply for more suggestion.

Hi,
Does the default appsrc_nvmm work in your environment?

Hi,
appsrc_nvmm is work.

Hi,
You may gradually adapt launch_stream to your use-case. May try

    << "appsrc name=mysource ! "
    << "video/x-raw(memory:NVMM),width="<< w <<",height="<< h <<",framerate=30/1,format=YUY2 ! "
    << "nvvidconv ! video/x-raw(memory:NVMM),format=NV12 ! "
    << "nvoverlaysink ";

And then

    << "appsrc name=mysource ! "
    << "video/x-raw(memory:NVMM),width="<< w <<",height="<< h <<",framerate=30/1,format=YUY2 ! "
    << "nvvidconv ! video/x-raw(memory:NVMM),format=NV12 ! "
    << "nvv4l2h264enc ! h264parse ! filesink location=a.h264 ";