Why jetson gstreamer plugin nvvidconv not support BGR format output?

We are using gstreamer pipeline to capture the rtsp video stream. The pipeline is as follow:

gst-launch-1.0 rtspsrc location=rtsp:// : @192.168.1.108:554 latency=200 drop-on-latency=true ! queue ! rtph264depay ! h264parse ! omxh264dec ! nvvidconv ! video/x-raw,format=RGBA,width=1280,height=720 ! autovideosink

It works but out stream is RGBA. But we need the stream in BGR format, but there is no support for BGR in ‘nvvidconv’ for BGR format.

So, How we can add the support for BGR format in ‘nvvidconv’ ?
Is any other hardware video convertor is available with in-built BGR format ? If any , please suggest us.

Best Regards.

Hi,
It is constraint in hardware converter. Please refer to discussion in
[Gstreamer] nvvidconv, BGR as INPUT

Please also check VPI:
VPI - Vision Programming Interface: Main Page
If you see the functions which can be applied to your use-case, please try it as a substitute of using OpenCV.

Thanks for your reply.
In addition, I would like to ask, how to use ffmpeg based on hardware acceleration to decode rtsp .h264 stream, and use hardware to convert NV12 format to BGR format? Such as rga hardware

Hi,
There’s ffmpeg package enabling hardware decoding. Please check Jetson Nano FAQ

The implementation is based on jetson_multimedia_api. For further functions, you would need to use the APIs and do customization. Please download source code of the package and build it manually.

Since hardware converter doesn’t support BGR, one possible solution is to convert to RGBA first through hardware converter, and re-order to BGR through software code.

Is ffmpeg support rtsp stream decode, These functions appear to support only local camera devices, nvv4l2dec_init_decoder、nvv4l2dec_decode、nvv4l2dec_decoder_get_frame, e.g

I tried to use ffmpeg + h264_nvv4l2dec to test multi-channel video decoding, but the cpu load seems to be quite high. The key part of the code is as follows:

int FFStreamerRtsp::launchStreamer() {
    printf("launchStreamer called\n");
    av_register_all();
    avformat_network_init();
    AVDictionary *options = nullptr;

    av_dict_set(&options, "buffer_size", "1024000", 0); 
    av_dict_set(&options, "rtsp_transport", "tcp", 0); 
    av_dict_set(&options, "stimeout", "3000000", 0); 
    av_dict_set(&options, "max_delay", "500000", 0); 

    mAvFormatContext = avformat_alloc_context(); 
    if(mAvFormatContext == nullptr) {
        printf("channel[%s] avformat_alloc_context fail!",mStreamInfo.mCamProfile->mId.c_str());
        return -1;
    }

    if (0 != avformat_open_input(&mAvFormatContext, mStreamInfo.mCamProfile->mUrl.c_str(), 0, &options)) {
        printf("channel[%s] open input stream fail!",mStreamInfo.mCamProfile->mId.c_str());
        return -1;
    }

    if (avformat_find_stream_info(mAvFormatContext, 0) < 0) {
        printf("channel[%s] get input stream information fail!",mStreamInfo.mCamProfile->mId.c_str());
        return -1;
    }

    int videoIndex = -1;
    unsigned int i = 0;

    AVStream *stream = nullptr;

    for (i = 0; i<mAvFormatContext->nb_streams; i++) {
        if (mAvFormatContext->streams[i]->codec->codec_type == AVMEDIA_TYPE_VIDEO) {
            videoIndex = i;
            stream = mAvFormatContext->streams[videoIndex];
            break;
        }
    }

    printf("111111111111111\n");

    if (videoIndex == -1) {
        printf("channel[%s] didn't find a video stream!",mStreamInfo.mCamProfile->mId.c_str());
        return -1;
    }

    mDec = avcodec_find_decoder_by_name("h264_nvv4l2dec");
    printf("22222222222222222\n");

    if(!mDec) {
        printf("not find h264_nvmi decodec\n");
        return -1;
    }

    mCodecCtx = avcodec_alloc_context3(mDec);
    if(!mCodecCtx) {
        printf("failed to allocate the decoder context for stream");
        return -1;
    }

    int ret = avcodec_parameters_to_context(mCodecCtx, stream->codecpar);
    if(ret < 0) {
        printf("failed to copy decoder parameters to input decoder context for stream");
        return ret;
    }

    mCodecCtx->framerate = av_guess_frame_rate(mAvFormatContext, stream, nullptr);
    ret = avcodec_open2(mCodecCtx, mDec, &options);
    if(ret < 0) {
        printf("failed to open decoder for stream");
        return ret;
    }
    AVPacket pkt;

    AVCodecContext * input_video_codec_ctx = mCodecCtx;
    while (true) {
        ret = av_read_frame(mAvFormatContext, &pkt);
        if(ret < 0) {
            continue;
        } else {
            auto start_time = DESystemClock::currentTimeMillis();
            AVFrame *avFrame = av_frame_alloc();
            auto end_time = DESystemClock::currentTimeMillis();
            printf("av_frame_alloc consume time : %d\n", (end_time - start_time));
            if(!avFrame) {
                ret = AVERROR(ENOMEM);
                if(&pkt) {
                    av_packet_unref(&pkt);
                }
                continue;
            }
            start_time = DESystemClock::currentTimeMillis();
            ret = avcodec_send_packet(input_video_codec_ctx, &pkt);
            end_time = DESystemClock::currentTimeMillis();
            printf("avcodec_send_packet consume time : %d\n", (end_time - start_time));
            if(ret < 0) {
                if(&pkt) {
                    av_packet_unref(&pkt);
                }
                av_frame_free(&avFrame);
                continue;
            }
            start_time = DESystemClock::currentTimeMillis();
            ret = avcodec_receive_frame(input_video_codec_ctx, avFrame);
            end_time = DESystemClock::currentTimeMillis();
            printf("avcodec_receive_frame consume time : %d\n", (end_time - start_time));
            if(ret == AVERROR(EAGAIN) || ret == AVERROR_EOF) {
                av_frame_free(&avFrame);
                continue;
            } else if(ret < 0) {
                continue;
            }

            printf(".");
            av_frame_free(&avFrame);
            av_packet_unref(&pkt);
        }
    }
    return 0;
}```

Hi,
Since the hardware engines in Jetson Nano do not support BGR, you would need to rely on CPU cores for the processing. This looks expected. We would suggest check 00_video_decode and see if you can do process on RGBA buffers.

Thanks for your reply. Now my problem is that it decodes to NV12 format without color space conversion, and the cpu load is still relatively high. Refer to the link in your Q&A : development guide, can you provide the corresponding Demo? Because The functions in the header file such as libavcodec/nvv4l2_dec.h seem to be inconsistent with those in the documentation.

Hi,
It is possible to see higher CPU usage in ffmpeg. Hardware decoder decodes frame data to hardware DMA buffer(named NvBuffer) and ffmpeg uses CPU buffer in the frameworks. So decoded data is in NvBuffer and has to copy to CPU buffer, and pass to upper ffmpeg frameworks. Due to CPU capability of Jetson Nano, it may not achieve target performance in high resolution.

Would suggest try 00_video_decode sample. The data in NvBuffer can be directly processed through CUDA, and rendered out.

so, Will there be the same problem with gstreamer?

Hi,
In gstreamer, NvBuffer is video/x-raw(memory:NVMM). If you can have all buffers in video/x-raw(memory:NVMM) from source to sink, the performance is optimal. If need to copy video/x-raw(memory:NVMM) to video/x-raw, CPU usage will be higher.

Hi,
Can you provide a demo that uses tegra_multimedia_api to deal with the decoding of the ip camera streamer, Thanks a lot

Hi,
In default 00_video_decode sample, H264/H265 stream is read from file and do decoding. You would need to implement code for receiving RTSP packets and depaying to H264/H265 stream.

The existing solution is in gstreamer. There are uridecodebin and rtspsrcplugins which can be used for IP cameras. You may try

$ gst-launch-1.0 uridecodebin uri='rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4' ! nvoverlaysink

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.