Nvjpegdec lost buffer nvds meta

Abama · August 13, 2021, 5:22am

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)

GPU T4
• DeepStream Version
deepstream-5.1

• TensorRT Version
tensorrt-7.2.2.3
• NVIDIA GPU Driver Version (valid for GPU only)

NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4

• Issue Type( questions, new requirements, bugs)

bugs
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

I create pipeline

appsrc ->queue → jpegdec → fakesink

read image(JPG) from disk and use ** gst_buffer_add_nvds_meta ** method add some meta into buffer , then I probe all element in pipeline, still find the meta in nvjpegdec sink probe callback , but the meta data in nvjpegdec src probe callback lost ,that’s why ?

more: I and probe the meta in this probe callback

appsrc src probe
queue sink probe
queue src probe
nvjpegdec sink probe

when nvjpegdec src probe callback run ,the meta lost

the probe callback code

static GstPadProbeReturn metric_probe(GstPad *pad, GstPadProbeInfo *info, gpointer u_data) {
        GstElement *element = (GstElement *) u_data;
        GstBuffer *buf = (GstBuffer *) info->data;

        GstMeta *gst_meta;
        NvDsMeta *dsmeta;
        gpointer state = NULL;
        NvDsBatchMeta *batch_meta = NULL;
        NvDsMetaList *l_frame = NULL;
        NvDsUserMeta *user_meta = NULL;
        NvDsMetaList *l_user_meta = NULL;
        GstMapInfo inmap = GST_MAP_INFO_INIT;

        logger_info("element %s pad %s received,buffer=%p,size=%ld,thread_id=%ld", GST_ELEMENT_NAME(element),
                    GST_PAD_NAME(pad),
                    buf, inmap.size, std::this_thread::get_id());


        gboolean find = false;
        batch_meta = gst_buffer_get_nvds_batch_meta(buf);
        if (batch_meta) {
            logger_info("gst_buffer_get_nvds_batch_meta,element %s pad %s ", GST_ELEMENT_NAME(element),
                        GST_PAD_NAME(pad));
            for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
                 l_frame = l_frame->next) {

                NvDsFrameMeta *frame_meta = (NvDsFrameMeta *) (l_frame->data);

                for (l_user_meta = frame_meta->frame_user_meta_list; l_user_meta != NULL;
                     l_user_meta = l_user_meta->next) {

                    logger_info("frame_user_meta_list,element %s pad %s ", GST_ELEMENT_NAME(element),
                                GST_PAD_NAME(pad));


                    user_meta = (NvDsUserMeta *) (l_user_meta->data);
                    if (user_meta->base_meta.meta_type == JA_HTTP_SRC_REQUEST_META) {
                        find = true;
                        gst_println("OKKKKKKKKKKK,element=%s pad=%s", GST_ELEMENT_NAME(element), GST_PAD_NAME(pad));
                    }
                }
            }
        } else {
            while ((gst_meta = gst_buffer_iterate_meta(buf, &state))) {
                dsmeta = (NvDsMeta *) gst_meta;
                logger_info("meta type:%d", dsmeta->meta_type);
                if (dsmeta->meta_type == JA_HTTP_SRC_REQUEST_META) {
                    find = true;
                    gst_println("OKKKKKKKKKKK--shi,element=%s pad=%s", GST_ELEMENT_NAME(element), GST_PAD_NAME(pad));
                }
            }
        }

        if (find) {
            logger_info("meta exist,element=%s pad=%s", GST_ELEMENT_NAME(element), GST_PAD_NAME(pad));
        }else {
            logger_info("meta lost,element=%s pad=%s", GST_ELEMENT_NAME(element), GST_PAD_NAME(pad));

        }

        return GST_PAD_PROBE_OK;
    }


    /**
     *
     * @param element
     * @param pad
     */
    void Metric::install_metric_probe(GstElement *element, std::string pad) {
        int prob_id = 0;
        logger_info("install metric probe for element=%s,pad=%s", GST_ELEMENT_NAME(element), pad.c_str());
        NVGSTDS_ELEM_ADD_PROBE(prob_id, element, pad.c_str(), metric_probe, GST_PAD_PROBE_TYPE_BUFFER, element);
    }

• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Fiona.Chen · August 13, 2021, 5:42am

We can not reproduce the problem with just the piece of code. Can you provide a simple source code to reproduce the problem?

Abama · August 13, 2021, 5:56am

YES , but the code is a little long


// meta struct 
class  HttpSrcRequestMeta {

    public:
        gpointer http_src_request;


    };


gpointer http_src_request_meta_copy_func(gpointer data, gpointer user_data) {
        logger_info("http_src_request_meta_copy_func");
        HttpSrcRequestMeta *src_request_meta = (HttpSrcRequestMeta *) data;
        HttpSrcRequestMeta *dst_request_meta = (HttpSrcRequestMeta *) g_malloc0(sizeof(HttpSrcRequestMeta));
        memcpy(dst_request_meta, src_request_meta, sizeof(HttpSrcRequestMeta));
        return (gpointer) dst_request_meta;

    }

    void http_src_request_meta_release_func(gpointer data, gpointer user_data) {
        logger_info("http_src_request_meta_release_func");

        HttpSrcRequestMeta *request_meta = (HttpSrcRequestMeta *) data;
        if (request_meta) {
            g_free(request_meta);
            request_meta = NULL;
        }
    }


    gpointer http_src_request_gst_to_nvds_meta_transform_func(gpointer data, gpointer user_data) {
        logger_info("http_src_request_gst_to_nvds_meta_transform_func");
        NvDsUserMeta *user_meta = (NvDsUserMeta *) data;
        HttpSrcRequestMeta *src_request_meta =
                (HttpSrcRequestMeta *) user_meta->user_meta_data;
        HttpSrcRequestMeta *dst_request_meta =
                (HttpSrcRequestMeta *) http_src_request_meta_copy_func(src_request_meta, NULL);
        return (gpointer) dst_request_meta;
    }

    void http_src_request_gst_nvds_meta_release_func(gpointer data, gpointer user_data) {
        logger_info("http_src_request_gst_nvds_meta_release_func");
        NvDsUserMeta *user_meta = (NvDsUserMeta *) data;
        HttpSrcRequestMeta *request_meta = (HttpSrcRequestMeta *) user_meta->user_meta_data;
        http_src_request_meta_release_func(request_meta, NULL);
    }



// probe utils 

static GstPadProbeReturn metric_probe(GstPad *pad, GstPadProbeInfo *info, gpointer u_data) {
        GstElement *element = (GstElement *) u_data;
        GstBuffer *buf = (GstBuffer *) info->data;

        GstMeta *gst_meta;
        NvDsMeta *dsmeta;
        gpointer state = NULL;
        NvDsBatchMeta *batch_meta = NULL;
        NvDsMetaList *l_frame = NULL;
        NvDsUserMeta *user_meta = NULL;
        NvDsMetaList *l_user_meta = NULL;
        GstMapInfo inmap = GST_MAP_INFO_INIT;

        logger_info("element %s pad %s received,buffer=%p,size=%ld,thread_id=%ld", GST_ELEMENT_NAME(element),
                    GST_PAD_NAME(pad),
                    buf, inmap.size, std::this_thread::get_id());


        gboolean find = false;
        batch_meta = gst_buffer_get_nvds_batch_meta(buf);
        if (batch_meta) {
            logger_info("gst_buffer_get_nvds_batch_meta,element %s pad %s ", GST_ELEMENT_NAME(element),
                        GST_PAD_NAME(pad));
            for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
                 l_frame = l_frame->next) {

                NvDsFrameMeta *frame_meta = (NvDsFrameMeta *) (l_frame->data);

                for (l_user_meta = frame_meta->frame_user_meta_list; l_user_meta != NULL;
                     l_user_meta = l_user_meta->next) {

                    logger_info("frame_user_meta_list,element %s pad %s ", GST_ELEMENT_NAME(element),
                                GST_PAD_NAME(pad));


                    user_meta = (NvDsUserMeta *) (l_user_meta->data);
                    if (user_meta->base_meta.meta_type == JA_HTTP_SRC_REQUEST_META) {
                        find = true;
                        gst_println("OKKKKKKKKKKK,element=%s pad=%s", GST_ELEMENT_NAME(element), GST_PAD_NAME(pad));
                    }
                }
            }
        } else {
            while ((gst_meta = gst_buffer_iterate_meta(buf, &state))) {
                dsmeta = (NvDsMeta *) gst_meta;
                logger_info("meta type:%d", dsmeta->meta_type);
                if (dsmeta->meta_type == JA_HTTP_SRC_REQUEST_META) {
                    find = true;
                    gst_println("OKKKKKKKKKKK--shi,element=%s pad=%s", GST_ELEMENT_NAME(element), GST_PAD_NAME(pad));
                }
            }
        }

        if (find) {
            logger_info("meta exist,element=%s pad=%s", GST_ELEMENT_NAME(element), GST_PAD_NAME(pad));
        }else {
            logger_info("meta lost,element=%s pad=%s", GST_ELEMENT_NAME(element), GST_PAD_NAME(pad));

        }

        return GST_PAD_PROBE_OK;
    }


    /**
     *
     * @param element
     * @param pad
     */
    void Metric::install_metric_probe(GstElement *element, std::string pad) {
        int prob_id = 0;
        logger_info("install metric probe for element=%s,pad=%s", GST_ELEMENT_NAME(element), pad.c_str());
        NVGSTDS_ELEM_ADD_PROBE(prob_id, element, pad.c_str(), metric_probe, GST_PAD_PROBE_TYPE_BUFFER, element);
    }



// bus calback 
gboolean static bus_call_back(GstBus *bus, GstMessage *message, gpointer user_data) {

    logger_info("received message on bus: source %s, msg_type %s",
                GST_MESSAGE_SRC_NAME(message), GST_MESSAGE_TYPE_NAME(message));
}



// pipeline worker thread 
static void worker(GstElement *pipeline) {
    if (gst_element_set_state(pipeline,
                              GST_STATE_PLAYING) == GST_STATE_CHANGE_FAILURE) {
        logger_error("can't set pipeline to playing state");
    }

}

// init pipeline  and add probe callback 
// appsrc ->queue -> jpegdec -> fakesink


static gboolean init_pipeline2(GstElement *pipeline, GstElement **app_src) {

    GstElement *queue, *jpegdec, *fakesink;

    *app_src = gst_element_factory_make("appsrc", "appsrc");

    g_object_set(G_OBJECT (app_src), "caps",
                 gst_caps_new_simple("image/jpeg",
//                                     "format",G_TYPE_STRING,"I420",
                                     "framerate", GST_TYPE_FRACTION, 0, 1,
                                     NULL), NULL);


    g_object_set(G_OBJECT (app_src),
                 "stream-type", 0,
                 "format", GST_FORMAT_TIME, NULL);
    Metric::install_metric_probe(*app_src, "src");


    queue = gst_element_factory_make("queue", "queue");

    Metric::install_metric_probe(queue, "sink");
    Metric::install_metric_probe(queue, "src");

    jpegdec = gst_element_factory_make("jpegdec", "jpegdec");

    Metric::install_metric_probe(jpegdec, "src");
    Metric::install_metric_probe(jpegdec, "sink");


    fakesink = gst_element_factory_make("fakesink", "fakesink");
    Metric::install_metric_probe(fakesink, "sink");

    gst_bin_add_many(GST_BIN(pipeline), *app_src,queue, jpegdec, fakesink, NULL);


    gboolean ret = element_link(*app_src, "src", NULL, 0, queue, "sink", NULL, 0);
    if (!ret) {
        logger_error("link app_src to convert ");
        return false;
    }


    ret = element_link(queue, "src", NULL, 0, jpegdec, "sink", NULL, 0);
    if (!ret) {
        logger_error("link queue to jpeg ");
        return false;
    }


    ret = element_link(jpegdec, "src", NULL, 0, fakesink, "sink", NULL, 0);
    if (!ret) {
        logger_error("link app_src to convert ");
        return false;
    }


}


// main 

int main(int argc, char *argv[]) {

    gst_init(&argc, &argv);
    log_init();


    GMainLoop *main_loop = g_main_loop_new(NULL, FALSE);
    GstBus *bus;
    int bus_id;
    GstElement *pipeline;
    GstElement *app_src;

    pipeline = gst_pipeline_new("test");
    init_pipeline2(pipeline, &app_src);

    bus = gst_pipeline_get_bus(GST_PIPELINE(pipeline));
    bus_id = gst_bus_add_watch(bus, bus_call_back, NULL);


    std::thread t(worker, pipeline);


    std::this_thread::sleep_for(std::chrono::seconds(5));

//
    static GstClockTime timestamp = 0;
    for (;;) {
        GstFlowReturn ret;
        GstBuffer *buffer;
        IplImage *img;
//
        std::ifstream is("../image/image.jpg");
        is.seekg(0, std::ios::end);
        int len = is.tellg();

        is.seekg(0, std::ios::beg);

        char buf[len];
        is.read(buf, len);
        int size = len;
        logger_info("read image len:%d", len);

        buffer = gst_buffer_new_allocate(NULL, size, NULL);
        GstMapInfo info;
        gst_buffer_map(buffer, &info, GST_MAP_WRITE);

        NvDsMeta *meta = NULL;


        HttpSrcRequestMeta *http_src_request_meta = (HttpSrcRequestMeta *) g_malloc0(sizeof(HttpSrcRequestMeta));

        if(http_src_request_meta == NULL)
        {
            return GST_FLOW_ERROR;
        }
        http_src_request_meta->http_src_request = new HttpSrcRequest(NULL, 0);

        meta = gst_buffer_add_nvds_meta(buffer, http_src_request_meta, NULL,
                                        http_src_request_meta_copy_func, http_src_request_meta_release_func);


        meta->meta_type = (GstNvDsMetaType) JA_HTTP_SRC_REQUEST_META;

        meta->gst_to_nvds_meta_transform_func = http_src_request_gst_to_nvds_meta_transform_func;

        meta->gst_to_nvds_meta_release_func = http_src_request_gst_nvds_meta_release_func;



        memcpy((guchar *) info.data, buf, gst_buffer_get_size(buffer));
        GST_BUFFER_PTS (buffer) = timestamp;
        GST_BUFFER_DURATION (buffer) = gst_util_uint64_scale_int(1, GST_SECOND, 2);
        timestamp += GST_BUFFER_DURATION (buffer);

        ret = gst_app_src_push_buffer(GST_APP_SRC(app_src), buffer);
        logger_info("push buffer:%d", ret);
        gst_buffer_unmap(buffer, &info);
        std::this_thread::sleep_for(std::chrono::seconds(5));
    }
    logger_info("start pipeline success.");
    g_main_loop_run(main_loop);
    logger_info("stopping pipeline!");
    gst_element_set_state(pipeline, GST_STATE_NULL);
    logger_info("deleting pipeline");
    gst_object_unref(pipeline);
    g_main_loop_unref(main_loop);

}

Fiona.Chen · August 13, 2021, 6:00am

Do you mean before nvjpegdec, the meta is available, only in nvjpegdec src pad, the meta is lost?

WHERE and HOW do you add the meta to gst_buf?

How much do you know about gstreamer?

Abama · August 13, 2021, 6:04am

Yes , nvjpegdec src and all element after nvjpegdec both lost .
the pipeline is appsrc-> queue->nvjpegdec-> sink

I add the meta to gst_buf and use gst_app_src_push_buffer method push .

Fiona.Chen · August 13, 2021, 6:07am

The nvds_meta is designed to mark the specific features for the gst_buf. nvjpegdec is a JPEG decoder plugin, the input gst_buf is the compressed JPEG data and the output gst_buf is the decompressed RAW data, so the output gst_buf should have totally new meta for itself. It will not copy any meta fron the input gst_buf.

Fiona.Chen · August 13, 2021, 6:09am

The meta is not lost. Just new gst_buf after decoder for new media data. The metadata is data related. You can not use it to transfer no data related information.

Abama · August 13, 2021, 6:14am

How can I get the meta after nvjpegdec element probe callback ?
Or other solution for push an image to appsrc then decode、infer and so on .

Fiona.Chen · August 13, 2021, 6:32am

If appsrc is a must. You can push decoded RAW data(RGB, YUV,…etc), and the meta can go with them.

Will you use gst-nvinfer to do the inferencing?

Abama · August 13, 2021, 6:45am

YES， appsrc is must, because the input image is through api passed , (http ,rpc or other protocol )

the infer plugin is gst-nvinfer

Fiona.Chen · August 13, 2021, 6:49am

If you want to use gst-nvinfer, please use nvstreammux to generate the batch meta. Gst-nvstreammux — DeepStream 6.1.1 Release documentation

The batch meta must be add with RAW data(YUV or RGB). Your usage is totally wrong.

Please refer to the deepstream document.

Abama · August 13, 2021, 7:13am

sorry , I known streammux, my final pipeline is

appsrc → jpegdec → nvvideoconvert → capsfilter(video/x-raw(memory:NVMM)) ->streammux → infer → sink

becuase as mentioned above ， jpgdec or nvjpegdec cannot pass the meta, that the reason why I post the question .

–
The batch meta must be add with RAW data(YUV or RGB). the buffer I added in appsrc is not batch meta , it is simple nvds meta , I also reference demo in deepstream-gst-metadata-test , key of this question is jpegdec ( nvjpegdec ) rebuild buffer .

Fiona.Chen · August 13, 2021, 7:21am

The suggestion is to push RAW data to appsrc instead of jpeg data to avoid using nvjpegdec or any kind of decoder.

Abama · August 13, 2021, 7:42am

OK , Thank you very much .

Topic		Replies	Views
gstnvdsmeta Not Working - Cannot attach metadata at appsrc level DeepStream SDK	12	2910	October 12, 2021
Deepstream read from folder of images DeepStream SDK	4	1995	September 27, 2021
Access video frame and use opencv to draw on it in pad probe callback DeepStream SDK	19	3898	October 12, 2021
DeepStream 6.1 new streammux CUDA unified vs. default problem DeepStream SDK	18	1124	October 8, 2022
Deepstream-image-decode-test not work as use cv::mat encode data input DeepStream SDK	12	467	January 16, 2023
Restarting pipeline on Deepstream2.0 DeepStream SDK	21	5053	October 12, 2021
Gst_element_set_state() is hanging while setting the element to NULL DeepStream SDK	15	1415	August 17, 2023
Confision with Deepstrem appsink \|\| Segmentation fault DeepStream SDK	4	1450	October 12, 2021
Restarting pipeline on DeepStream 5.0 DeepStream SDK	15	2604	October 12, 2021
Extract raw image from appsink always empty DeepStream SDK	3	257	April 30, 2024

Nvjpegdec lost buffer nvds meta

Related topics