Nvjpegdec lost buffer nvds meta

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)

GPU T4
• DeepStream Version
deepstream-5.1

• TensorRT Version
tensorrt-7.2.2.3
• NVIDIA GPU Driver Version (valid for GPU only)

NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4

• Issue Type( questions, new requirements, bugs)

bugs
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

I create pipeline

appsrc ->queue → jpegdec → fakesink

read image(JPG) from disk and use ** gst_buffer_add_nvds_meta ** method add some meta into buffer , then I probe all element in pipeline, still find the meta in nvjpegdec sink probe callback , but the meta data in nvjpegdec src probe callback lost ,that’s why ?

more: I and probe the meta in this probe callback

  1. appsrc src probe
  2. queue sink probe
  3. queue src probe
  4. nvjpegdec sink probe

when nvjpegdec src probe callback run ,the meta lost

the probe callback code

static GstPadProbeReturn metric_probe(GstPad *pad, GstPadProbeInfo *info, gpointer u_data) {
        GstElement *element = (GstElement *) u_data;
        GstBuffer *buf = (GstBuffer *) info->data;

        GstMeta *gst_meta;
        NvDsMeta *dsmeta;
        gpointer state = NULL;
        NvDsBatchMeta *batch_meta = NULL;
        NvDsMetaList *l_frame = NULL;
        NvDsUserMeta *user_meta = NULL;
        NvDsMetaList *l_user_meta = NULL;
        GstMapInfo inmap = GST_MAP_INFO_INIT;

        logger_info("element %s pad %s received,buffer=%p,size=%ld,thread_id=%ld", GST_ELEMENT_NAME(element),
                    GST_PAD_NAME(pad),
                    buf, inmap.size, std::this_thread::get_id());


        gboolean find = false;
        batch_meta = gst_buffer_get_nvds_batch_meta(buf);
        if (batch_meta) {
            logger_info("gst_buffer_get_nvds_batch_meta,element %s pad %s ", GST_ELEMENT_NAME(element),
                        GST_PAD_NAME(pad));
            for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
                 l_frame = l_frame->next) {

                NvDsFrameMeta *frame_meta = (NvDsFrameMeta *) (l_frame->data);

                for (l_user_meta = frame_meta->frame_user_meta_list; l_user_meta != NULL;
                     l_user_meta = l_user_meta->next) {

                    logger_info("frame_user_meta_list,element %s pad %s ", GST_ELEMENT_NAME(element),
                                GST_PAD_NAME(pad));


                    user_meta = (NvDsUserMeta *) (l_user_meta->data);
                    if (user_meta->base_meta.meta_type == JA_HTTP_SRC_REQUEST_META) {
                        find = true;
                        gst_println("OKKKKKKKKKKK,element=%s pad=%s", GST_ELEMENT_NAME(element), GST_PAD_NAME(pad));
                    }
                }
            }
        } else {
            while ((gst_meta = gst_buffer_iterate_meta(buf, &state))) {
                dsmeta = (NvDsMeta *) gst_meta;
                logger_info("meta type:%d", dsmeta->meta_type);
                if (dsmeta->meta_type == JA_HTTP_SRC_REQUEST_META) {
                    find = true;
                    gst_println("OKKKKKKKKKKK--shi,element=%s pad=%s", GST_ELEMENT_NAME(element), GST_PAD_NAME(pad));
                }
            }
        }

        if (find) {
            logger_info("meta exist,element=%s pad=%s", GST_ELEMENT_NAME(element), GST_PAD_NAME(pad));
        }else {
            logger_info("meta lost,element=%s pad=%s", GST_ELEMENT_NAME(element), GST_PAD_NAME(pad));

        }

        return GST_PAD_PROBE_OK;
    }


    /**
     *
     * @param element
     * @param pad
     */
    void Metric::install_metric_probe(GstElement *element, std::string pad) {
        int prob_id = 0;
        logger_info("install metric probe for element=%s,pad=%s", GST_ELEMENT_NAME(element), pad.c_str());
        NVGSTDS_ELEM_ADD_PROBE(prob_id, element, pad.c_str(), metric_probe, GST_PAD_PROBE_TYPE_BUFFER, element);
    }

• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

We can not reproduce the problem with just the piece of code. Can you provide a simple source code to reproduce the problem?

YES , but the code is a little long


// meta struct 
class  HttpSrcRequestMeta {

    public:
        gpointer http_src_request;


    };


gpointer http_src_request_meta_copy_func(gpointer data, gpointer user_data) {
        logger_info("http_src_request_meta_copy_func");
        HttpSrcRequestMeta *src_request_meta = (HttpSrcRequestMeta *) data;
        HttpSrcRequestMeta *dst_request_meta = (HttpSrcRequestMeta *) g_malloc0(sizeof(HttpSrcRequestMeta));
        memcpy(dst_request_meta, src_request_meta, sizeof(HttpSrcRequestMeta));
        return (gpointer) dst_request_meta;

    }

    void http_src_request_meta_release_func(gpointer data, gpointer user_data) {
        logger_info("http_src_request_meta_release_func");

        HttpSrcRequestMeta *request_meta = (HttpSrcRequestMeta *) data;
        if (request_meta) {
            g_free(request_meta);
            request_meta = NULL;
        }
    }


    gpointer http_src_request_gst_to_nvds_meta_transform_func(gpointer data, gpointer user_data) {
        logger_info("http_src_request_gst_to_nvds_meta_transform_func");
        NvDsUserMeta *user_meta = (NvDsUserMeta *) data;
        HttpSrcRequestMeta *src_request_meta =
                (HttpSrcRequestMeta *) user_meta->user_meta_data;
        HttpSrcRequestMeta *dst_request_meta =
                (HttpSrcRequestMeta *) http_src_request_meta_copy_func(src_request_meta, NULL);
        return (gpointer) dst_request_meta;
    }

    void http_src_request_gst_nvds_meta_release_func(gpointer data, gpointer user_data) {
        logger_info("http_src_request_gst_nvds_meta_release_func");
        NvDsUserMeta *user_meta = (NvDsUserMeta *) data;
        HttpSrcRequestMeta *request_meta = (HttpSrcRequestMeta *) user_meta->user_meta_data;
        http_src_request_meta_release_func(request_meta, NULL);
    }



// probe utils 

static GstPadProbeReturn metric_probe(GstPad *pad, GstPadProbeInfo *info, gpointer u_data) {
        GstElement *element = (GstElement *) u_data;
        GstBuffer *buf = (GstBuffer *) info->data;

        GstMeta *gst_meta;
        NvDsMeta *dsmeta;
        gpointer state = NULL;
        NvDsBatchMeta *batch_meta = NULL;
        NvDsMetaList *l_frame = NULL;
        NvDsUserMeta *user_meta = NULL;
        NvDsMetaList *l_user_meta = NULL;
        GstMapInfo inmap = GST_MAP_INFO_INIT;

        logger_info("element %s pad %s received,buffer=%p,size=%ld,thread_id=%ld", GST_ELEMENT_NAME(element),
                    GST_PAD_NAME(pad),
                    buf, inmap.size, std::this_thread::get_id());


        gboolean find = false;
        batch_meta = gst_buffer_get_nvds_batch_meta(buf);
        if (batch_meta) {
            logger_info("gst_buffer_get_nvds_batch_meta,element %s pad %s ", GST_ELEMENT_NAME(element),
                        GST_PAD_NAME(pad));
            for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
                 l_frame = l_frame->next) {

                NvDsFrameMeta *frame_meta = (NvDsFrameMeta *) (l_frame->data);

                for (l_user_meta = frame_meta->frame_user_meta_list; l_user_meta != NULL;
                     l_user_meta = l_user_meta->next) {

                    logger_info("frame_user_meta_list,element %s pad %s ", GST_ELEMENT_NAME(element),
                                GST_PAD_NAME(pad));


                    user_meta = (NvDsUserMeta *) (l_user_meta->data);
                    if (user_meta->base_meta.meta_type == JA_HTTP_SRC_REQUEST_META) {
                        find = true;
                        gst_println("OKKKKKKKKKKK,element=%s pad=%s", GST_ELEMENT_NAME(element), GST_PAD_NAME(pad));
                    }
                }
            }
        } else {
            while ((gst_meta = gst_buffer_iterate_meta(buf, &state))) {
                dsmeta = (NvDsMeta *) gst_meta;
                logger_info("meta type:%d", dsmeta->meta_type);
                if (dsmeta->meta_type == JA_HTTP_SRC_REQUEST_META) {
                    find = true;
                    gst_println("OKKKKKKKKKKK--shi,element=%s pad=%s", GST_ELEMENT_NAME(element), GST_PAD_NAME(pad));
                }
            }
        }

        if (find) {
            logger_info("meta exist,element=%s pad=%s", GST_ELEMENT_NAME(element), GST_PAD_NAME(pad));
        }else {
            logger_info("meta lost,element=%s pad=%s", GST_ELEMENT_NAME(element), GST_PAD_NAME(pad));

        }

        return GST_PAD_PROBE_OK;
    }


    /**
     *
     * @param element
     * @param pad
     */
    void Metric::install_metric_probe(GstElement *element, std::string pad) {
        int prob_id = 0;
        logger_info("install metric probe for element=%s,pad=%s", GST_ELEMENT_NAME(element), pad.c_str());
        NVGSTDS_ELEM_ADD_PROBE(prob_id, element, pad.c_str(), metric_probe, GST_PAD_PROBE_TYPE_BUFFER, element);
    }



// bus calback 
gboolean static bus_call_back(GstBus *bus, GstMessage *message, gpointer user_data) {

    logger_info("received message on bus: source %s, msg_type %s",
                GST_MESSAGE_SRC_NAME(message), GST_MESSAGE_TYPE_NAME(message));
}



// pipeline worker thread 
static void worker(GstElement *pipeline) {
    if (gst_element_set_state(pipeline,
                              GST_STATE_PLAYING) == GST_STATE_CHANGE_FAILURE) {
        logger_error("can't set pipeline to playing state");
    }

}

// init pipeline  and add probe callback 
// appsrc ->queue -> jpegdec -> fakesink


static gboolean init_pipeline2(GstElement *pipeline, GstElement **app_src) {

    GstElement *queue, *jpegdec, *fakesink;

    *app_src = gst_element_factory_make("appsrc", "appsrc");

    g_object_set(G_OBJECT (app_src), "caps",
                 gst_caps_new_simple("image/jpeg",
//                                     "format",G_TYPE_STRING,"I420",
                                     "framerate", GST_TYPE_FRACTION, 0, 1,
                                     NULL), NULL);


    g_object_set(G_OBJECT (app_src),
                 "stream-type", 0,
                 "format", GST_FORMAT_TIME, NULL);
    Metric::install_metric_probe(*app_src, "src");


    queue = gst_element_factory_make("queue", "queue");

    Metric::install_metric_probe(queue, "sink");
    Metric::install_metric_probe(queue, "src");

    jpegdec = gst_element_factory_make("jpegdec", "jpegdec");

    Metric::install_metric_probe(jpegdec, "src");
    Metric::install_metric_probe(jpegdec, "sink");


    fakesink = gst_element_factory_make("fakesink", "fakesink");
    Metric::install_metric_probe(fakesink, "sink");

    gst_bin_add_many(GST_BIN(pipeline), *app_src,queue, jpegdec, fakesink, NULL);


    gboolean ret = element_link(*app_src, "src", NULL, 0, queue, "sink", NULL, 0);
    if (!ret) {
        logger_error("link app_src to convert ");
        return false;
    }


    ret = element_link(queue, "src", NULL, 0, jpegdec, "sink", NULL, 0);
    if (!ret) {
        logger_error("link queue to jpeg ");
        return false;
    }


    ret = element_link(jpegdec, "src", NULL, 0, fakesink, "sink", NULL, 0);
    if (!ret) {
        logger_error("link app_src to convert ");
        return false;
    }


}


// main 

int main(int argc, char *argv[]) {

    gst_init(&argc, &argv);
    log_init();


    GMainLoop *main_loop = g_main_loop_new(NULL, FALSE);
    GstBus *bus;
    int bus_id;
    GstElement *pipeline;
    GstElement *app_src;

    pipeline = gst_pipeline_new("test");
    init_pipeline2(pipeline, &app_src);

    bus = gst_pipeline_get_bus(GST_PIPELINE(pipeline));
    bus_id = gst_bus_add_watch(bus, bus_call_back, NULL);


    std::thread t(worker, pipeline);


    std::this_thread::sleep_for(std::chrono::seconds(5));

//
    static GstClockTime timestamp = 0;
    for (;;) {
        GstFlowReturn ret;
        GstBuffer *buffer;
        IplImage *img;
//
        std::ifstream is("../image/image.jpg");
        is.seekg(0, std::ios::end);
        int len = is.tellg();

        is.seekg(0, std::ios::beg);

        char buf[len];
        is.read(buf, len);
        int size = len;
        logger_info("read image len:%d", len);

        buffer = gst_buffer_new_allocate(NULL, size, NULL);
        GstMapInfo info;
        gst_buffer_map(buffer, &info, GST_MAP_WRITE);

        NvDsMeta *meta = NULL;


        HttpSrcRequestMeta *http_src_request_meta = (HttpSrcRequestMeta *) g_malloc0(sizeof(HttpSrcRequestMeta));

        if(http_src_request_meta == NULL)
        {
            return GST_FLOW_ERROR;
        }
        http_src_request_meta->http_src_request = new HttpSrcRequest(NULL, 0);

        meta = gst_buffer_add_nvds_meta(buffer, http_src_request_meta, NULL,
                                        http_src_request_meta_copy_func, http_src_request_meta_release_func);


        meta->meta_type = (GstNvDsMetaType) JA_HTTP_SRC_REQUEST_META;

        meta->gst_to_nvds_meta_transform_func = http_src_request_gst_to_nvds_meta_transform_func;

        meta->gst_to_nvds_meta_release_func = http_src_request_gst_nvds_meta_release_func;



        memcpy((guchar *) info.data, buf, gst_buffer_get_size(buffer));
        GST_BUFFER_PTS (buffer) = timestamp;
        GST_BUFFER_DURATION (buffer) = gst_util_uint64_scale_int(1, GST_SECOND, 2);
        timestamp += GST_BUFFER_DURATION (buffer);

        ret = gst_app_src_push_buffer(GST_APP_SRC(app_src), buffer);
        logger_info("push buffer:%d", ret);
        gst_buffer_unmap(buffer, &info);
        std::this_thread::sleep_for(std::chrono::seconds(5));
    }
    logger_info("start pipeline success.");
    g_main_loop_run(main_loop);
    logger_info("stopping pipeline!");
    gst_element_set_state(pipeline, GST_STATE_NULL);
    logger_info("deleting pipeline");
    gst_object_unref(pipeline);
    g_main_loop_unref(main_loop);

}


Do you mean before nvjpegdec, the meta is available, only in nvjpegdec src pad, the meta is lost?

WHERE and HOW do you add the meta to gst_buf?

How much do you know about gstreamer?

Yes , nvjpegdec src and all element after nvjpegdec both lost .
the pipeline is appsrc-> queue->nvjpegdec-> sink

I add the meta to gst_buf and use gst_app_src_push_buffer method push .

The nvds_meta is designed to mark the specific features for the gst_buf. nvjpegdec is a JPEG decoder plugin, the input gst_buf is the compressed JPEG data and the output gst_buf is the decompressed RAW data, so the output gst_buf should have totally new meta for itself. It will not copy any meta fron the input gst_buf.

The meta is not lost. Just new gst_buf after decoder for new media data. The metadata is data related. You can not use it to transfer no data related information.

How can I get the meta after nvjpegdec element probe callback ?
Or other solution for push an image to appsrc then decode、infer and so on .

If appsrc is a must. You can push decoded RAW data(RGB, YUV,…etc), and the meta can go with them.

Will you use gst-nvinfer to do the inferencing?

YES, appsrc is must, because the input image is through api passed , (http ,rpc or other protocol )

the infer plugin is gst-nvinfer

If you want to use gst-nvinfer, please use nvstreammux to generate the batch meta. Gst-nvstreammux — DeepStream 5.1 Release documentation

The batch meta must be add with RAW data(YUV or RGB). Your usage is totally wrong.

Please refer to the deepstream document.

sorry , I known streammux, my final pipeline is

appsrc → jpegdec → nvvideoconvert → capsfilter(video/x-raw(memory:NVMM)) ->streammux → infer → sink

becuase as mentioned above , jpgdec or nvjpegdec cannot pass the meta, that the reason why I post the question .


The batch meta must be add with RAW data(YUV or RGB). the buffer I added in appsrc is not batch meta , it is simple nvds meta , I also reference demo in deepstream-gst-metadata-test , key of this question is jpegdec ( nvjpegdec ) rebuild buffer .

The suggestion is to push RAW data to appsrc instead of jpeg data to avoid using nvjpegdec or any kind of decoder.

OK , Thank you very much .