Send jpeg encoded object before tiler plugin processes output

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hi. I am trying to complete this task:

  1. Process 8 (eight) 1080p RTSP streams as source
  2. Apply GIE, SGIE and tracker to that
  3. Send the jpeg bytes (and some metadata as the source_id and parent tracker_id) of a particular object detected by the SGIE to a ZMQ endpoint to be analyzed by an external software.
  4. Monitor the detections in real time using the tiler plugin and a single output sink (type=4 / RTSP out using h265 enc).

…and I am almost there, just one step to achieve all that:

I took the deepstream-test5-app as base and modified it. What I am currently struggling to achieve is the 4th step: to stream the ouput using the tiler plugin and RTSP sink.

This is what I’ve done so far:

a) The jpeg bytes are encoded like the app example deepstream_image_meta_test. The encoding process is fired in pgie_src_pad_buffer_probe, calling nvds_obj_enc_process.

b) The “sending to ZMQ” logic (step 3) is called from the “generate_event_msg_meta” called from bbox_generated_probe_after_analytics function (I commented all the msg-broker stuff as I am using ZMQ by myself). That works as expected, in that stage I have everything that I need, the source_id, the tracker_id and the encoded object in the metadata. Si it works.
linked

The problem is that when I activate the tiler, in generate_event_msg_meta I loose the source_id, since the tiler merges everything in a single source (with source_id 0).

As a workaround I tried to define individual sinks to stream every processed source individually using RTP, but that fails as I explained in a separate topic: Error encoding multiple RTSP out

So maybe the question would be, where could I fire my ZMQ send logic, in a step that already has the encoded jpeg object, but before the tiler merges everything?

One more thing. The bbox_generated_probe_after_analytics that does this ZMQ stuff and retrieves the jpeg object is attached like this to the pipeline:

create_pipeline (appCtx[i], NULL, bbox_generated_probe_after_analytics, perf_cb, overlay_graphics);

I changed the order to:

create_pipeline (appCtx[i], bbox_generated_probe_after_analytics, NULL, perf_cb, overlay_graphics);

and that way I have the separate source_id in my generate_event_msg_meta with the tiler still enabled, but this way I cannot retrieve the jpeg bytes in that logic (it seems the encoded object is not there in the metadata yet at that stage of the pipeline).

Any help or suggestions please? Thanks!

• Hardware Platform (Jetson / GPU) dGPU 2080Ti
• DeepStream Version 5.0 (NVIDIA NGC 20.07)
• JetPack Version (valid for Jetson only) N/A
• TensorRT Version N/A
• NVIDIA GPU Driver Version (valid for GPU only) 450.66

3

Can you upload your config file so that we can know your pipeline?

where did you put “pgie_src_pad_buffer_probe”? With which plugin’s which pad? The bbox_generated_probe_after_analytics is called in analytics_done_buf_prob(), analytics_done_buf_prob() is the analytics src pad callback which is downstream of nvinfer.

5

Hello, sorry for the delay. I’ve ended implementing a workaround, which was to store the source id of the detection as user metadata, and then retrieve it later when I send the encoded JPEG through ZMQ. My config files and the source changed too much, so I think they won’t help now. You may close the thread, still it would be nice to know how could we get a source id to send from generate_event_msg_meta method using the tiler plugin at the same time. Thanks,

6

hello @horacio.vico
How did you get jpeg frame and send it to ZMQ/kafka?

2 Likes