I was reading through this RTSP in/out solution in this link. Then i want to improve with tracker and MQTT.
deepstream-test2 sample then copy-paste the tracker part of the code, doesn’t it?• Hardware: Jetson Nano
• DeepStream 5.1
• JetPack 4.5.1
• TensorRT Version 7.1.3
I don’t understand your description. After nvstreammux, the multiple input streams are batched together as one batch. It is easy to get the source id in the batch meta. MetaData in the DeepStream SDK — DeepStream 6.3 Release documentation
With nvstreamdemux, the src pad index just match to the source id of the stream.
You can refer to the tracker part of “deepstream-test2”.
Please make sure you are familiar with basic knowldge and coding skills for gstreamer before you start with deepstream. https://gstreamer.freedesktop.org/
For clarification, I use this code then modified it. I also currently learning gstreamer functions and its pipeline. Mind that I primarily use python on here.
For the updates, I’ve already modifed the code using MQTT and tracker. However, for MQTT, I copied the whole osd_sink_pad_buffer_probe function and differentiate them. So it ended up like this:
in main
...
osdsinkpad1 = nvosd[0].get_static_pad("sink")
osdsinkpad2 = nvosd[1].get_static_pad("sink")
if not osdsinkpad1:
sys.stderr.write(" Unable to get sink pad of nvosd \n")
osdsinkpad1.add_probe(Gst.PadProbeType.BUFFER, osd_sink_pad_buffer_probe1, 0)
osdsinkpad2.add_probe(Gst.PadProbeType.BUFFER, osd_sink_pad_buffer_probe2, 0)
...
while in each osd_sink_pad_buffer_probe i placed MQTT program. For the results, the MQTT can publish properly to the broker for now. However, because i placed directly on the probe. The publish has no fixed period which I assume the publish speed is the same as stream FPS (I haven’t tried FPS check on the code, but I won’t discuss the FPS for now).
For the specifics:
def osd_sink_pad_buffer_probe(pad,info,u_data): #i have two different function names with same function but different mqtt topic
frame_number=0
#Intiallizing object counter with 0.
obj_counter = {
PGIE_CLASS_ID_VEHICLE:0,
PGIE_CLASS_ID_PERSON:0,
PGIE_CLASS_ID_BICYCLE:0,
PGIE_CLASS_ID_ROADSIGN:0
}
num_rects=0
gst_buffer = info.get_buffer()
if not gst_buffer:
print("Unable to get GstBuffer ")
return
# Retrieve batch metadata from the gst_buffer
# Note that pyds.gst_buffer_get_nvds_batch_meta() expects the
# C address of gst_buffer as input, which is obtained with hash(gst_buffer)
batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
l_frame = batch_meta.frame_meta_list
while l_frame is not None:
try:
# Note that l_frame.data needs a cast to pyds.NvDsFrameMeta
# The casting is done by pyds.NvDsFrameMeta.cast()
# The casting also keeps ownership of the underlying memory
# in the C code, so the Python garbage collector will leave
# it alone.
frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
except StopIteration:
break
frame_number=frame_meta.frame_num
num_rects = frame_meta.num_obj_meta
l_obj=frame_meta.obj_meta_list
while l_obj is not None:
try:
# Casting l_obj.data to pyds.NvDsObjectMeta
obj_meta=pyds.NvDsObjectMeta.cast(l_obj.data)
except StopIteration:
break
obj_counter[obj_meta.class_id] += 1
try:
l_obj=l_obj.next
except StopIteration:
break
# Acquiring a display meta object. The memory ownership remains in
# the C code so downstream plugins can still access it. Otherwise
# the garbage collector will claim it when this probe function exits.
display_meta=pyds.nvds_acquire_display_meta_from_pool(batch_meta)
display_meta.num_labels = 1
py_nvosd_text_params = display_meta.text_params[0]
# Setting display text to be shown on screen
# Note that the pyds module allocates a buffer for the string, and the
# memory will not be claimed by the garbage collector.
# Reading the display_text field here will return the C address of the
# allocated string. Use pyds.get_string() to get the string content.
py_nvosd_text_params.display_text = "Frame Number={} Number of Objects={} Vehicle_count={} Person_count={}".format(frame_number, num_rects, obj_counter[PGIE_CLASS_ID_VEHICLE], obj_counter[PGIE_CLASS_ID_PERSON])
# Now set the offsets where the string should appear
py_nvosd_text_params.x_offset = 10
py_nvosd_text_params.y_offset = 12
# Font , font-color and font-size
py_nvosd_text_params.font_params.font_name = "Serif"
py_nvosd_text_params.font_params.font_size = 10
# set(red, green, blue, alpha); set to White
py_nvosd_text_params.font_params.font_color.set(1.0, 1.0, 1.0, 1.0)
# Text background color
py_nvosd_text_params.set_bg_clr = 1
# set(red, green, blue, alpha); set to Black
py_nvosd_text_params.text_bg_clr.set(0.0, 0.0, 0.0, 1.0)
# Using pyds.get_string() to get display_text as string
print(pyds.get_string(py_nvosd_text_params.display_text))
pyds.nvds_add_display_meta_to_frame(frame_meta, display_meta)
### i insterted MQTT program here
try:
l_frame=l_frame.next
except StopIteration:
break
### i copied tracker (from deepstream-test2) here
return Gst.PadProbeReturn.OK
Update: I successfully made MQTT interval in the program. The hint is read about common/FPS.py. I copied the class to the program and customize into MQTT publish, so the code is in one program unlike FPS one so it doesn’t need to copy the whole osd_sink_pad_buffer_probe function
More updates, sorry for the late report:
I had done the tracker and also added ROI overcrowd detection. Then, i have a question about latency and buffering. Before I ask, here’s the RSTP camera specification:
I tried using mplayer to visualize the display to measure the latency. I got aprrox. 2 seconds without deepstream application (which means purely from rtsp). Then, when I used my app, it resulted into 20 seconds lag. However, for the MQTT itself it published almost no lags. For deepstream RTSP output specification:
Here are the stats for latency:
| Method | Latency (s) |
| Pure RSTP | 2 |
| Original | 20 |
| Modified | 20 |
Latency aside, I checked again for encoding. I tried to compare h264 and h265. I used mplayer again and vlc from another computer. I’d seen h265 buffers much that the video sometimes greyed out. This one even used queue each pipeline link. Here’s the log while running (mplayer):
[rtsp @ 0x7f88b6ce60]max delay reached. need to consume packet
[rtsp @ 0x7f88b6ce60]RTP: missed 7 packets
V: 15.4 0/ 0 13% 0% 0.0% 0 0
[rtsp @ 0x7f88b6ce60]max delay reached. need to consume packet
[rtsp @ 0x7f88b6ce60]RTP: missed 308 packets
V: 15.5 0/ 0 13% 0% 0.0% 0 0
[rtsp @ 0x7f88b6ce60]max delay reached. need to consume packet
[rtsp @ 0x7f88b6ce60]RTP: missed 2 packets
[hevc @ 0x7f880feab8]Could not find ref with POC 308
V: 16.9 0/ 0 12% 0% 0.0% 0 0
[rtsp @ 0x7f88b6ce60]max delay reached. need to consume packet
[rtsp @ 0x7f88b6ce60]RTP: missed 2 packets
V: 17.3 0/ 0 12% 0% 0.0% 0 0
[rtsp @ 0x7f88b6ce60]max delay reached. need to consume packet
[rtsp @ 0x7f88b6ce60]RTP: missed 87 packets
V: 17.3 0/ 0 12% 0% 0.0% 0 0
The code surrounding the encoding are not modified compared to original code, except the IP address and rtsp protocol.
Here are my concerns so far. I think I need more advice regarding latency and buffering issues.