Unable to retrieve source_id in frame_meta, hence unable to distinguish which stream is generating what output

Setup Info

• Hardware Platform (Jetson / GPU) - GPU
• DeepStream Version - deepstream:5.0-20.07-devel
• TensorRT Version - Haven’t changed the default that comes with container deepstream:5.0-20.07-devel
• NVIDIA GPU Driver Version (valid for GPU only) - 440.33.01
• Sample application and the configuration file content - …/sample_apps/deepstream-nvdsanalytics-test/deepstream_nvdsanalytics_test.cpp

• Reproduce steps
I was trying to save the cropped image being generated after analytics using gst-dsexample plugin. Want to change filename of image being saved so that in multi-stream scenario it can be identified which image was generated by which stream. For this, was trying to access “source_id” obtained from “NvDsFrameMeta” . Issue is, that the frame_meta is always coming as zero.

I checked few other samples as well, which prints stream-id on console, however on looking into the code it was observed that they are actually printing “pad_index”. Are pad_index and source_id same and can be used interchangeably ?

I followed these link1, link2, which seems to be having similar issue, but couldn’t conclude what can be resolution to this issue.

I am accessing frame_meta like this, basically that’s the default code which comes along:
image

Lastly, FYI, I made certain change to deepstream-app and inserted dsexample-plugin after osd-plugin so that I could capture the BBoxes in saved image. But, I believe that should not be the problem.

This is a major blocker for us. Hoping to get some insights soon.

Thanks

So you have modified the codes of deepstream_nvdsanalytics_test.cpp since there is no dsexample in it.

Can you tell us your new pipeline?

@Fiona.Chen I used the another makefile provided by name of “Makefile.ds” which builds nvdsanalytics on top of deepstream-app. And then, I made changes to deepstream-app itself and brought ds-example plugin inserted after OSD element. No major changes to the pipeline except that.

A follow-up question, I was looking this thread and it’s suggested to map pad_index to source_id. Can you please elaborate on this ?

It is correct to “map pad_index to source_id” but I’m not sure if it can meet your case since you have not provided the whole pipeline.

Since you are using deepstream-app, what is your config file? Can you upload the config file? I need these information to know what is happening.

@Fiona.Chen please find the config file below:

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=0
rows=5
columns=6
width=1280
height=720
gpu-id=0
#(0): nvbuf-mem-default - Default memory allocated, specific to particular platform
#(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory, applicable for Tesla
#(2): nvbuf-mem-cuda-device - Allocate Device cuda memory, applicable for Tesla
#(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory, applicable for Tesla
#(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
nvbuf-memory-type=0

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
uri=file:///opt/nvidia/deepstream/deepstream-5.0/samples/streams/traffic.mp4
num-sources=1
camera-id=0
#drop-frame-interval=2
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

#[source1]
#enable=1
##Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
#type=3
#uri=file://../../streams/sample_1080p_h264.mp4
#num-sources=1
#gpu-id=0
## (0): memtype_device   - Memory type Device
## (1): memtype_pinned   - Memory type Host Pinned
## (2): memtype_unified  - Memory type Unified
#cudadec-memtype=0

[source1]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
uri=file://../../streams/sample_1080p_h264.mp4
num-sources=1
camera-id=1
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File
type=2
sync=1
source-id=0
gpu-id=0
nvbuf-memory-type=0

[sink1]
enable=0
type=3
#1=mp4 2=mkv
container=1
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=0
#iframeinterval=10
bitrate=2000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
output-file=out.mp4
source-id=0

[sink2]
enable=0
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=1
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=0
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
# set below properties in case of RTSPStreaming
rtsp-port=8554
udp-port=5400


[osd]
enable=1
gpu-id=0
border-width=1
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=0
batch-size=30
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
## Set muxer output width and height
width=1920
height=1080
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0
## If set to TRUE, system timestamp will be attached as ntp timestamp
## If set to FALSE, ntp timestamp from rtspsrc, if available, will be attached
# attach-sys-ts-as-ntp=1

# config-file property is mandatory for any gie section.
# Other properties are optional and if set will override the properties set in
# the infer config file.
[primary-gie]
enable=1
gpu-id=0
model-engine-file=../../models/Primary_Detector/resnet10.caffemodel_b30_gpu0_int8.engine
#Required to display the PGIE labels, should be added even when using config-file
#property
batch-size=30
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=0
#Required by the app for SGIE, when used along with config-file property
gie-unique-id=1
nvbuf-memory-type=0
config-file=config_infer_primary.txt

[tests]
file-loop=0


[nvds-analytics]
enable=1
config-file=/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-nvdsanalytics-test/config_nvdsanalytics.txt

[ds-example]
enable=1
processing-width=1280
processing-height=720
full-frame=1
#batch-size for batch supported optimized plugin
#batch-size=30
unique-id=15
gpu-id=0
  1. There is “source_id” in NvDsFrameMeta, you can use it directly.
    https://docs.nvidia.com/metropolis/deepstream/dev-guide/DeepStream_Development_Guide/baggage/struct__NvDsFrameMeta.html
  2. I’ve tried your config file with deepstream-app, I can get “source_id” inside dsexample.
    The following is what I print out:
    ****+++++frame id in DSEXAMPLE 1
    ****+++++frame id in DSEXAMPLE 1
    ****+++++frame id in DSEXAMPLE 1
    ****+++++frame id in DSEXAMPLE 1
    ****+++++frame id in DSEXAMPLE 1
    ****+++++frame id in DSEXAMPLE 0
    ****+++++frame id in DSEXAMPLE 0
    ****+++++frame id in DSEXAMPLE 0
    ****+++++frame id in DSEXAMPLE 0
    ****+++++frame id in DSEXAMPLE 0
    Just add one line print out in gstdsexample.cppgstdsexample.cpp (34.2 KB)

@Fiona.Chen Attaching my deepstream_app.c file. Could it be because of the changes I made to pipeline that the source_id is no longer being published ?

I had to tweak the pipeline as full-frame image along with boundary-box drawn were a requirement.

deepstream_app.c (51.3 KB)

I’ve tried your deepstream_app.c, it can also print out both stream “source_id”. Everything is OK.
Can the original DeepStream codes work for this case in your device?

Can you change the “traffic.mp4” to our test video? I don’t have your “traffic.mp4” file, so I use our own test videos.

@Fiona.Chen, I tried with other streams as well but same result. I noticed a pattern in the image files being generated:

image

Here’s the code that generates the file:

DsExampleOutput *DsExampleProcess(ImageData *image_data, NvDsFrameMeta* frame_meta)
{
    if(image_data->cvmat) {
        static unsigned int count = 0; 
        cv::imwrite(std::to_string(frame_meta->frame_num) + "_" + std::to_string(frame_meta->source_id) + "_"  + std::to_string(count) + ".jpg", *(image_data->cvmat));
        count++;
    }
}

with counter incrementing on every loop and frame_num being passed by respective stream meta (there are 3 running simultaneously), I was expecting count to surpass frame_num, in multiple of 3 at some point. But that never happened.

Now, I feel like there’s some serious error during pipeline setup. @Fiona.Chen do you also find this erroneous ?

@Fiona.Chen have checked and the pipeline seems to be correct. Traced the buffer flow in nvdsanalytics pipeline and there were some interesting observations. The meta-buffer seems to be traveling in this order :

nvds (parse_nvdsanalytics_meta_data func) --> deepstream-app (process-meta func) --> dsexample (gst_dsexample_transform_ip func). This I obtained by attaching gdb and stepping through the functions manually. In frame_meta ntp seems to be maintained throughout hence took it’s reference to analyse frame_meta. Here are the conclusions which I could draw:

  • Finding says that the buffer travels asynchronously. First the meta appears in “/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-nvdsanalytics-test/deepstream_nvdsanalytics_meta.cpp” in function “parse_nvdsanalytics_meta_data” where it remains intact, stream_id etc remains intact.
  • Then to “/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-app/deepstream_app.c” in function “process_meta” which is a probe meta handler attached to osd for now. Here the frame meta, including stream_id has been mostly distorted. stream_id becomes 0.
  • Finally, it reaches “/opt/nvidia/deepstream/deepstream-5.0/sources/gst-plugins/gst-dsexample/gstdsexample.cpp” in function “gst_dsexample_transform_ip”. The frame meta, as it’s parent, comes distorted. stream_id still 0.

Have attached debugger trail below (See Sno. 11,18 and 48):

The issue was related to erroneous config itself. There were certain blocks which were declared twice. I would suggest removing the blocks entirely and not to just disable them as the last bloack declared with same key is enforced.
Also, for sink block you need to configure source-id without fail.