Using deep stream in custom application

DeepStream is just a bunch of plugins for GStreamer. As long as you understand GStreamer, you’ll be fine, but that itself is a lot of work (that is worth it). To understand GStreamer, just retype the tutorials and read their explanations.

I say “retype” because it’s important to do that in order to learn. If you don’t retype it, you won’t learn. It’s not magic. If you need any help, no matter how simple the problem, ask here or on the GStreamer forums. It is “hard code”, but it’s worth the investment and frustration.

The advantage is speed. DeepStream (and GStreamer) is not friendly, but it is fast, and you’ll learn ways to make it friendlier the more experienced you get. It helps to not write GStreamer in C, but you should still know GStreamer in C because that’s what’s under the hood.

1 Like

Thanks so much, @mdegans @bcao
I want to get the output of tracker in the apps, I want to know say me how many object is there. this number is show in the screen and above each of objects , but I want to access to that number, How I can to access to number?

YW, @LoveNvidia
So, when learning about GStreamer in the tutorials, you’ll eventually get to a tutorial with pad probes. In the example you’re probably referring to, a pad probe function is connected in such a way that on every buffer that flows across that pad, a function is called.

It’s in such a function that the things you’re interested in can be obtained. It’s now possible to do it in Python as well, but the interface looks like C and is frankly sometimes easier to work with in C. It’s worth it to learn. even if you stick with higher level language later, since it’ll make your life easier in the long term understanding GStreamer itself.

If you have any trouble with the tutorials, don’t hesitate to ask, even if it’s a simple thing. From experience, there are some assumptions about knowledge in those tutorials that should not be there, so it’s expected you’ll get stuck. Don’t let that stop you from learning.

Hi @mdegans
Thanks for your advice, you really right, but I don’t like the gstreamer forms, because its form is bad design, and isn’t a place to easy solve the problems and its community is bad in term of answering, I really enjoy from NVIDIA forms, in my opinion, NVIDIA forms is best that I ever seen, I ask any question in mind and give me solution and or at least hint, In Nvidia form is it possible to ask Gstreamer question? Or Do you know other form for gstreamer to be like NVIDIA?

@LoveNvidia

So, everything FOSS is ugly (90s web design and a visceral hatred of JavaScript), but the code quality also tends to be better than propretary. If you want advice on DeepStream specifically, here is probably the best place to ask. Otherwise, if it’s a general GStreamer question, you’re likely to get the best answer on a less aesthetically pleasing forum :)

Also, they can certainly be rude, but that’s also par for the course with FOSS. Really, the best suggestion I can make, if you want to learn GStreamer and DeepStream, is to retype the tutorials and ask if you get stuck (there is breakage, so that will likely happen). It has a steep learning curve, but there really aren’t many (good, mature) alternatives for streaming video or these sorts of analytics. It’s hard, and sometimes it might feel impossible, but stick with it and ask questions and you’ll learn.

1 Like

Thanks so much,
The caps and bins in gstreaemer, what are they?

Bins are a kindf of Element that contain other elements. They’re element groups basically, but you can treat them a a kind of Element. A Pipeline is a kind of Bin inself, actually.

Caps are structures that let GStreamer know whether Pads (on Elements) can be connected to each other. For example, a video source Element might have a src Pad Caps of video/x-raw, and that can only be connected to another video Pad of appropriate direction. Some elements have ANY caps that can accept a connection from anything.

This page probably explains it better and has diagrams explaining the major concepts:
https://gstreamer.freedesktop.org/documentation/application-development/introduction/basics.html?gi-language=c

Thanks so much,
Please keep on the discussion of gstreamer in the thread.

Re: other thread, it’s probably better if it stays in one thread since this is related.

A pipeline is comprised of Elements, which have Pads that connect each other like pieces of equipment are connected via ports. Pads have:

  • Direction: source or sink
  • Caps: what is sent over a pad (video, audio, text, etc).
  • Availability: always, sometimes, request

All of those affect whether an element can be linked via it’s pads. A source cannot be linked to a source, for example, a video paid cannot be linked to an audio pad, and a pad that does not exist because it has not been created or requested cannot be connected to an static pad that always exists.

In this case, the former part (video/x-raw) is the what. The (memory:NVMM) specifies where.

In this case, yes. You can use converter elements to send buffers back and forth if you need to.

A bit more than function. If you look at the source code of an element there are a whole bunch of them related to setup, teardown, property setting, and the business of how to handle buffers that flow through the element. You don’t really need to know any of that, however. You can treat elements like a black boxes and just interact with their publicly exposed properties. (eg. a file source might have a property so you can set/get the source file)

4- Each element only have one sink pad and one source pads? Is it possible to have one more sink/src pads in one element?

Yes, it’s possible to have any number of sources or sinks on a given pad. Sometimes they are always/static pads, which always exist. Sometimes they are “sometimes” pads that are created by the element themselves in reponse to something (eg. a source that might or might not have an audio pad). Sometimes they are request pads that you request from an element. You’ll see examples of all of them if you do the tutorials and it’s proably better explained at the above link.

Thanks so much, @mdegans, I get all of your explain. excellent. If possible and you have free time please keep on this thread for gathering more sharing knowledge.

I don’t get this part :

Availability: always, sometimes, request

Than’s mean, If we set the pad to always, that pad only work with specific caps like: video/-x-raw , and If we set the pad to sometimes, that pad can work sometime with video and sometime with audio and sometime with text and so on, If we set the pad to request, that pad only related to doing request? If so, Is it possible to have two input pad, one be for video and one be for request?

You don’t really need to know any of that,

If I want to add my custom tensor flow deep model(face recognition) is like classification task with 100-d output dimension, Is it possible to put my model instead of secondary classification task is deep stream task in pipeline? If yes, I need one step more is searching the embedded of faces in database after face recognition step, For this part(searching face part),in your opinion, I write custom element for that part and then add to pipeline of deepstream, Or It’s better to get buffers from classification element and do processing on the its outputs in the sink_pad_function_prob way then process with this output like python way in the other function?

my mean is that: If you see this link: It used tiler_sink_pad_buffer_probe for capturing the frames into numpy array, but I want to do such way:

tiler_sink_pad_buffer_probe(.....):
        ......
        search_face(metadata)
       ......
    return .....

seach_face(data):
       // do processing on data

As you know, it’s better to don’t put any processing in the prob function, because the performance of system is drop, But I want to do somethings like python asyncio loop.create_task( search_face(metadata)) in the tiler_sink_pad_buffer_probe function, In your opinion, Is it possible?

That’s what I thought too, but out of about 3-4 questions asked there on nabble, I only got 1 useful answer. You’ll most likely get answers from easy questions, but the annoying thing is that for easy questions you can already read the documentation. When there’s a really nasty gstreamer problem, nobody can answer it.

If we set the pad to always, that pad only work with specific caps like: video/-x-raw , and If we set the pad to sometimes, that pad can work sometime with video and sometime with audio and sometime with text and so on,

I think you’re confusing pad availability (sometimes, always, request) with pad capabilities (audio, video, etc). Pad availaiblity you can’t set, rather it’s a part of an element’s design (you can check with gst-inspect-1.0 followed by an element name)

It’s possible to have two (or more) static, pads, that exist always, two or more request pads, that you create on demand, or two or more sometimes pads, that the element creates (and you handle with an event handler function).

Yeah, I can’t help with that, sorry. No experience. ¯_(ツ)_/¯

Yeah. Then the answer is buried in the source somewhere :)

Thanks, @mdegans,
If you have experience in the deepstream python apps, please answer this question if possible.

@mdegans,
Some of elements in deepstream isn’t exist in gst-inpect-1.0 and they are custom elements for deeepstream, like this:

tiler=Gst.ElementFactory.make(“nvmultistreamtiler”, “nvtiler”)

Q1- I want to know for these elements that isn’t there in gst-inspect-1.0, How I can to access their properties? How I can to access their source code?

Q2- in the DS-python-apps, what’s the lines of 211,212?

Q3- If I want to have difference nvvideoconvert element for difference rtsp sources, How I can to define this? My goal is to define difference ROI(src-crop, dest-crop) property of nvvideoconvert for each RTSP stream?

Gst-inspect should necessarily have any element that you can use if it’s in a path GStreamer looks for. Example:

 $ gst-inspect-1.0 nvmultistreamtiler
Factory Details:
  Rank                     primary (256)
  Long-name                Stream Tiler DS 4.0
  Klass                    Generic
  Description              Tile input multistream buffer into a 2D array
  Author                   NVIDIA Corporation. Post on Deepstream for Tesla forum for any queries @ https://devtalk.nvidia.com/default/board/209/

Plugin Details:
  Name                     nvdsgst_multistreamtiler
  Description              NVIDIA Multistream Tiler plugin
  Filename                 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_multistreamtiler.so
  Version                  5.0.0
  License                  Proprietary
  Source module            nvmultistreamTiler
  Binary package           NVIDIA Multistream Plugins
  Origin URL               http://nvidia.com/

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBaseTransform
                         +----GstNvMultiStreamTiler

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw(memory:NVMM)
                 format: { (string)NV12, (string)RGBA }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
  
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw(memory:NVMM)
                 format: { (string)NV12, (string)RGBA }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SINK: 'sink'
    Pad Template: 'sink'
  SRC: 'src'
    Pad Template: 'src'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "nvmultistreamtiler0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  qos                 : Handle Quality-of-Service events
                        flags: readable, writable
                        Boolean. Default: false
  columns             : Number of columns in the Tiled 2D output
                        flags: readable, writable
                        Unsigned Integer. Range: 1 - 4294967295 Default: 1 
  rows                : Number of rows in the Tiled 2D output
                        flags: readable, writable
                        Unsigned Integer. Range: 1 - 4294967295 Default: 1 
  width               : Width of the tiled output in pixels
                        flags: readable, writable
                        Unsigned Integer. Range: 16 - 4294967295 Default: 1920 
  height              : Height of the tiled output in pixels
                        flags: readable, writable
                        Unsigned Integer. Range: 16 - 4294967295 Default: 1080 
  gpu-id              : Set GPU Device ID
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 0 
  show-source         : ID of the source to be shown. If -1 all the sources will be tiled else only a single source will be scaled into the output buffer.
                        flags: readable, writable
                        Integer. Range: -1 - 2147483647 Default: -1 
  nvbuf-memory-type   : Type of NvBufSurface Memory to be allocated for output buffers
                        flags: readable, writable, changeable only in NULL or READY state
                        Enum "GstNvBufMemoryType" Default: 0, "nvbuf-mem-default"
                           (0): nvbuf-mem-default - Default memory allocated, specific to particular platform
                           (4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
  custom-tile-config  : Specifies individual tile resolution for all involved sources
                        flags: writable
                        Pointer. Write only
    uri_decode_bin.connect("pad-added",cb_newpad,nbin)
    uri_decode_bin.connect("child-added",decodebin_child_added,nbin)

The uridecodebin has sometimes pads, which are created by the element itself. In order to handle this, you need to connect the pad-added signal to a callback with the proper signature as shown. In other words, the first line says “when you get a new pad, call the cb_newpad function with nbin as the third parameter”.

The second line calls a function every time a child is added to uridecodebin. Bins are groups of elements, so they have child elements, and you can attach a function to modify what happens when a child is added. In this case, I believe Nvidia uses it to set some decoder options when the decoder is added to the bin.

So, it sounds like you’ll have to add nvvideoconvert elemnts after your decodebins but before your stream muxer. You can do this in cb_newpad probably. You’ll need to create a new element, add it to the bin (final parameter given to the func), and link it to the stream muxer. You’ll have to use gst-inspect-1.0 to figure out all of necessary properties you must set.

Thanks so much, @mdegans,
Q1- How I can add Deepstream GST to path to show me any plugins of gstreamer?

Q2- Is it possible to don’t use of this for uridecodebin? I want to know there are necessary id for uridecodebin element or optional?

uri_decode_bin.connect("pad-added",cb_newpad,nbin)
uri_decode_bin.connect("child-added",decodebin_child_added,nbin) 

Q3- Is it possible to have another than args in cb_newpad and decodebin_child_added functions? In this sample code, these sunction have these args cb_newpad(decodebin, decoder_src_pad,data) and decodebin_child_added(child_proxy,Object,name,user_data), I want to know these args are predined fo this function or we can pass other args to these function? If so, How there args are passed via uri_decode_bin.connect to these functions?

Q4- For last suggestion please show me with snippet code in this link, thanks.

Q5- All of elements can to have get_request_pad property?

Deepstream plugins should already be in the gstreamer plugin path.

The normal way to link withuridecodebin is with callbacks like this, however if you create a bin using gst_parse_bin_from_description (and friends) and the supplied string contains uridecodebin, it will link for you automatically as it would with gst-launch, but you’ll have less control.

The last argument can be anything you want. It’s a void* in C. In Python you can put anything there (I believe even multiple additional arguments). In Vala or Python, if the callback is a method, this or self respectively is implicitly passed, so that’s often easier than manually supplying a third argument.

Line 160 is (probably) the callback to link you’re looking for.

No, when you run gst-inspect-1.0, it will tell you whether an element has sometimes, request, or always (static) pads. You use get_static_pad with always pads, get_request_pad with request pads, and connect to pad-added to link with sometimes pads.

1 Like

@mdegans, Hi,
Q1- Some plugins of nvidia like tracker/streammux , … that are used in deepstream sdk, and these are located in the :

/usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream

I want to know these plugins are installed when we install gstreamer packages or installed when we install deepstream sdk? I want to know these plugins are registered publicly in gstreamer or these are comes with install deepstream sdk?

Q2- Suppose I want to write a custom gstreamer plugin in python like blue the image and I one solution is that convert the buffer to numpy array image and then some processing on, like this:

def do_transform_ip(self, buffer: Gst.Buffer) -> Gst.FlowReturn:
        try:
            # convert Gst.Buffer to np.ndarray
            image = gst_buffer_with_caps_to_ndarray(buffer, self.sinkpad.get_current_caps())

            # apply gaussian blur to image
            image[:] = gaussian_blur(image, self.kernel_size, sigma=(self.sigma_x, self.sigma_y))
        except Exception as e:
            logging.error(e)

        return Gst.FlowReturn.OK

I want to know, Is there a way to have not convert to numpy array and do processing on buffer data directly, I want to know converting data from buffer to numpy has overhead? If buffer located in the NVVM buffer, I have to do cuda programming codes for processing the signal?

Q3- In the above example, we know the buffer is image and I convert to numpy array with function, I want to know If we do inference deep models or trackers in deep stream. How they handle the signals? The image and bounding boxes of objects located in the buffer or each of buffers are separated when data flows in pipeline?

Q4-

It’s possible to have two (or more) static , pads, that exist always, two or more request pads, that you create on demand, or two or more sometimes pads, that the element creates (and you handle with an event handler function).

For sometimes pads that are created by self-element, and these are dynamic pads, I want to know in the pipeline these pads, Is it possible to open and close the port several times during the streaming process? My mean is that when the sometimes pad is created, then this pad is always exist? or in the vary times this pad is created and removed? only one time this pad created?

Q5-

GStreamer handles multithreading automatically,
In the multi-threading of python due to GIL issue, the multi task not perform really as parallel , but in the c/c++ the multi-threading perform parallel, right?
I want to know gstreamer python api, also has GIL problem?