Using /x-raw(memory:NVMM) in gstreamer program

I am trying to see if encoding from our camera using monochome videois faster than RGB (Seems like it would be). Problem is, I can’t get my gstreamer program to work the same way that a gst-launch-1.0 script works. Here’s the crux of it:

My gst-launch-1.0 pipeline looks like this, and runs fine, creating a monochrome video mp4 file:

gst-launch-1.0 videotestsrc !
‘video/x-raw, format=(string)GRAY8, width=(int)1280, height=(int)720’ !
nvvidconv !
‘video/x-raw(memory:NVMM),format=(string)I420’ !
omxh264enc bitrate=10000000 !
‘video/x-h264, stream-format=(string)byte-stream’ !
h264parse !
qtmux !
filesink location=test.mp4 -e

Note the ‘video/x-raw(memory:NVMM)’ caps filter after the nvvidconv. This appears to be important to making this run fast.

Now, I tried to reproduce this pipeline in a program so that I could hook this up to my appsrc (I can’t use v4l2src because my camera isn’t a v4l2 camera, so I wrote an appsrc to grab from it). When I try to make the nvvidconv filter and the caps filter after it, my code looks like this:

    gstreamerAppSrc.convert	= gst_element_factory_make ("nvvidconv", "nvvidconv1");
gstreamerAppSrc.convFilter = gst_element_factory_make ("capsfilter", "convfilter");


/* caps for output of nvvidconv: */
gstreamerAppSrc.convFilterCaps = gst_caps_new_simple (
	"video/x-raw(memory:NVMM)",
	"format", G_TYPE_STRING, "I420",	
	NULL);
	
if (!gstreamerAppSrc.convFilterCaps) {
	g_printerr( "convFilterCaps could not be created. Exiting.\n");
	return -1;
}

g_object_set (G_OBJECT (gstreamerAppSrc.convFilter), "caps", gstreamerAppSrc.convFilterCaps, NULL);
gst_caps_unref (gstreamerAppSrc.convFilterCaps);

This builds fine, but when I run the pipeline, it gives me this error:

(gst-app-src:18367): GStreamer-CRITICAL **: gst_structure_new_empty: assertion ‘gst_structure_validate_name (name)’ failed
convFilterCaps could not be created. Exiting.

I think it’s choking on “/x-raw(memory:NVMM)”, which gst-launch handles just fine. Do I need to escape some characters in that caps string? Anybody gotten that to work?

It’s just not working? As far as I remember, I came across this message when using memory: NVMM, but everything worked

Hello, realdealneil:
This clip works in my side.

GstCaps *videosrc_caps = gst_caps_new_simple("video/x-raw",
        "format", G_TYPE_STRING, "GRAY8",
        "width", G_TYPE_INT, 1280,
        "height", G_TYPE_INT, 720,
        NULL);
    GstCaps *vid_caps = gst_caps_new_simple("video/x-raw(memory:NVMM)",
        "format", G_TYPE_STRING, "I420",
        NULL);
    if (!gst_element_link_filtered(videosrc, vidconv, videosrc_caps)) {
        g_printerr("Fail to gst_element_link_filtered videosrc -- vidconv\n");
        return -1;
    }
    gst_caps_unref(videosrc_caps);

    if (!gst_element_link_filtered(vidconv, videoenc, vid_caps)) {
        g_printerr("Fail to gst_element_link_filtered vidconv -- videoenc\n");
        return -1;
    }
    gst_caps_unref(vid_caps);

br
ChenJian

It looks like I was treating a warning as an error and exiting out early. I also appear to get the same performance whether I use (memory:NVMM) or not, so maybe this wasn’t as big a deal as I thought. Thanks for the help, Jachen, and sv_91_!

This problem can be critical in Tx1

I made a gstreamer-1.0 (Version 1.6) based recording application on Tx1

if the string parameter including (memory:NVMM) postfix (e.g. “video/x-raw(memory:NVMM)”) is passed into

gst_caps_new_simple API, then it just return the NULL GstCaps* with the assertion message ( GStreamer-CRITICAL **: gst_structure_new_empty: assertion ‘gst_structure_validate_name (name)’ failed)

Thus, must use the get_caps_from_string(gchar*) instead of gst_caps_new_simple

Thanks for clue gudbooy!

I was trying to make hardware accelerated output, so struggled with setting caps filter between nvvidconf and nvoverlaysink on TX1 for a day or two, nothing helped except gst_caps_from_string(“video/x-raw(memory:NVMM),format=I420”)

Worked for me too! Though I’m not clear on why gst_caps_new_simple fails

Work for me too !
but use gst_caps_from_string not get_caps_from_string misspelling in #5

Sample :

GstCaps *capsnvvidconv; // nv omxh264enc only support I420 or NV12 as input formats see doc	
capsnvvidconv = gst_caps_from_string("video/x-raw(memory:NVMM), width=(int)640, height=(int)480, format=(string)I420");
/*problem with this code :
capsnvvidconv = gst_caps_new_simple("video/x-raw(memory:NVMM)",
		"format", G_TYPE_STRING, "I420",
		"width", G_TYPE_INT, m_width / 2,
		"height", G_TYPE_INT, m_height / 2,
			NULL);*/

This would appear to be a longstanding bug in the nvarguscamerasrc gstreamer plugin.

Has NVIDIA listed it as such?
do they intend to fix it?

here is the relevant part of the test

<b>// connects to rpi camera in mode 4 as intended</b>

//  gst-launch-1.0 nvarguscamerasrc  ! 'video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, framerate=(fraction)120/1' ! nvoverlaysink

#if 0
  // <b>connects to rpi camera in mode 4 as intended</b>
  caps = gst_caps_from_string("video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, framerate=(fraction)120/1");
#else
// <b>recieves the following errors</b>
//(gsttest:8868): GStreamer-CRITICAL **: 00:19:52.400: gst_structure_new_empty: assertion 'gst_structure_validate_name (name)' failed
//** (gsttest:8868): CRITICAL **: 00:19:52.400: void print_caps(const GstCaps*, const gchar*): assertion 'caps != NULL' failed
//(gsttest:8868): GStreamer-CRITICAL **: 00:19:52.401: gst_mini_object_unref: assertion 'mini_object != NULL' failed

  <b>// connects to rpi camera in mode 2 not as intended</b>
  caps = gst_caps_new_simple ("video/x-raw(memory:NVMM)",
          "width", G_TYPE_INT, 1280,
          "height", G_TYPE_INT, 720,
          "format", G_TYPE_STRING, "NV12",
          "framerate", GST_TYPE_FRACTION, 120, 1,
          NULL);
#endif

[b]// caps is NULL in the debugger when using gst_caps_new_simple
// and nothing is printed when using gst_caps_new_simple[/b]
    print_caps (caps, "      ");

Full compileable file.
change the define @ line 120 to see bug.
tested on jetson nano with rPi camera.

#include <gst/gst.h>
//  gcc -g gsttest.cpp -o gsttest `pkg-config --cflags --libs gstreamer-1.0`
//  gst-launch-1.0 nvarguscamerasrc  ! 'video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, framerate=(fraction)120/1' ! autovideosink
// https://devtalk.nvidia.com/default/topic/934515/using-x-raw-memory-nvmm-in-gstreamer-program/

// gst-launch-1.0 nvarguscamerasrc ! "video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)120/1" ! nvoverlaysink -e

static gboolean
link_elements_with_filter (GstElement *element1, GstElement *element2);

int main(int argc, char *argv[]) {
  GstElement *source,
        *sink,
        *pipeline;

  GstBus *bus;
  GstMessage *msg;
  GstStateChangeReturn ret;

  /* Initialize GStreamer */
  gst_init (&argc, &argv);

  /* Create the elements */
#if 0
  source = gst_element_factory_make ("videotestsrc", "source");
  g_object_set (source, "pattern", 0, NULL);
#else
    source = gst_element_factory_make ("nvarguscamerasrc", "source");  
#endif

    sink = gst_element_factory_make ("nvoverlaysink", "sink");
 
  /* Create the empty pipeline */
  pipeline = gst_pipeline_new ("test-pipeline");
 
  if (!pipeline || !source || !sink ) {
    g_printerr ("!pipeline || !source || !sink \n");
    return -1;
  }
  
  /* Build the pipeline */
  gst_bin_add_many (GST_BIN (pipeline), source, sink, NULL);
    
  if(link_elements_with_filter(source,sink) != TRUE){  
    g_printerr ("Elements source,sink could not be linked.\n");   
    gst_object_unref (pipeline);
    return -1;
  }

  /* Start playing left*/
  ret = gst_element_set_state (pipeline, GST_STATE_PLAYING);
  if (ret == GST_STATE_CHANGE_FAILURE) {
    g_printerr ("Unable to set the pipeline to the playing state.\n");
    gst_object_unref (pipeline);
    return -1;
  }

  /* Wait until error or EOS */
  bus = gst_element_get_bus (pipeline);
  msg = gst_bus_timed_pop_filtered (bus, GST_CLOCK_TIME_NONE, (GstMessageType)(GST_MESSAGE_ERROR | GST_MESSAGE_EOS));

  /* Parse message */
  if (msg != NULL) {
    GError *err;
    gchar *debug_info;

    switch (GST_MESSAGE_TYPE (msg)) {
      case GST_MESSAGE_ERROR:
        gst_message_parse_error (msg, &err, &debug_info);
        g_printerr ("Error received from element %s: %s\n", GST_OBJECT_NAME (msg->src), err->message);
        g_printerr ("Debugging information: %s\n", debug_info ? debug_info : "none");
        g_clear_error (&err);
        g_free (debug_info);
        break;
      case GST_MESSAGE_EOS:
        g_print ("End-Of-Stream reached.\n");
        break;
      default:
        /* We should not reach here because we only asked for ERRORs and EOS */
        g_printerr ("Unexpected message received.\n");
        break;
    }
    gst_message_unref (msg);
  }

  /* Free resources */
  gst_object_unref (bus);
  gst_element_set_state (pipeline, GST_STATE_NULL);
  gst_object_unref (pipeline);
  return 0;
}

#if 0
Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw(memory:NVMM)
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
                 format: { (string)NV12 }
              framerate: [ 0/1, 120/1 ]

#endif

static void print_caps (const GstCaps * caps, const gchar * pfx);

static gboolean
link_elements_with_filter (GstElement *element1, GstElement *element2)
{
  gboolean link_ok;
  GstCaps *caps;

// connects to rpi camera in mode 4 as intended
//  gst-launch-1.0 nvarguscamerasrc  ! 'video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, framerate=(fraction)120/1' ! nvoverlaysink

#if 0
  // connects to rpi camera in mode 4 as intended
  caps = gst_caps_from_string("video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, framerate=(fraction)120/1");
#else
// recieves the following errors
//(gsttest:8868): GStreamer-CRITICAL **: 00:19:52.400: gst_structure_new_empty: assertion 'gst_structure_validate_name (name)' failed
//** (gsttest:8868): CRITICAL **: 00:19:52.400: void print_caps(const GstCaps*, const gchar*): assertion 'caps != NULL' failed
//(gsttest:8868): GStreamer-CRITICAL **: 00:19:52.401: gst_mini_object_unref: assertion 'mini_object != NULL' failed

  // connects to rpi camera in mode 2 not as intended
  caps = gst_caps_new_simple ("video/x-raw(memory:NVMM)",
          "width", G_TYPE_INT, 1280,
          "height", G_TYPE_INT, 720,
          "format", G_TYPE_STRING, "NV12",
          "framerate", GST_TYPE_FRACTION, 120, 1,
          NULL);
#endif

// caps is NULL in the debugger when using gst_caps_new_simple
// and nothing is printed when using gst_caps_new_simple
    print_caps (caps, "      ");

  link_ok = gst_element_link_filtered (element1, element2, caps);
  gst_caps_unref (caps);

  if (!link_ok) {
    g_warning ("Failed to link element1 and element2!");
  }

  return link_ok;
}

/* Functions below print the Capabilities in a human-friendly format */
static gboolean print_field (GQuark field, const GValue * value, gpointer pfx) {
  gchar *str = gst_value_serialize (value);

  g_print ("%s  %15s: %s\n", (gchar *) pfx, g_quark_to_string (field), str);
  g_free (str);
  return TRUE;
}

static void print_caps (const GstCaps * caps, const gchar * pfx) {
  guint i;

  g_return_if_fail (caps != NULL);

  if (gst_caps_is_any (caps)) {
    g_print ("%sANY\n", pfx);
    return;
  }
  if (gst_caps_is_empty (caps)) {
    g_print ("%sEMPTY\n", pfx);
    return;
  }

  for (i = 0; i < gst_caps_get_size (caps); i++) {
    GstStructure *structure = gst_caps_get_structure (caps, i);

    g_print ("%s%s\n", pfx, gst_structure_get_name (structure));
    gst_structure_foreach (structure, print_field, (gpointer) pfx);
  }
}

Hopefully this post gets a bit more visibility, I’ve spent a couple of trying to determine what this issue is.