Gstreamer Pipeline freezes while seeking because of omxh264dec element.

I am using the below pipeline:

“filesrc location=test.avi ! avidemux ! video/x-h264 ! tee name=t ! queue ! h264parse ! video/x-h264, stream-format=byte-stream, alignment=au ! appsink name=sinkRaw sync=false max-buffers=50 drop=false t. ! queue ! h264parse ! omxh264dec ! nvvidconv ! video/x-raw, format=RGBA ! appsink name=sinkDec sync=false max-buffers=50 drop=false”

While seeking, sometimes the pipeline freezes and the logs show below line continuously.

videodecoder gstvideodecoder.c:3321:gst_video_decoder_decode_frame: decoder frame list getting long: 17 frames,possible internal leaking?

After searching at various forums, I understand that this can be solved by using the newer version of gst-omx but I am unable to install the gst-omx plugin to work with latest gstreamer version 1.14.

I read at https://devtalk.nvidia.com/default/topic/1031320/gst-omx-for-latest-gstreamer-version-1-14/ that upgrading gstreamer revision is uncertain for jetson tx1.

So will I be able to solve this problem with gst-omx plugin version 1.0.0?

Hirastogishubham20,
Do you also observe the issue in using avdec_h264?

I have tested the pipeline with avdec_h264 element and seek is working fine. The pipeline doesn’t freeze at all.
So I’m guessing the issue is with omxh264dec element only.

But since in my application I have to do hardware decoding, so I cannot use avdec_h264 as an alternative.

Hi rastogishubham20,
Please share us a test code and steps to build/run it so that we can reproduce the issue.

Please find below the test code

/**
 * Author : shubhamr
 * Compile : g++ -g testSeek.cpp -o testSeek `pkg-config --cflags --libs gstreamer-1.0 gstreamer-app-1.0` 
 *
 * Brief Description : Grab raw and decoded data from h264 video after seeking to a particular frame and then 
 *                     create video from raw data and images from decoded data.
 *
 * Stepwise decription of this program:
 * 1. Create a pipeline containing two parallel pipelines 
 * 2. In one part of the pipeline, I am simply parsing the video using h264parse to get raw data.
 * 3. In second part of the pipeline, I am decoding the data using omxh264dec and converting it into Gray.
 * 4. Get preroll buffer from appsink of decoded part of the pipeline to get framerate, width and height.
 *
 * Funcion Description:
 * getRawFrame()     :: Called to pull sample from RAW appsink and write in a .avi file.
 * getDecodedFrame() :: Called to pull sample from DEC appsink and write in a .ppm file.
 * seek()            :: Called to seek to the given frame number.
 */

#include <iostream>
#include <stdlib.h>
#include <string.h>
#include <string>
#include <unistd.h>
#include <fstream>

#include <gst/gst.h>
#include <gst/app/gstappsink.h>

using namespace std;

//Global variables
GstElement *g_gstPipeline, *g_gstSinkRaw, *g_gstSinkDec;

int g_imgWidth, g_imgHeight;

int64_t g_timeBase;



//pull data from appSinkRaw
int getRawFrame(unsigned char *data)
{
  int frameSize = -1;
  GstSample *sample = gst_app_sink_pull_sample(GST_APP_SINK(g_gstSinkRaw));

  GstBuffer *buffer = gst_sample_get_buffer (sample);
  if (!buffer)
  {
    cout << "GStreamer :: ERROR :: Failed to pull raw buffer." << endl;
    return -1;
  }

  GstMapInfo map;
  if (gst_buffer_map (buffer, &map, GST_MAP_READ)) 
  {
    if (map.data && data)
    {
      memcpy (data, map.data, map.size);
      frameSize = map.size;       
      cout << "raw pts :: " << buffer->pts << endl;
      cout << "raw dts :: " << buffer->dts << endl;

    }
  }

  gst_buffer_unmap(buffer, &map);
  // gst_buffer_unref(buffer);
  //usleep(20000);
  gst_sample_unref (sample);

  return frameSize;
}

//pull data from appSinkDec
int getDecodedFrame(unsigned char *data)
{
  int frameSize = -1;
  GstSample *sample = gst_app_sink_pull_sample(GST_APP_SINK(g_gstSinkDec));

  GstBuffer *buffer = gst_sample_get_buffer (sample);
  if (!buffer)
  {
    cout << "GStreamer :: ERROR :: Failed to pull raw buffer." << endl;
    return -1;
  }

  GstMapInfo map;
  if (gst_buffer_map (buffer, &map, GST_MAP_READ)) 
  {
    if (map.data && data)
    {
      memcpy (data, map.data, map.size);
      frameSize = map.size; 
    }
  }

  gst_buffer_unmap(buffer, &map);
  // gst_buffer_unref(buffer);
  //usleep(20000);
  gst_sample_unref (sample);

  return frameSize;
}


int seek(int frameNum)
{   
  //Converting frameNum to time
  int64_t seekPos = g_timeBase * frameNum;

  cout << "seekPos :: " << seekPos << endl;

  if (!gst_element_seek (g_gstPipeline, 1.0, GST_FORMAT_TIME, (GstSeekFlags)(GST_SEEK_FLAG_FLUSH | GST_SEEK_FLAG_KEY_UNIT | GST_SEEK_FLAG_SEGMENT), GST_SEEK_TYPE_SET, seekPos, GST_SEEK_TYPE_NONE, -1))
  {
    cout << "GStreamer :: ERROR :: Seeking Failed." << endl;
    return -1;
  }

  GstStateChangeReturn ret = gst_element_get_state (g_gstPipeline, NULL, NULL, 5 * GST_SECOND);
  if (ret == GST_STATE_CHANGE_FAILURE)
  {
    cout << "GStreamer :: ERROR :: Failed to play the stream after Seeking." << endl;
    return -1;
  }

  cout << "GStreamer :: Seek successful.\n";
  return 0;
}


int main(int argc, char **argv)
{
  if(argc != 2)
  {
    cout << "Usage :: " << argv[0] << " AviFilePath \n";
    return -1;
  }

  gst_init (NULL, NULL);

  string filePath(argv[1]);

  //Creating pipeline
  gchar *descr = g_strdup_printf ("filesrc location=%s ! avidemux ! video/x-h264 ! tee name=t ! queue ! h264parse ! video/x-h264, stream-format=byte-stream, alignment=au ! appsink name=sinkRaw sync=false max-buffers=50 drop=false t. ! queue ! h264parse ! omxh264dec ! nvvidconv ! video/x-raw, format=GRAY8 ! appsink name=sinkDec sync=false max-buffers=50 drop=true", filePath.c_str());
 
  //This pipeline works fine in seeking. In this i am using avdec_h264 instead of omxh264dec
  //gchar *descr = g_strdup_printf ("filesrc location=%s ! avidemux ! video/x-h264 ! tee name=t ! queue ! h264parse ! video/x-h264, stream-format=byte-stream, alignment=au ! appsink name=sinkRaw sync=false max-buffers=50 drop=false t. ! queue ! h264parse ! avdec_h264 ! videoconvert ! video/x-raw, format=GRAY8 ! appsink name=sinkDec sync=false max-buffers=50 drop=true", filePath.c_str());

  GError *error = NULL;

  g_gstPipeline = gst_parse_launch (descr, &error);

  if (error != NULL) 
  {
    g_print ("GStreamer :: ERROR :: Could not construct pipeline: %s\n", error->message);
    g_clear_error (&error);
    exit (-1);
  }

  //Create the elements g_gstSinkRaw and g_gstSinkDec from the pipeline.
  g_gstSinkRaw = gst_bin_get_by_name (GST_BIN (g_gstPipeline), "sinkRaw");
  if ( !g_gstSinkRaw )
  {
    g_print ("GStreamer :: ERROR :: No element named sinkRaw found in the pipeline.\n");
    exit (-1);
  }

  g_gstSinkDec = gst_bin_get_by_name (GST_BIN (g_gstPipeline), "sinkDec");
  if ( !g_gstSinkDec )
  {
    g_print ("GStreamer :: ERROR :: No element named sinkDec found in the pipeline.\n");
    exit (-1);
  }

  //setting the pipeline to play.
  GstStateChangeReturn ret = gst_element_set_state (g_gstPipeline, GST_STATE_PLAYING);

  switch (ret) 
  {
    case GST_STATE_CHANGE_FAILURE:
      g_print ("GStreamer :: ERROR :: failed to play the stream\n");
      exit (-1);

    case GST_STATE_CHANGE_NO_PREROLL:
      g_print ("GStreamer :: ERROR :: Element successfully changed its state but is not able to provide data\n");
      exit (-1);

    default:
      break;
  }


  //Checking if the pipeline state is changed successfully.
  ret = gst_element_get_state (g_gstPipeline, NULL, NULL, 5 * GST_SECOND);
  if (ret == GST_STATE_CHANGE_FAILURE) 
  {
    g_print ("GStreamer :: ERROR :: failed to play the stream\n");
    exit (-1);
  }

  //get the preroll buffer from decoded appsink
  GstSample *sampleDec = NULL;
  g_signal_emit_by_name (g_gstSinkDec, "pull-preroll", &sampleDec, NULL);

  if (sampleDec)
  {
    GstCaps *caps = gst_sample_get_caps (sampleDec);
    if (!caps) 
    {
      g_print ("GStreamer :: ERROR :: could not get decoded snapshot format\n");
      exit (-1);
    }

    GstStructure *s = gst_caps_get_structure (caps, 0);

    // Get the final caps on the buffer to get the size and framerate
    // framerate is required to convert the requested frame number in seek function to time 
    gboolean res;
    res = gst_structure_get_int (s, "width", &g_imgWidth);
    res |= gst_structure_get_int (s, "height", &g_imgHeight);

    const GValue *val = gst_structure_get_value (s, "framerate");
    if (val) 
    {
      gint fpsNum = gst_value_get_fraction_numerator (val);
      gint fpsDen = gst_value_get_fraction_denominator (val);
      cout << "GStreamer :: ******* fps " << fpsNum << " " << fpsDen << endl;

      g_timeBase = ( (fpsDen) * GST_SECOND ) / fpsNum * 2; //for h264 multiplying by 2
    }
    if (!res) 
    {
      g_print ("GStreamer :: ERROR :: could not get decoded snapshot dimension\n");
      exit (-1);
    }
    gst_sample_unref (sampleDec);
  } 
  else 
  {
    g_print ("GStreamer :: ERROR :: could not make decoded snapshot\n");
    exit(-1);
  }

  //Allocate memory for raw and decoded data
  unsigned char *dataRaw = new unsigned char[g_imgWidth*g_imgHeight];
  unsigned char *dataDec = new unsigned char[g_imgWidth*g_imgHeight];


  //Creating object of ofstream to write the video file using raw data.
  ofstream vidFile("output.avi", ios::binary);

  //Creating a loop in which I call functions getRawFrame and getDecodedFrame and correspondingly write data to video file and image file.
  //Image is written for each tenth iteration.
  //I am also calling seek function at each iteration just to check at which frame seeking is causing the pipeline to freeze.
  int i = 0;
  while(1)
  {
    int frameSizeRaw = getRawFrame(dataRaw);
    int frameSizeDec = getDecodedFrame(dataDec);
    cout << "FrameNum :: " << i << " Raw Frame size :: " << frameSizeRaw << endl;
    cout << "FrameNum :: " << i << " Dec Frame size :: " << frameSizeDec << endl;

    //write to .avi file
    vidFile.write(reinterpret_cast<const char*> (dataRaw), frameSizeRaw);

    //save image to ppm
    if(i % 10 == 0)
    {
      char loc[512];
      sprintf(loc, "dump/img%04d.ppm", i);

      ofstream imgfile (loc, ios::binary);  
      imgfile << "P5" << endl;
      imgfile << g_imgWidth << " " << g_imgHeight << endl;
      imgfile << 255 << endl;
      imgfile.write(reinterpret_cast<const char*> (dataDec), frameSizeDec);
      imgfile.close();
    }

    // PROBLEM ::
    // Pipeline freezes at seek(12)
    //  
    // set log level
    // export GST_DEBUG=0,videodecoder:6
    //
    // Possible issue : While seeking to a non key frame, the deocder keeps on storing the frames until a keyframe is found as depicted from log:
    // videodecoder gstvideodecoder.c:3321:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> decoder frame list getting long: 26 frames,possible internal leaking?    
    seek(i);

    i++ ;

    if(frameSizeRaw < 0)
      break;
  }

  //Cleanup and exit
  delete[] dataRaw;
  delete[] dataDec;
  vidFile.close();
  gst_element_set_state (g_gstPipeline, GST_STATE_NULL);
  gst_object_unref (g_gstPipeline);

  return 0;
}

Compile the code as:

g++ -g testSeek.cpp -o testSeek `pkg-config --cflags --libs gstreamer-1.0 gstreamer-app-1.0`

Run the binary as:

./testSeek input.avi

The above code requires a video to run.
You can find the video at this link

https://www.dropbox.com/s/br9h6o6t6e9o1n3/input.avi?dl=0

Hi rastogishubham20,

We run omxh264dec and avdec_h264, both can repro freezes issue.
Could you share your fail logs?

Thanks!

Hi carolyuu,
Could you please tell at which part pipeline is freezing in case of avdec_h264 element, because I cannot find the freezing issue in using avdec_h264.

If pipeline is freezing at last frame, then it is fine because I am using segment seeking and it will not emit eos at the end of file.
But in case of omxh264dec, pipeline is also freezing while seeking at initial frames.

Please find the possible fails log below in case of omxh264dec:

0:00:00.359866892 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:2543:gst_video_decoder_new_frame:<omxh264dec-omxh264dec0> Created new frame 0x7fa8003b20 (sfn:305)
0:00:00.359886736 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:3313:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> PTS 0:00:01.600000000, DTS 0:00:01.600000000
0:00:00.359909757 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:3314:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> dist 305
0:00:00.359927829 17913       0x623ed0 DEBUG           videodecoder gstvideodecoder.c:3321:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> decoder frame list getting long: 17 frames,possible internal leaking?
0:00:00.360056995 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:2386:gst_video_decoder_chain:<omxh264dec-omxh264dec0> chain PTS 0:00:01.640000000, DTS 0:00:01.640000000 duration 0:00:00.040000000 size 9677
0:00:00.360088869 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:2543:gst_video_decoder_new_frame:<omxh264dec-omxh264dec0> Created new frame 0x7fac015710 (sfn:306)
0:00:00.360108661 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:3313:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> PTS 0:00:01.640000000, DTS 0:00:01.640000000
0:00:00.360131838 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:3314:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> dist 306
0:00:00.360149598 17913       0x623ed0 DEBUG           videodecoder gstvideodecoder.c:3321:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> decoder frame list getting long: 18 frames,possible internal leaking?
0:00:00.360273868 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:2386:gst_video_decoder_chain:<omxh264dec-omxh264dec0> chain PTS 0:00:01.680000000, DTS 0:00:01.680000000 duration 0:00:00.040000000 size 8480
0:00:00.360306055 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:2543:gst_video_decoder_new_frame:<omxh264dec-omxh264dec0> Created new frame 0x7fa401b000 (sfn:307)
0:00:00.360325169 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:3313:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> PTS 0:00:01.680000000, DTS 0:00:01.680000000
0:00:00.360348086 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:3314:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> dist 307
0:00:00.360365794 17913       0x623ed0 DEBUG           videodecoder gstvideodecoder.c:3321:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> decoder frame list getting long: 19 frames,possible internal leaking?
0:00:00.360482876 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:2386:gst_video_decoder_chain:<omxh264dec-omxh264dec0> chain PTS 0:00:01.720000000, DTS 0:00:01.720000000 duration 0:00:00.040000000 size 8295
0:00:00.360526209 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:2543:gst_video_decoder_new_frame:<omxh264dec-omxh264dec0> Created new frame 0x7fa80033b0 (sfn:308)
0:00:00.360545948 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:3313:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> PTS 0:00:01.720000000, DTS 0:00:01.720000000
0:00:00.360568708 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:3314:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> dist 308
0:00:00.360585844 17913       0x623ed0 DEBUG           videodecoder gstvideodecoder.c:3321:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> decoder frame list getting long: 20 frames,possible internal leaking?
0:00:00.360706468 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:2386:gst_video_decoder_chain:<omxh264dec-omxh264dec0> chain PTS 0:00:01.760000000, DTS 0:00:01.760000000 duration 0:00:00.040000000 size 8236
0:00:00.360737353 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:2543:gst_video_decoder_new_frame:<omxh264dec-omxh264dec0> Created new frame 0x7fa8062090 (sfn:309)
0:00:00.360756728 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:3313:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> PTS 0:00:01.760000000, DTS 0:00:01.760000000
0:00:00.360779488 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:3314:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> dist 309
0:00:00.360796988 17913       0x623ed0 DEBUG           videodecoder gstvideodecoder.c:3321:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> decoder frame list getting long: 21 frames,possible internal leaking?
0:00:00.360914955 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:2386:gst_video_decoder_chain:<omxh264dec-omxh264dec0> chain PTS 0:00:01.800000000, DTS 0:00:01.800000000 duration 0:00:00.040000000 size 8031
0:00:00.360945892 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:2543:gst_video_decoder_new_frame:<omxh264dec-omxh264dec0> Created new frame 0x7fa40043b0 (sfn:310)
0:00:00.360964798 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:3313:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> PTS 0:00:01.800000000, DTS 0:00:01.800000000
0:00:00.360987090 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:3314:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> dist 310
0:00:00.361004173 17913       0x623ed0 DEBUG           videodecoder gstvideodecoder.c:3321:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> decoder frame list getting long: 22 frames,possible internal leaking?
0:00:00.361121672 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:2386:gst_video_decoder_chain:<omxh264dec-omxh264dec0> chain PTS 0:00:01.840000000, DTS 0:00:01.840000000 duration 0:00:00.040000000 size 8561
0:00:00.361154120 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:2543:gst_video_decoder_new_frame:<omxh264dec-omxh264dec0> Created new frame 0x7fa8062a20 (sfn:311)
0:00:00.361172817 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:3313:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> PTS 0:00:01.840000000, DTS 0:00:01.840000000
0:00:00.361195161 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:3314:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> dist 311
0:00:00.361226827 17913       0x623ed0 DEBUG           videodecoder gstvideodecoder.c:3321:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> decoder frame list getting long: 23 frames,possible internal leaking?
0:00:00.361349274 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:2386:gst_video_decoder_chain:<omxh264dec-omxh264dec0> chain PTS 0:00:01.880000000, DTS 0:00:01.880000000 duration 0:00:00.040000000 size 8366
0:00:00.361380471 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:2543:gst_video_decoder_new_frame:<omxh264dec-omxh264dec0> Created new frame 0x7fa401b660 (sfn:312)
0:00:00.361399378 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:3313:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> PTS 0:00:01.880000000, DTS 0:00:01.880000000
0:00:00.361421669 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:3314:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> dist 312
0:00:00.361439377 17913       0x623ed0 DEBUG           videodecoder gstvideodecoder.c:3321:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> decoder frame list getting long: 24 frames,possible internal leaking?
0:00:00.361568595 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:2386:gst_video_decoder_chain:<omxh264dec-omxh264dec0> chain PTS 0:00:01.920000000, DTS 0:00:01.920000000 duration 0:00:00.040000000 size 8326
0:00:00.361605469 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:2543:gst_video_decoder_new_frame:<omxh264dec-omxh264dec0> Created new frame 0x7fac077a60 (sfn:313)
0:00:00.361625365 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:3313:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> PTS 0:00:01.920000000, DTS 0:00:01.920000000
0:00:00.361648698 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:3314:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> dist 313
0:00:00.361667083 17913       0x623ed0 DEBUG           videodecoder gstvideodecoder.c:3321:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> decoder frame list getting long: 25 frames,possible internal leaking?
0:00:00.361805467 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:2386:gst_video_decoder_chain:<omxh264dec-omxh264dec0> chain PTS 0:00:01.960000000, DTS 0:00:01.960000000 duration 0:00:00.040000000 size 8382
0:00:00.361843748 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:2543:gst_video_decoder_new_frame:<omxh264dec-omxh264dec0> Created new frame 0x7fa8034080 (sfn:314)
0:00:00.361864894 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:3313:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> PTS 0:00:01.960000000, DTS 0:00:01.960000000
0:00:00.361890727 17913       0x623ed0 LOG             videodecoder gstvideodecoder.c:3314:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> dist 314
0:00:00.361910623 17913       0x623ed0 DEBUG           videodecoder gstvideodecoder.c:3321:gst_video_decoder_decode_frame:<omxh264dec-omxh264dec0> decoder frame list getting long: 26 frames,possible internal leaking?

Could you tell why pipeline is freezing and how this can be solved?

Thanks!

Hi rastogishubham20,
Please share the logic of calculating seekPos. It is embarrassing that we don’t have much experience and your code is a bit difficult to us. Not sure but it seems like you seek in a small interval such as 100 ms or less.

Hi DaneLLL,

Let me explain you how I am calculating the seekPos. Before that let me explain you what I’m trying to achieve by seeking.
In my application, I want to start creating the video and dump images from a given frame number.
So to achieve that first I seek to the given frame number and then start grabbing the raw and decoded frames from the pipeline. Although in above test program I am seeking to each frame just to show you how pipeline is freezing while seeking to some frames.
Another thing I want to mention you is that seeking to some random frame seems wrong because if it is not a keyframe, the decoder will not be able to decode that but I assume if the frame is not decodable, the decoder should skip and start giving the frame from the next keyframe which is happening if seek doesn’t freeze.

Not let me explain the logic of calculating the seekPos.
Given a frame number, I convert that frame number to GST_FORMAT_TIME in nanoseconds. For that, I use a variable g_timeBase which is time duration of one frame.
g_timeBase = (1/frameRate)GST_SECOND2.
GST_SECOND : https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/GstClock.html#GST-SECOND:CAPS

For the shared video, you can exclude the multiplication of 2 in calculating g_timeBase.
I am multiplying by 2 because while testing with other h264 videos, the time duration of one frame is actually (1/frameRate)GST_SECOND2.

So in above test program with shared video, if I am seeking to 2nd frame, I am actually seeking to 4th frame.

Example for better explanation of calculation of seekPos.
Video Frame Rate = 50fps.
So frame duration of one frame = (1/50)seconds
= 0.02 seconds
= 2*10^7 nanoseconds.

For seek(5):
Frame number = 5
Frame time in nanoseconds = 5 * 2 * 10^7 nanoseconds.
Therefore, seekPos = 10^8 nanoseconds.

Can you explain what do you mean by seeking in a small interval such as 100 ms or less.

Hi,
We don’t see issue with attached sample. Can you try it also?

Test clip: http://www.dvdloc8.com/clip.php?movieid=12954&clipid=1

$ g++ -Wall -std=c++11  test2.cpp -o test $(pkg-config --cflags --libs gstreamer-app-1.0) -ldl
$ ./test
#include <cstdlib>
#include <gst/gst.h>
#include <gst/gstinfo.h>
#include <gst/app/gstappsink.h>
#include <glib-unix.h>
#include <dlfcn.h>

#include <iostream>
#include <sstream>
#include <thread>

using namespace std;

#define USE(x) ((void)(x))

static GstPipeline *gst_pipeline = nullptr;
static string launch_string;
static int frame_count = 0;
//static int sleep_count = 0;
static int eos = 0;

static void appsink_eos(GstAppSink * appsink, gpointer user_data)
{
    printf("app sink receive eos\n");
    eos = 1;
//    g_main_loop_quit (hpipe->loop);
}

static GstFlowReturn new_buffer(GstAppSink *appsink, gpointer user_data)
{
    GstSample *sample = NULL;

    g_signal_emit_by_name (appsink, "pull-sample", &sample,NULL);

    if (sample)
    {
        GstBuffer *buffer = NULL;
        GstCaps   *caps   = NULL;
        GstMapInfo map    = {0};

        caps = gst_sample_get_caps (sample);
        if (!caps)
        {
            printf("could not get snapshot format\n");
        }
        gst_caps_get_structure (caps, 0);
        buffer = gst_sample_get_buffer (sample);
        gst_buffer_map (buffer, &map, GST_MAP_READ);

        printf("map.size = %lu, pts: %lu \n", map.size, buffer->pts);
        frame_count++;

        gst_buffer_unmap(buffer, &map);

        gst_sample_unref (sample);
    }
    else
    {
        g_print ("could not make snapshot\n");
    }

    return GST_FLOW_OK;
}

int main(int argc, char** argv) {
    USE(argc);
    USE(argv);

    gst_init (&argc, &argv);

    GMainLoop *main_loop;
    main_loop = g_main_loop_new (NULL, FALSE);
    ostringstream launch_stream;
    int w = 1920;
    int h = 816;
    GstAppSinkCallbacks callbacks = {appsink_eos, NULL, new_buffer};

    launch_stream
//    << "nvcamerasrc ! "
//    << "video/x-raw(memory:NVMM), width="<< w <<", height="<< h <<", framerate=30/1 ! " 
    << "filesrc location=Bourne_trailer.mp4 ! qtdemux ! h264parse ! omxh264dec ! "
    << "tee name=nt "
    << "nt. ! queue ! nvvidconv ! "
    << "video/x-raw, format=I420, width="<< w <<", height="<< h <<" ! "
    << "appsink name=mysink "
    << "nt. ! queue ! nvoverlaysink ";

    launch_string = launch_stream.str();

    g_print("Using launch string: %s\n", launch_string.c_str());

    GError *error = nullptr;
    gst_pipeline  = (GstPipeline*) gst_parse_launch(launch_string.c_str(), &error);

    if (gst_pipeline == nullptr) {
        g_print( "Failed to parse launch: %s\n", error->message);
        return -1;
    }
    if(error) g_error_free(error);

    GstElement *appsink_ = gst_bin_get_by_name(GST_BIN(gst_pipeline), "mysink");
    gst_app_sink_set_callbacks (GST_APP_SINK(appsink_), &callbacks, NULL, NULL);

    gst_element_set_state((GstElement*)gst_pipeline, GST_STATE_PLAYING); 

    /*while (eos == 0) {
        sleep(1);
        sleep_count++;
    }*/
    sleep(3);
    guint64 index = 1;
    guint64 seekPos;
    for (index=1; index < 100; index ++) {
        seekPos = (index%25)*1000000000;
        printf("[%ld]seek to %ld \n", index, seekPos);
        gst_element_seek ((GstElement*)gst_pipeline, 1.0, GST_FORMAT_TIME, (GstSeekFlags)(GST_SEEK_FLAG_FLUSH | GST_SEEK_FLAG_KEY_UNIT | GST_SEEK_FLAG_SEGMENT), GST_SEEK_TYPE_SET, seekPos, GST_SEEK_TYPE_NONE, -1);

        GstStateChangeReturn ret = gst_element_get_state ((GstElement*)gst_pipeline, NULL, NULL, 5 * GST_SECOND);
        if (ret == GST_STATE_CHANGE_FAILURE)
        {
          printf("seek failed \n");
          break;
        }
        sleep(1);
    }
    sleep(3);
    //sleep(90);
    //g_main_loop_run (main_loop);

    gst_element_set_state((GstElement*)gst_pipeline, GST_STATE_NULL);
    gst_object_unref(GST_OBJECT(gst_pipeline));
    g_main_loop_unref(main_loop);

    g_print("going to exit \n");
    return 0;
}

[Posting on rastogishubham20’s behalf]

Hi DaneLLL,

I have tested your sample code and it is working fine but it seems your pipeline is only generating decoded data. In my application, raw data is also required.

I have modified your source code according to my pipeline and it is still freezing.
One more thing to notice is that when pipeline is freezing, the state change is async.

Please find modified code at link below:
https://pastebin.com/sgfNXhFU

It is quite obvious that the issue is with omxh264dec element because if I use avdec_h264 element instead, the pipeline is working fine.
I have tested the above code with the video I had provided earlier.

Kindly help me in getting this issue resolved.

Hi techbull,
Somehow I cannot access the link.

Because we have public source code of libgstomx.so, you may also check it yourself.
https://developer.nvidia.com/embedded/dlc/l4t-tx1-sources-28-2-ga

Hi DaneLLL,

You can download the code from link below:
https://www.dropbox.com/s/y5hixrzdofb8gk0/gst_code.cpp?dl=0

In the mean time, we will also check library code and will try to find out possible problem.

Hi techbull,
Still not observe the issue with the attached test code. FYR.

#include <cstdlib>
#include <gst/gst.h>
#include <gst/gstinfo.h>
#include <gst/app/gstappsink.h>
#include <glib-unix.h>
#include <dlfcn.h>

#include <iostream>
#include <sstream>
#include <thread>

using namespace std;

#define USE(x) ((void)(x))

static GstPipeline *gst_pipeline = nullptr;
static string launch_string;
static int frame_count = 0;
//static int sleep_count = 0;
static int eos = 0;

static void appsink_eos(GstAppSink * appsink, gpointer user_data)
{
    printf("app sink receive eos\n");
    eos = 1;
//    g_main_loop_quit (hpipe->loop);
}

static GstFlowReturn new_buffer(GstAppSink *appsink, gpointer user_data)
{
    GstSample *sample = NULL;

    g_signal_emit_by_name (appsink, "pull-sample", &sample,NULL);

    if (sample)
    {
        GstBuffer *buffer = NULL;
        GstCaps   *caps   = NULL;
        GstMapInfo map    = {0};

        caps = gst_sample_get_caps (sample);
        if (!caps)
        {
            printf("could not get snapshot format\n");
        }
        gst_caps_get_structure (caps, 0);
        buffer = gst_sample_get_buffer (sample);
        gst_buffer_map (buffer, &map, GST_MAP_READ);

        //printf("map.size = %lu, pts: %lu \n", map.size, buffer->pts);
        frame_count++;

        gst_buffer_unmap(buffer, &map);

        gst_sample_unref (sample);
    }
    else
    {
        g_print ("could not make snapshot\n");
    }

    return GST_FLOW_OK;
}

static void rawsink_eos(GstAppSink * appsink, gpointer user_data)
{
    printf("raw sink receive eos\n");
}

static GstFlowReturn raw_buffer(GstAppSink *appsink, gpointer user_data)
{
    GstSample *sample = NULL;

    g_signal_emit_by_name (appsink, "pull-sample", &sample,NULL);

    if (sample)
    {
        GstBuffer *buffer = NULL;
        GstCaps   *caps   = NULL;
        GstMapInfo map    = {0};

        caps = gst_sample_get_caps (sample);
        if (!caps)
        {
            printf("could not get snapshot format\n");
        }
        gst_caps_get_structure (caps, 0);
        buffer = gst_sample_get_buffer (sample);
        gst_buffer_map (buffer, &map, GST_MAP_READ);

        printf("raw map.size = %lu, pts: %lu \n", map.size, buffer->pts);

        gst_buffer_unmap(buffer, &map);

        gst_sample_unref (sample);
    }
    else
    {
        g_print ("could not make snapshot\n");
    }

    return GST_FLOW_OK;
}

int main(int argc, char** argv) {
    USE(argc);
    USE(argv);

    gst_init (&argc, &argv);

    GMainLoop *main_loop;
    main_loop = g_main_loop_new (NULL, FALSE);
    ostringstream launch_stream;
    int w = 1920;
    int h = 816;
    GstAppSinkCallbacks callbacks = {appsink_eos, NULL, new_buffer};
    GstAppSinkCallbacks rawcallbacks = {rawsink_eos, NULL, raw_buffer};

    launch_stream
    << "filesrc location=Bourne_trailer.mp4 ! qtdemux ! tee name=nt "
    << "nt. ! queue ! h264parse ! video/x-h264, stream-format=byte-stream, alignment=au ! "
    <<"appsink name=rawsink "
    << "nt. ! queue ! h264parse ! omxh264dec ! "
    << "nvvidconv ! "
    << "video/x-raw, format=GRAY8, width="<< w <<", height="<< h <<" ! "
    << "appsink name=mysink ";

    launch_string = launch_stream.str();

    g_print("Using launch string: %s\n", launch_string.c_str());

    GError *error = nullptr;
    gst_pipeline  = (GstPipeline*) gst_parse_launch(launch_string.c_str(), &error);

    if (gst_pipeline == nullptr) {
        g_print( "Failed to parse launch: %s\n", error->message);
        return -1;
    }
    if(error) g_error_free(error);

    GstElement *appsink_ = gst_bin_get_by_name(GST_BIN(gst_pipeline), "mysink");
    gst_app_sink_set_callbacks (GST_APP_SINK(appsink_), &callbacks, NULL, NULL);

    GstElement *rawsink_ = gst_bin_get_by_name(GST_BIN(gst_pipeline), "rawsink");
    gst_app_sink_set_callbacks (GST_APP_SINK(rawsink_), &rawcallbacks, NULL, NULL);

    gst_element_set_state((GstElement*)gst_pipeline, GST_STATE_PLAYING); 

    /*while (eos == 0) {
        sleep(1);
        sleep_count++;
    }*/
    sleep(3);
    guint64 index = 1;
    guint64 seekPos;
    for (index=1; index < 100; index ++) {
        seekPos = (GST_SECOND/25)*2*index;//(index%25)*1000000000;
        printf("[%ld]seek to %ld \n", index, seekPos);
        gst_element_seek ((GstElement*)gst_pipeline, 1.0, GST_FORMAT_TIME, (GstSeekFlags)(GST_SEEK_FLAG_FLUSH | GST_SEEK_FLAG_KEY_UNIT | GST_SEEK_FLAG_SEGMENT), GST_SEEK_TYPE_SET, seekPos, GST_SEEK_TYPE_NONE, -1);

        GstStateChangeReturn ret = gst_element_get_state ((GstElement*)gst_pipeline, NULL, NULL, 5 * GST_SECOND);
        if (ret == GST_STATE_CHANGE_FAILURE)
        {
          printf("seek failed \n");
          break;
        }
        sleep(1);
    }
    sleep(3);
    //sleep(90);
    //g_main_loop_run (main_loop);

    gst_element_set_state((GstElement*)gst_pipeline, GST_STATE_NULL);
    gst_object_unref(GST_OBJECT(gst_pipeline));
    g_main_loop_unref(main_loop);

    g_print("going to exit \n");
    return 0;
}

Hi DaneLLL,

While seeking in videos, we are still facing freeze issue in our pipeline (with tee).

As your single decoded/raw pipeline wasn’t freezing, we have used the same logic to modify our pipeline.

Rather than using tee, we are now opening two individual pipelines (one for raw and other for decoded) and doing seek. Now, that we have two pipelines, code for synchronization between two pipelines is additional but yes it seems to work :)

We have tested the code on some sample videos and it is working fine but we are yet to do rigorous testing.