FPS=0 when I use enc-type=1 (use software encoder x256/x264) in deepstream app using splitmuxsink

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hardware Platform (Jetson / GPU): GPU NVIDIA GeForce RTX 2080 Ti
DeepStream Version: 6.3
TensorRT Version: 8.5.3
NVIDIA GPU Driver Version (valid for GPU only): 530.30.02
Issue Type( questions, new requirements, bugs): Bug
How to reproduce the issue ?
I have problem with software encoder. I’m using deepstream app with Splitmuxsink, when I use enc-type=1 in config file, the app run with fps=0 (I also test with file sink and this problem not happen). You can see this picture:

image
image1430×346 16.2 KB

  • My config:
[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=1
rows=1
columns=1
width=2736
height=2736
gpu-id=0
#(0): nvbuf-mem-default - Default memory allocated, specific to particular platform
#(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory, applicable for Tesla
#(2): nvbuf-mem-cuda-device - Allocate Device cuda memory, applicable for Tesla
#(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory, applicable for Tesla
#(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
nvbuf-memory-type=0

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
uri=file://../../videos/manypeoples.mp4
num-sources=1
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0
drop-frame-interval=0
latency=10
rtsp-reconnect-interval-sec=3
rtsp-reconnect-attempts=-1
select-rtp-protocol=4

[source1]
enable=0
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
uri=file://../../videos/manypeoples.mp4
num-sources=1
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0
drop-frame-interval=0
latency=10
rtsp-reconnect-interval-sec=3
rtsp-reconnect-attempts=-1
select-rtp-protocol=4

[source2]
enable=0
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
uri=file://../../videos/manypeoples.mp4
num-sources=1
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0
drop-frame-interval=0
latency=10
rtsp-reconnect-interval-sec=3
rtsp-reconnect-attempts=-1
select-rtp-protocol=4

[source3]
enable=0
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
uri=file://../../videos/manypeoples.mp4
num-sources=1
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0
drop-frame-interval=0
latency=10
rtsp-reconnect-interval-sec=3
rtsp-reconnect-attempts=-1
select-rtp-protocol=4

[user-plugin]
enable=1
gpu-id=0
app-config-path=configs/vpd/appConfig.json
nvbuf-memory-type=3

[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=h264 2=h265
codec=2
sync=1
source-id=0
bitrate=50000
# set below properties in case of RTSPStreaming
rtsp-port=8554
udp-port=5400
width=1000
height=1000
enc-type=0

[sink1]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=3
output-file=./out%05d.mp4
#1=h264 2=h265
codec=2
sync=0
source-id=0
bitrate=5000000
#1=mp4 2=mkv
container=1
width=2736
height=2736
enc-type=1
max-size-bytes=100000000
max-size-time=0

[osd]
enable=0
gpu-id=0
border-width=3
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=0
batch-size=1
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=100000
## Set muxer output width and height
width=2736
height=2736
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0
buffer-pool-size=4
drop-pipeline-eos=1

# config-file property is mandatory for any gie section.
# Other properties are optional and if set will override the properties set in
# the infer config file.
[primary-gie]
enable=1
gpu-id=0
batch-size=4
gie-unique-id=1
nvbuf-memory-type=0
config-file=config_infer_primary_yoloV4.txt

[tracker]
enable=1
tracker-width=1280
tracker-height=1280
ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
ll-config-file=config_tracker_NvDCF.yml
#ll-config-file=config_tracker_DeepSORT.yml
display-tracking-id=1

[tests]
file-loop=1
  • This is my update to use Splitmuxsink in deepstream_sink_bin.c:

/**
 * Function to create sink bin to generate encoded output.
 */
static gboolean
create_encode_file_bin (NvDsSinkEncoderConfig * config, NvDsSinkBinSubBin * bin)
{
  GstCaps *caps = NULL;
  gboolean ret = FALSE;
  gchar elem_name[50];
  int probe_id = 0;
  gulong bitrate = config->bitrate;
  guint profile = config->profile;
  const gchar * latency = g_getenv("NVDS_ENABLE_LATENCY_MEASUREMENT");

  uid++;

  g_snprintf (elem_name, sizeof (elem_name), "sink_sub_bin%d", uid);
  bin->bin = gst_bin_new (elem_name);
  if (!bin->bin) {
    NVGSTDS_ERR_MSG_V ("Failed to create '%s'", elem_name);
    goto done;
  }

  g_snprintf (elem_name, sizeof (elem_name), "sink_sub_bin_queue%d", uid);
  bin->queue = gst_element_factory_make (NVDS_ELEM_QUEUE, elem_name);
  if (!bin->queue) {
    NVGSTDS_ERR_MSG_V ("Failed to create '%s'", elem_name);
    goto done;
  }

  g_snprintf (elem_name, sizeof (elem_name), "sink_sub_bin_transform%d", uid);
  bin->transform = gst_element_factory_make (NVDS_ELEM_VIDEO_CONV, elem_name);
  if (!bin->transform) {
    NVGSTDS_ERR_MSG_V ("Failed to create '%s'", elem_name);
    goto done;
  }

  g_snprintf (elem_name, sizeof (elem_name), "sink_sub_bin_cap_filter%d", uid);
  bin->cap_filter = gst_element_factory_make (NVDS_ELEM_CAPS_FILTER, elem_name);
  if (!bin->cap_filter) {
    NVGSTDS_ERR_MSG_V ("Failed to create '%s'", elem_name);
    goto done;
  }
  if (config->codec == NV_DS_ENCODER_MPEG4
      || config->enc_type == NV_DS_ENCODER_TYPE_SW)
    caps = gst_caps_from_string ("video/x-raw, format=I420");
  else
    caps = gst_caps_from_string ("video/x-raw(memory:NVMM), format=I420");
  g_object_set (G_OBJECT (bin->cap_filter), "caps", caps, NULL);

  g_snprintf (elem_name, sizeof (elem_name), "sink_sub_bin_encoder%d", uid);
  switch (config->codec) {
    case NV_DS_ENCODER_H264:
      if (config->enc_type == NV_DS_ENCODER_TYPE_SW)
        bin->encoder =
            gst_element_factory_make (NVDS_ELEM_ENC_H264_SW, elem_name);
      else
        bin->encoder =
            gst_element_factory_make (NVDS_ELEM_ENC_H264_HW, elem_name);
      break;
    case NV_DS_ENCODER_H265:
      if (config->enc_type == NV_DS_ENCODER_TYPE_SW)
        bin->encoder =
            gst_element_factory_make (NVDS_ELEM_ENC_H265_SW, elem_name);
      else
        bin->encoder =
            gst_element_factory_make (NVDS_ELEM_ENC_H265_HW, elem_name);
      break;
    case NV_DS_ENCODER_MPEG4:
      bin->encoder = gst_element_factory_make (NVDS_ELEM_ENC_MPEG4, elem_name);
      break;
    default:
      goto done;
  }
  if (!bin->encoder) {
    NVGSTDS_ERR_MSG_V ("Failed to create '%s'", elem_name);
    goto done;
  }

  NVGSTDS_ELEM_ADD_PROBE (probe_id,
      bin->encoder, "sink",
      seek_query_drop_prob, GST_PAD_PROBE_TYPE_QUERY_UPSTREAM, bin);

  probe_id = probe_id;

  if (config->codec == NV_DS_ENCODER_MPEG4)
    config->enc_type = NV_DS_ENCODER_TYPE_SW;

  struct cudaDeviceProp prop;
  cudaGetDeviceProperties (&prop, config->gpu_id);

  if (config->copy_meta == 1) {
      g_object_set (G_OBJECT (bin->encoder), "copy-meta", TRUE, NULL);
  }

  if (config->enc_type == NV_DS_ENCODER_TYPE_HW) {
    switch (config->output_io_mode) {
      case NV_DS_ENCODER_OUTPUT_IO_MODE_MMAP:
      default:
        g_object_set (G_OBJECT (bin->encoder), "output-io-mode",
                NV_DS_ENCODER_OUTPUT_IO_MODE_MMAP, NULL);
        break;
      case NV_DS_ENCODER_OUTPUT_IO_MODE_DMABUF_IMPORT:
        g_object_set (G_OBJECT (bin->encoder), "output-io-mode",
                NV_DS_ENCODER_OUTPUT_IO_MODE_DMABUF_IMPORT, NULL);
        break;
    }
  }

  if (config->enc_type == NV_DS_ENCODER_TYPE_HW) {
    g_object_set (G_OBJECT (bin->encoder), "profile", profile, NULL);
    g_object_set (G_OBJECT (bin->encoder), "iframeinterval",
        config->iframeinterval, NULL);
    g_object_set (G_OBJECT (bin->encoder), "bitrate", bitrate, NULL);
  } else {
    if (config->codec == NV_DS_ENCODER_MPEG4)
      g_object_set (G_OBJECT (bin->encoder), "bitrate", bitrate, NULL);
    else {
      //bitrate is in kbits/sec for software encoder x264enc and x265enc
      g_object_set (G_OBJECT (bin->encoder), "bitrate", bitrate / 1000, NULL);
      g_object_set (G_OBJECT (bin->encoder), "speed-preset", config->sw_preset, NULL);
    }
  }

  switch (config->codec) {
    case NV_DS_ENCODER_H264:
      bin->codecparse = gst_element_factory_make ("h264parse", "h264-parser");
      break;
    case NV_DS_ENCODER_H265:
      bin->codecparse = gst_element_factory_make ("h265parse", "h265-parser");
      break;
    case NV_DS_ENCODER_MPEG4:
      bin->codecparse =
          gst_element_factory_make ("mpeg4videoparse", "mpeg4-parser");
      break;
    default:
      goto done;
  }

  g_snprintf (elem_name, sizeof (elem_name), "sink_sub_bin_mux%d", uid);
  //disabling the mux when latency measurement logs are enabled
  if (latency) {
    bin->mux = gst_element_factory_make (NVDS_ELEM_IDENTITY, elem_name);
  }
  else {
    switch (config->container) {
      case NV_DS_CONTAINER_MP4:
        bin->mux = gst_element_factory_make (NVDS_ELEM_MUX_MP4, elem_name);
        break;
      case NV_DS_CONTAINER_MKV:
        bin->mux = gst_element_factory_make (NVDS_ELEM_MKV, elem_name);
        break;
      default:
        goto done;
    }
  }

  if (!bin->mux) {
    NVGSTDS_ERR_MSG_V ("Failed to create '%s'", elem_name);
    goto done;
  }

  g_snprintf (elem_name, sizeof (elem_name), "sink_sub_bin_sink%d", uid);
#ifdef USE_SPLITMUXSINK
  bin->sink = gst_element_factory_make (NVDS_ELEM_SINK_SPLITMUX, elem_name);
#else
  bin->sink = gst_element_factory_make (NVDS_ELEM_SINK_FILE, elem_name);
#endif
  if (!bin->sink) {
    NVGSTDS_ERR_MSG_V ("Failed to create '%s'", elem_name);
    goto done;
  }

#ifdef USE_SPLITMUXSINK
  g_object_set (G_OBJECT (bin->sink), "max-size-bytes", config->max_size_bytes,
      "max-size-time", config->max_size_time, "muxer", bin->mux, NULL);
  g_object_set (G_OBJECT (bin->sink), "location", config->output_file_path,NULL);
#else
  g_object_set (G_OBJECT (bin->sink), "location", config->output_file_path,
      "sync", config->sync, "async", FALSE, NULL);
#endif
  g_object_set (G_OBJECT (bin->transform), "gpu-id", config->gpu_id, NULL);
  gst_bin_add_many (GST_BIN (bin->bin), bin->queue,
      bin->transform, bin->codecparse, bin->cap_filter,
#ifdef USE_SPLITMUXSINK
      bin->encoder, bin->sink, NULL);
#else
      bin->encoder, bin->mux, bin->sink, NULL);
#endif

  NVGSTDS_LINK_ELEMENT (bin->queue, bin->transform);

  NVGSTDS_LINK_ELEMENT (bin->transform, bin->cap_filter);
  NVGSTDS_LINK_ELEMENT (bin->cap_filter, bin->encoder);

  NVGSTDS_LINK_ELEMENT (bin->encoder, bin->codecparse);
#ifdef USE_SPLITMUXSINK
  NVGSTDS_LINK_ELEMENT (bin->codecparse, bin->sink);
#else
  NVGSTDS_LINK_ELEMENT (bin->codecparse, bin->mux);
  NVGSTDS_LINK_ELEMENT (bin->mux, bin->sink);
#endif
  NVGSTDS_BIN_ADD_GHOST_PAD (bin->bin, bin->queue, "sink");

  ret = TRUE;

done:
  if (caps) {
    gst_caps_unref (caps);
  }
  if (!ret) {
    NVGSTDS_ERR_MSG_V ("%s failed", __func__);
  }
  return ret;
}

Thank you for your support!

3

Please provide complete information as applicable to your setup. Thanks
Hardware Platform (Jetson / GPU)
DeepStream Version
JetPack Version (valid for Jetson only)
TensorRT Version
NVIDIA GPU Driver Version (valid for GPU only)
Issue Type( questions, new requirements, bugs)
How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

4

Thanks for your response. I updated it.

5

Could you try the command below?

gst-launch-1.0 uridecodebin uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! splitmuxsink muxer=matroskamux location=chunk%01d.mp4 max-size-time=10

If this pipeline works properly, it means that your environment supports hardware encoding. You need to check your code logic yourself.

6

But with hardware encoder, I still run it ok, no problem in here. I only have a problem with software encoder x264enc/x265enc.

7

OK. If you replace the nvv4l2h264enc in the pipeline above with x264enc, will it still work properly?

8

It runs fine. I think the reason is only in the splitmuxsink code I added on create_encode_file_bin()

9

OK. Could you try to set the parameters like the pipeline attached?

  g_object_set (G_OBJECT (bin->sink), "max-size-bytes", config->max_size_bytes,
      "max-size-time", config->max_size_time, "muxer-factory", "matroskamux", NULL);

And open more log with GST_DEBUG=3 at the begginning of the command.
Did it work properly when you set the enc-type=0?

10

Yes. I set GST_DEBUG=3 but I don’t see any error. With enc-type=0, I running fine. And you know, when I use filesink (# else), it also runs fine with enc-type=1. In another case (splitmuxsink), when I off sink1=0 (enable=0) and set enc-type=1(sink0), RTSP ran fine.
I tried this param but It still drop fps to 0.

  g_object_set (G_OBJECT (bin->sink), "max-size-bytes", config->max_size_bytes,
      "max-size-time", config->max_size_time, "muxer-factory", "matroskamux", NULL);
11

Let me summarize briefly and see if it matches your scenarios.
1.When you enable the rtspsink and filesink, both encoding modes work properly.
2.When you enable the rtspsink and splitmuxsink, only enc-type=0 for splitmuxsink works properly.
3.When you enable the rtspsink and disable the splitmuxsink, both encoding modes work properly.
Have you tried disable the rtspsink and enable the splitmuxsink with enc_type=1?

1 Like
12

Yes, the summary is enough for my issue.
When I try to disable rtspsink (sink 0) and only run with splitmuxsink (sink 1), the app run fails with this error:

0:00:00.492629098 34227 0x563db33bd200 INFO        GST_ELEMENT_PADS gstelement.c:925:gst_element_get_static_pad: no such pad 'sink' in element "sink_sub_bin_sink1"
** ERROR: <create_pipeline:1171>: Could not find 'sink' in 'sink_sub_bin_sink1'
** ERROR: <create_pipeline:1291>: create_pipeline failed
** ERROR: <main:690>: Failed to create pipeline
Quitting
  • This error also only happens with splitmuxsink.
13

I have follow this issue and fix the error: Could not find ‘sink’ in ‘sink_sub_bin_sink1’. But when I enable rtspsink, it error same.

  • But in case only run splitmuxsink, fps still drop to 0.
14

You can try to set the tune parameter for the x264enc.

g_object_set (G_OBJECT (bin->encoder), "tune", 4, NULL);
15

Oh now, It does not drop fps to 0 but fps is still slow (normal fps = 10) with x265enc.
New update:

16

Due to the software encoding you are using, it is normal for the fps to be slow.

1 Like
17

Oh. Do we have to accept reduced fps when using x265enc? With RTSP sink I still use hardware encoder, why does my fps performance decrease?

18

When I use “speed-preset” to superfast (x265enc), fps exceeds input

g_object_set (G_OBJECT (bin->encoder), "speed-preset", 2, NULL);
19

We use tee plugin to split data to multiple pads. This plugin is not actually copying data, so the two channels used the same buffer before the buffer copy. They will influence each other’s perf.

1 Like
20

Yes. Thank you so much, Mr. Yuweiw. I understood. So, this issue will close with your suggestion is zero latency for “tune” and I also set “speed-preset” to very fast. And we need to accept performance will not be perfect.

 #ifdef USE_SPLITMUXSINK
   g_object_set (G_OBJECT (bin->encoder), "speed-preset", 3, NULL); // Very fast
   g_object_set (G_OBJECT (bin->encoder), "tune", 4, NULL); // Zero latency

And one more question about pads. You know when I use both rtspsink and splitmuxsink, it runs fine. But if I disable rtspsink and only enable splitmuxsink, The app runs false because pad splimuxsink is “video” but the default pad in deepstream is “sink”. If I change “sink” to “video”, I cannot run rtspsink. Do you have a solution for this problem?

21

These are all official plugins of gstreamer, and you can only recognize them based on your own scenario and make special adjustments for the different name of pads.