Deepstream pipeline does not go into playing state in first run inside docker

Hardware Platform (Jetson / GPU): Jetson Orin NX on Avermedia Carrier Board
• DeepStream Version: 6.3 DP (docker container)
• JetPack Version (valid for Jetson only): 5.1.2
• TensorRT Version: 8.5.2.2
• Issue Type( questions, new requirements, bugs): Fails to load some DeepStream GStreamer plugins

I am running JetPack 5.1.2 (l4t 35.4.1). I have 4 RTSP deepstream pipelines which decode the RTSP stream and send raw frames using shared memory for other apps. Alongside, jpeg encoder is used to encode raw frames and save them in a ring buffer on RAM. I am using nvurisrcbin element with nvvideoconvert/jpegenc etc..

I am using C++ with gst-parse-launch.
I created a docker image with base jetpack5.1.2 on jetson orin nx 16GB, which is also having same jetpack version.

Observation.
When running the pipeline app for the first time, it always does not go into playing state. All 4 pipelines are transitioned to async state and stay there.

But after terminating 1st run, all pipelines are correctly transitioned into playing state in the second run. This exact behavior is observed each and every time.

This is my docker run script

sudo docker run -it --runtime nvidia --network host
–privileged
–cap-add=IPC_LOCK
–cap-add=SYS_ADMIN
–memory=8g
–memory-swap=8g
-v /run/user/$(id -u):/run/user/$(id -u)
-v /tmp/argus_socket:/tmp/argus_socket
-v /home/user/app/videos:/home/user/app/videos
-v /home/user/app/logs:/home/user/app/logs
-v /home/user/app/config.json:/home/user/app/config.json:ro
-v /dev/shm:/dev/shm
–ipc host
-e NVIDIA_VISIBLE_DEVICES=all
-e NVIDIA_DRIVER_CAPABILITIES=all
-e GST_REGISTRY=/home/user/.cache/gstreamer-1.0/registry.aarch64.bin
-e GST_REGISTRY_UPDATE=yes
–device /dev/dri:/dev/dri
-v /home/user/app.0/gst-cache:/home/user/.cache/gstreamer-1.0
-v /tmp/.X11-unix:/tmp/.X11-unix
app-runtime:jp5.1.2

I am using base nvcr.io/nvidia/l4t-jetpack:r35.4.1 and I have another docker which installs applications on top on base docker image.

any help is appreciated.

Hello @srimal.l,

That is an interesting behavior.

Does it only happen on Docker ? Or does the same happens when running the app on host?

best,
Andrew
Embedded Software Engineer at ProventusNova

Hi Andrew,

This behavior is only observed in docker container.
There is no such issue in host.

Best Regards,

Srimal

What does your pipeline look like? Can you describe it in full, or share reproducible code?

Are you using any deepstream elements? Deepsteam typically doesn’t require four separate pipelines; It typically uses nvstreammux/nvinfer to combine different streams.

This file is usually related to the runtime environment, and copying/mapping it may cause some problems.

Deepstream typically uses nvcr.io/nvidia/deepstream:xxx-triton-multiarch image

Hi junshengy,

We are using another python application to process raw images and execute inference. This due to very specific requirements of our application.

My single pipeline looks like this.
nvurisrcbin queue → tee t1 → queue → nvvideoconvert → jpgenc → appsink t2 → queue → nvvideoconvert → appsink t3 → queue → nvvideoconvert → appsink

Here jpeg branch appsink, saves jpegs in RAM in a ring buffer.

t2 branch appsink uses fastdds to send raw images in small resolution

t3 branch appsink uses fastdds to send raw images in a large resolution.

When the pipeline runs fine, I committed current docker image to new image using docker commit <>. Interestingly, pipelines work on first run inside new docker image. I think there might be some cache buildup issues in gestreamer.

Best Regards,

I’m using AGX Orin, JP-6.2, and DS-7.1. I haven’t encountered any problems in the host and container. This is my pipeline, built according to your description. Please try upgrading JP/DS, or refer to the sample code I provided.

docker run -it --rm --privileged --runtime nvidia nvcr.io/nvidia/deepstream-l4t:7.1-triton-multiarch 
/opt/nvidia/deepstream/deepstream/user_additional_install.sh 
apt-get install --reinstall libflac8 libmp3lame0 libxvidcore4 ffmpeg
/*
 * gst_parse_launch() sample:
 * - t1: JPEGs stored in RAM in a ring buffer
 * - t2: appsink writes raw images to files (small resolution)
 * - t3: appsink writes raw images to files (large resolution)
 */

#include <gst/app/gstappsink.h>
#include <gst/gst.h>

#include <glib-unix.h>
#include <glib.h>

#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

typedef struct {
  guint8 *data;
  gsize size;
  gint64 ts_us;
} JpegFrame;

typedef struct {
  JpegFrame *frames;
  guint capacity;
  guint head;  /* next write index */
  guint count; /* <= capacity */
  GMutex lock;
} JpegRing;

static void jpeg_ring_init(JpegRing *ring, guint capacity) {
  memset(ring, 0, sizeof(*ring));
  ring->capacity = MAX(1u, capacity);
  ring->frames = g_new0(JpegFrame, ring->capacity);
  g_mutex_init(&ring->lock);
}

static void jpeg_ring_clear(JpegRing *ring) {
  if (!ring || !ring->frames)
    return;

  for (guint i = 0; i < ring->capacity; i++) {
    g_free(ring->frames[i].data);
    ring->frames[i].data = NULL;
    ring->frames[i].size = 0;
    ring->frames[i].ts_us = 0;
  }
  ring->head = 0;
  ring->count = 0;
}

static void jpeg_ring_free(JpegRing *ring) {
  if (!ring)
    return;
  g_mutex_lock(&ring->lock);
  jpeg_ring_clear(ring);
  g_free(ring->frames);
  ring->frames = NULL;
  g_mutex_unlock(&ring->lock);
  g_mutex_clear(&ring->lock);
}

static void jpeg_ring_push_copy(JpegRing *ring, const guint8 *data, gsize size,
                                gint64 ts_us) {
  if (!ring || !ring->frames || !data || size == 0)
    return;

  guint8 *copy = g_memdup2(data, size);
  if (!copy)
    return;

  g_mutex_lock(&ring->lock);

  JpegFrame *slot = &ring->frames[ring->head];
  g_free(slot->data);
  slot->data = copy;
  slot->size = size;
  slot->ts_us = ts_us;

  ring->head = (ring->head + 1) % ring->capacity;
  if (ring->count < ring->capacity)
    ring->count++;

  g_mutex_unlock(&ring->lock);
}

typedef enum {
  SINK_T1_JPEG_RING,
  SINK_T2_FILE_SMALL,
  SINK_T3_FILE_LARGE,
} SinkKind;

typedef struct {
  const char *name; /* t1/t2/t3 */
  SinkKind kind;
  guint64 frames;
  gint64 start_us;
  guint64 saved;
  guint save_every;
  guint save_limit;
  gchar *out_dir;

  /* t1 */
  JpegRing *jpeg_ring;

  /* t2/t3 */
  guint width;
  guint height;
  gchar *format;
} SinkCtx;

typedef struct {
  GMainLoop *loop;
  GstElement *pipeline;
  JpegRing jpeg_ring;
  SinkCtx t1;
  SinkCtx t2;
  SinkCtx t3;
} AppCtx;

static guint env_u32_or_default(const gchar *name, guint def) {
  const gchar *val = g_getenv(name);
  if (!val || !*val)
    return def;

  gchar *end = NULL;
  unsigned long parsed = strtoul(val, &end, 10);
  if (end == val || (end && *end != '\0') || parsed == 0 ||
      parsed > G_MAXUINT) {
    g_printerr("WARN: invalid %s='%s', using %u\n", name, val, def);
    return def;
  }
  return (guint)parsed;
}

static gboolean ensure_out_dir(const gchar *out_dir) {
  if (!out_dir || !*out_dir)
    return FALSE;
  if (g_mkdir_with_parents(out_dir, 0755) != 0) {
    g_printerr("Failed to create output dir '%s': %s\n", out_dir,
               g_strerror(errno));
    return FALSE;
  }
  return TRUE;
}

static gboolean write_bytes_to_file(const gchar *path, const guint8 *data,
                                    gsize size) {
  FILE *fp = fopen(path, "wb");
  if (!fp) {
    g_printerr("Failed to open %s: %s\n", path, g_strerror(errno));
    return FALSE;
  }
  size_t written = fwrite(data, 1, size, fp);
  fclose(fp);
  if (written != size) {
    g_printerr("Short write to %s (%zu/%zu)\n", path, written, (size_t)size);
    return FALSE;
  }
  return TRUE;
}

static gboolean sink_set_video_info_from_caps(SinkCtx *ctx, GstCaps *caps) {
  if (!caps || gst_caps_is_empty(caps))
    return FALSE;

  GstStructure *s = gst_caps_get_structure(caps, 0);
  if (!s)
    return FALSE;

  gint w = 0, h = 0;
  if (!gst_structure_get_int(s, "width", &w) ||
      !gst_structure_get_int(s, "height", &h))
    return FALSE;

  const gchar *fmt = gst_structure_get_string(s, "format");
  if (!fmt)
    fmt = "raw";

  ctx->width = (guint)w;
  ctx->height = (guint)h;
  g_free(ctx->format);
  ctx->format = g_strdup(fmt);
  return TRUE;
}

static void maybe_print_caps(const char *prefix, GstCaps *caps) {
  if (!caps)
    return;
  gchar *caps_str = gst_caps_to_string(caps);
  if (caps_str) {
    g_print("%s caps: %s\n", prefix, caps_str);
    g_free(caps_str);
  }
}

static GstFlowReturn on_new_sample(GstAppSink *appsink, gpointer user_data) {
  SinkCtx *ctx = (SinkCtx *)user_data;
  GstSample *sample = gst_app_sink_pull_sample(appsink);
  if (!sample)
    return GST_FLOW_OK;

  ctx->frames++;
  gint64 now_us = g_get_monotonic_time();

  GstCaps *caps = gst_sample_get_caps(sample);
  if (ctx->frames == 1) {
    maybe_print_caps(ctx->name, caps);
    ctx->start_us = now_us;

    if (ctx->kind == SINK_T2_FILE_SMALL || ctx->kind == SINK_T3_FILE_LARGE)
      (void)sink_set_video_info_from_caps(ctx, caps);
  }

  GstBuffer *buffer = gst_sample_get_buffer(sample);
  if (!buffer) {
    gst_sample_unref(sample);
    return GST_FLOW_OK;
  }

  GstMapInfo map;
  if (!gst_buffer_map(buffer, &map, GST_MAP_READ)) {
    g_printerr("%s: failed to map buffer\n", ctx->name);
    gst_sample_unref(sample);
    return GST_FLOW_OK;
  }

  if (ctx->kind == SINK_T1_JPEG_RING) {
    jpeg_ring_push_copy(ctx->jpeg_ring, map.data, map.size, now_us);
    if ((ctx->frames % 60) == 0) {
      g_print("%s: ring stored frames=%" G_GUINT64_FORMAT
              " (cap=%u) last_size=%zu\n",
              ctx->name, ctx->frames,
              ctx->jpeg_ring ? ctx->jpeg_ring->capacity : 0u, (size_t)map.size);
    }
  } else {
    /* write files to simulate "sending" */
    if (ctx->save_every == 0 || ctx->save_limit == 0) {
      /* disabled */
    } else if ((ctx->frames % ctx->save_every) == 0 &&
               ctx->saved < ctx->save_limit) {
      if ((ctx->width == 0 || ctx->height == 0) && caps)
        (void)sink_set_video_info_from_caps(ctx, caps);

      const gchar *fmt = ctx->format ? ctx->format : "raw";
      gchar *path = g_strdup_printf(
          "%s/%s_%06" G_GUINT64_FORMAT "_%ux%u_%s.bin", ctx->out_dir, ctx->name,
          ctx->saved + 1, ctx->width, ctx->height, fmt);
      if (path) {
        if (write_bytes_to_file(path, map.data, map.size)) {
          ctx->saved++;
          g_print("%s: wrote %s (bytes=%zu)\n", ctx->name, path,
                  (size_t)map.size);
        }
        g_free(path);
      }
    }
  }

  gst_buffer_unmap(buffer, &map);

  if ((ctx->frames % 120) == 0 && ctx->start_us > 0) {
    gdouble sec = (now_us - ctx->start_us) / 1000000.0;
    if (sec > 0.0)
      g_print("%s: frames=%" G_GUINT64_FORMAT ", fps=%.2f\n", ctx->name,
              ctx->frames, ctx->frames / sec);
  }

  gst_sample_unref(sample);
  return GST_FLOW_OK;
}

static gboolean on_bus_msg(GstBus *bus, GstMessage *msg, gpointer user_data) {
  (void)bus;
  AppCtx *app = (AppCtx *)user_data;

  switch (GST_MESSAGE_TYPE(msg)) {
  case GST_MESSAGE_ERROR: {
    GError *err = NULL;
    gchar *dbg = NULL;
    gst_message_parse_error(msg, &err, &dbg);
    g_printerr("ERROR from %s: %s\n", GST_OBJECT_NAME(msg->src),
               err ? err->message : "(null)");
    if (dbg)
      g_printerr("Debug details: %s\n", dbg);
    g_clear_error(&err);
    g_free(dbg);
    g_main_loop_quit(app->loop);
    break;
  }
  case GST_MESSAGE_EOS:
    g_print("EOS\n");
    g_main_loop_quit(app->loop);
    break;
  default:
    break;
  }

  return G_SOURCE_CONTINUE;
}

static gboolean on_sigint(gpointer user_data) {
  AppCtx *app = (AppCtx *)user_data;
  g_print("SIGINT: sending EOS...\n");
  if (app->pipeline)
    gst_element_send_event(app->pipeline, gst_event_new_eos());
  return G_SOURCE_CONTINUE;
}

static const gchar *pick_jpegenc(void) {
  if (gst_element_factory_find("nvjpegenc"))
    return "nvjpegenc";
  if (gst_element_factory_find("jpegenc"))
    return "jpegenc";
  return NULL;
}

int main(int argc, char *argv[]) {
  gst_init(&argc, &argv);

  const gchar *default_uri = "file:///opt/nvidia/deepstream/deepstream/samples/"
                             "streams/sample_720p.h264";
  const gchar *uri = (argc >= 2) ? argv[1] : default_uri;

  /* t2/t3 scaled output sizes */
  guint t2_w = env_u32_or_default("T2_WIDTH", 640);
  guint t2_h = env_u32_or_default("T2_HEIGHT", 360);
  guint t3_w = env_u32_or_default("T3_WIDTH", 1280);
  guint t3_h = env_u32_or_default("T3_HEIGHT", 720);

  guint ring_capacity = env_u32_or_default("T1_RING", 64);

  const gchar *out_dir = g_getenv("OUT_DIR");
  if (!out_dir || !*out_dir)
    out_dir = "./appsink_out";

  if (!ensure_out_dir(out_dir))
    return 1;

  guint t2_save_every = env_u32_or_default("T2_SAVE_EVERY", 30);
  guint t2_save_limit = env_u32_or_default("T2_SAVE_LIMIT", 30);
  guint t3_save_every = env_u32_or_default("T3_SAVE_EVERY", 30);
  guint t3_save_limit = env_u32_or_default("T3_SAVE_LIMIT", 30);

  const gchar *jpegenc = pick_jpegenc();
  if (!jpegenc) {
    g_printerr("No jpeg encoder found (need nvjpegenc or jpegenc).\n");
    return 1;
  }

  AppCtx app;
  memset(&app, 0, sizeof(app));

  app.loop = g_main_loop_new(NULL, FALSE);
  if (!app.loop) {
    g_printerr("Failed to create main loop\n");
    return 1;
  }

  jpeg_ring_init(&app.jpeg_ring, ring_capacity);

  app.t1 = (SinkCtx){.name = "t1",
                     .kind = SINK_T1_JPEG_RING,
                     .frames = 0,
                     .start_us = 0,
                     .saved = 0,
                     .save_every = 0,
                     .save_limit = 0,
                     .out_dir = g_strdup(out_dir),
                     .jpeg_ring = &app.jpeg_ring,
                     .width = 0,
                     .height = 0,
                     .format = NULL};
  app.t2 = (SinkCtx){.name = "t2",
                     .kind = SINK_T2_FILE_SMALL,
                     .frames = 0,
                     .start_us = 0,
                     .saved = 0,
                     .save_every = t2_save_every,
                     .save_limit = t2_save_limit,
                     .out_dir = g_strdup(out_dir),
                     .jpeg_ring = NULL,
                     .width = 0,
                     .height = 0,
                     .format = NULL};
  app.t3 = (SinkCtx){.name = "t3",
                     .kind = SINK_T3_FILE_LARGE,
                     .frames = 0,
                     .start_us = 0,
                     .saved = 0,
                     .save_every = t3_save_every,
                     .save_limit = t3_save_limit,
                     .out_dir = g_strdup(out_dir),
                     .jpeg_ring = NULL,
                     .width = 0,
                     .height = 0,
                     .format = NULL};

  gchar *pipeline_desc = g_strdup_printf(
      "nvurisrcbin uri=\"%s\" ! queue ! tee name=t "
      "t. ! queue ! nvvideoconvert ! video/x-raw,format=I420 ! %s ! appsink "
      "name=t1 emit-signals=true sync=false max-buffers=1 drop=true "
      "t. ! queue ! nvvideoconvert ! "
      "video/x-raw,format=RGBA,width=%u,height=%u ! appsink name=t2 "
      "emit-signals=true sync=false max-buffers=1 drop=true "
      "t. ! queue ! nvvideoconvert ! "
      "video/x-raw,format=RGBA,width=%u,height=%u ! appsink name=t3 "
      "emit-signals=true sync=false max-buffers=1 drop=true",
      uri, jpegenc, t2_w, t2_h, t3_w, t3_h);

  if (!pipeline_desc) {
    g_printerr("Failed to build pipeline string\n");
    g_free(app.t1.out_dir);
    g_free(app.t2.out_dir);
    g_free(app.t3.out_dir);
    jpeg_ring_free(&app.jpeg_ring);
    g_main_loop_unref(app.loop);
    return 1;
  }

  g_print("Pipeline:\n%s\n", pipeline_desc);

  GError *err = NULL;
  app.pipeline = gst_parse_launch(pipeline_desc, &err);
  g_free(pipeline_desc);

  if (!app.pipeline) {
    g_printerr("gst_parse_launch failed: %s\n",
               err ? err->message : "(unknown)");
    g_clear_error(&err);
    g_free(app.t1.out_dir);
    g_free(app.t2.out_dir);
    g_free(app.t3.out_dir);
    jpeg_ring_free(&app.jpeg_ring);
    g_main_loop_unref(app.loop);
    return 1;
  }

  GstElement *sink1 = gst_bin_get_by_name(GST_BIN(app.pipeline), "t1");
  GstElement *sink2 = gst_bin_get_by_name(GST_BIN(app.pipeline), "t2");
  GstElement *sink3 = gst_bin_get_by_name(GST_BIN(app.pipeline), "t3");

  if (!sink1 || !sink2 || !sink3) {
    g_printerr("Failed to get appsink elements by name (t1/t2/t3)\n");
    if (sink1)
      gst_object_unref(sink1);
    if (sink2)
      gst_object_unref(sink2);
    if (sink3)
      gst_object_unref(sink3);
    gst_object_unref(app.pipeline);
    g_free(app.t1.out_dir);
    g_free(app.t2.out_dir);
    g_free(app.t3.out_dir);
    jpeg_ring_free(&app.jpeg_ring);
    g_main_loop_unref(app.loop);
    return 1;
  }

  g_signal_connect(sink1, "new-sample", G_CALLBACK(on_new_sample), &app.t1);
  g_signal_connect(sink2, "new-sample", G_CALLBACK(on_new_sample), &app.t2);
  g_signal_connect(sink3, "new-sample", G_CALLBACK(on_new_sample), &app.t3);

  gst_object_unref(sink1);
  gst_object_unref(sink2);
  gst_object_unref(sink3);

  GstBus *bus = gst_element_get_bus(app.pipeline);
  gst_bus_add_watch(bus, on_bus_msg, &app);
  gst_object_unref(bus);

  g_unix_signal_add(SIGINT, on_sigint, &app);

  if (gst_element_set_state(app.pipeline, GST_STATE_PLAYING) ==
      GST_STATE_CHANGE_FAILURE) {
    g_printerr("Failed to set pipeline to PLAYING\n");
    gst_object_unref(app.pipeline);
    g_free(app.t1.out_dir);
    g_free(app.t2.out_dir);
    g_free(app.t3.out_dir);
    jpeg_ring_free(&app.jpeg_ring);
    g_main_loop_unref(app.loop);
    return 1;
  }

  g_main_loop_run(app.loop);

  gst_element_set_state(app.pipeline, GST_STATE_NULL);
  gst_object_unref(app.pipeline);

  g_free(app.t2.format);
  g_free(app.t3.format);

  g_free(app.t1.out_dir);
  g_free(app.t2.out_dir);
  g_free(app.t3.out_dir);

  jpeg_ring_free(&app.jpeg_ring);
  g_main_loop_unref(app.loop);
  return 0;
}

Hi ,

Unfortunately, we cannot upgrade neither jetpack nor deepstream version as of now.

Thanks a lot for your code. I will investigate the coding pattern and apply necessary changes to my code.

Do you know how to build gst cache inside docker container correctly? I tried few ways including , running a dummy pipeline first, mounting host cache into docker etc. But none of them seems to be working.

GStreamer always generates a cache the first time a GStreamer application runs, and externally introduced caches can cause some issues.

Usually, this doesn’t cause any problems, how does this affect your application?

Try copying registry.aarch64.bin from a fully built container and using it as a pre-cached registry.aarch64.bin . That might work, I’m not sure.

1 Like