Can nvarguscamerasrc and RTSP sources work together in DeepStream?

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Can nvarguscamerasrc and RTSP sources work together in DeepStream?

Environment

  • Hardware: Jetson AGX Orin

  • JetPack: 6.2

  • DeepStream: 7.1

  • Camera: ZED X (via nvarguscamerasrc)

  • RTSP: Dahua/Axis IP cameras (~10fps H.264)

Problem Summary

We cannot get nvarguscamerasrc and RTSP sources to work together in a single DeepStream pipeline. After 50+ configuration attempts across both OLD and NEW nvstreammux, no combination produces acceptable results.

What works individually:

  • ZED X alone → Works ok

  • RTSP alone → Works ok

  • Multiple RTSP cameras together → Works ok

What doesn’t work:

  • ZED X + RTSP together → Always has issues (micro-freezes, latency, dropped frames)

What We Tried

| Attempt | Configuration | Result |

|---------|--------------|--------|

| OLD mux, sync-inputs=0 | Default settings | RTSP appears but minutes behind real-time |

| OLD mux, sync-inputs=1 | Synchronized batching | RTSP frames dropped entirely |

| NEW mux, sync-inputs=0 | `USE_NEW_NVSTREAMMUX=yes` With `attach-sys-ts=TRUE` | RTSP works, ZED X terrible FPS |

| NEW mux + scaling | Pre-mux nvvidconv+capsfilter | RTSP latency is bad |

| Buffer tuning | 40+ combinations of queue/latency settings | No improvement |

| FPS matching | ZED X sensor-mode=3 (30fps) | Better ratio, still broken |

| nvurisrcbin | over stantandard rtspsrc | Glitching issues |

Question

Is mixing nvarguscamerasrc with RTSP sources in a single nvstreammux pipeline a supported use case?

If yes:

  • What is the correct configuration?

  • Are there example pipelines or reference applications?

If no:

  • Is this a known limitation?

  • What is the recommended architecture for applications that need both local cameras and RTSP streams?

Minimal Pipeline Example


# This is what we're trying to achieve:

nvarguscamerasrc → nvvidconv → nvstreammux (pad 0) ─┬─→ nvinfer → output

rtspsrc → decodebin → nvvidconv → nvstreammux (pad 1) ─┘

Any guidance would be appreciated. We’ve invested significant time trying to make this work and are considering abandoning DeepStream if this is not a supported configuration.

4

nvstreammux is used to form a batch of frames from multiple input sources. there is no source type limitation for nvstreammux.

  1. regarding " * ZED X + RTSP together → Always has issues (micro-freezes, latency, dropped frames)", what is the complete media pipeline? could you share the configurations of nvurisrcbin and nvstreammux?
  2. to narrow down the issue, if using the following cmd, will the issue persist?
nvarguscamerasrc → nvvidconv → nvstreammux (pad 0) ─┬─tiler-> nv3dsink (sync=0)

rtspsrc → decodebin → nvvidconv → nvstreammux (pad 1) ─┘
  1. ‘nvvidconv’ is not DeepStream element, please use ‘nvvideoconvert’ instead with other DeepStream elements. Please refer to this topic.
5

Thank you for the response! Here’s a detailed breakdown of my pipeline structure:

Pipeline Overview:

My pipeline processes multiple video sources (1 ZED X camera via nvarguscamerasrc + 3 RTSP streams) through a shared inference path, then demultiplexes back to individual streams for recording and RTSP output.

Main Pipeline Structure:

  1. Source Stage:
  • nvarguscamerasrc → capsfilter (640x360, NV12, NVMM) → nvstreammux sink_0
  • RTSP chain: rtspsrc → rtph264depay → h264parse → nvv4l2decoder → queue → nvstreammux sink_1
  1. Processing Stage:
  • nvstreammux (batch-size=2, batched-push-timeout=40000) → nvinfer (PGIE) → nvtracker (optional) → nvvideoconvert → nvstreamdemux
  1. Per-Source Output Branches (dynamically created for each source):
  • nvstreamdemux src_N → nvdsosd_N → branching logic:
    • For main_cam: tee splits to:
      • RTSP branch: queue → nvv4l2h265enc → rtspclientsink (rtsp://127.0.0.1:8554/main_cam)
      • Recording branch: queue → nvv4l2h264enc (iframeinterval=1, insert-sps-pps=1, idrinterval=1) → h264parse → Smart Record bin
    • For RTSP sources: Direct path: nvdsosd_N → queue → nvv4l2h264enc → h264parse → Smart Record bin

Key Properties:

  • nvstreammux: sync-inputs=1, live-source=1, width=640, height=360
  • nvv4l2h264enc (recording): bitrate=4000000, strict keyframe settings for Smart Record compatibility
  • nvv4l2h265enc (RTSP): bitrate=1000000
  • rtspsrc: protocols=TCP
  • All queues: max-size-time=500ms to limit latency

Dynamic Components:

  • Pad Probes:
    1. streamdemux_sink_pad_buffer_probe: Pre-demux probe for ROI filtering, object counting, and triggering Smart Record events
    2. encode_probe: Post-tracker/PGIE probe for capturing full-frame JPEG snapshots on detection/FOV requests
  • Smart Record: Triggered via g_timeout_add callbacks when new objects detected, records MP4 clips (default 20s duration, 10s pre-buffer)
  • Dynamic Pad Linking: rtspsrc uses pad-added signal to link dynamically to rtph264depay

Synchronization:

  • All dynamically added elements use gst_element_sync_state_with_parent()
  • Demux output branches pre-allocated during initialization before setting pipeline to PLAYING

The issue manifests as frame drops on RTSP sources (not the Argus camera) after ~30 minutes of runtime. All sources share the same nvstreammux and inference path, so I’m unsure why only RTSP sources are affected.

6

Thanks for the sharing! Here are some methods to narrow down the issue.

  1. what’s the argus camera’s fps? Is there mossic/articrafts, which means decoding failed, in the output video? or there is only frame dropping in the output video? Plesase refer to this faq for how to set batched-push-timeout and other parameters for nvstreammux.
  2. noticing batch-size of nvstreammux is set to 2, for a better perforamnce, please set batch-size of nvstreammux, and nvinfer to source’s number.
  3. the default latency of rtsprc is 2000ms, and the default drop-on-latency is false. if the tcp receivng is not good, the frames will be dropped by nvstreammux because the buffers are too late. you can use “export GST_DEBUG=3,nvstreammux:6” to check nvstream’s logs. Hence, you can try rtp receiving video and lower the latency.
7

Sorry I wrote that wrong. Issue is not frame drops that was for my old configuration. With the above configuration issue is rtsp starts late and latency increase over time. There are no frame drops in above configuration.
1- camera fps is 30. there is no mosaics. When it runs with RTSP together video output fps lowers and there are micro freezes and/or slowing.
2- Its two because argus + rtsp is the problematic case. 1 camera for each type works. 4 rtsp together works.
3- sorry for wrong info rtsp frames are not dropped in this case.

8

as written in the faq above, If you want to get high fps, set the batched-push-timeout to the (1000000 us/maximum fps among the videos). the correct batched-push-timeout should be 1000000/max fps = 1000000/30 = 33000. the fps of rtsp source is low(10fps) while camera’s fps is high, when batched-push-timeout is too big(40000), nvstreammux will wait sometimes to form batched data due to low rtsp fps.

9

fps is good, rtsp is behind real time is what I was trying to say.

10

Regarding “rtsp is behind real time”? do you mean playing rtsp://127.0.0.1:8554/main_cam has delay and latency increase over time? Is the only problem now the delay? There are some steps to narrow down.

  1. if using nv3dsink instead of “queue → nvv4l2h265enc->…” in RTSP branch, will the dealy issue persist? wondering if the issue is related to rtspserver.
  2. what values did you set for rtspsrc’s latency, drop-on-latency and nvstreammux’s batched-push-timeout?
11 
g_object_set(G_OBJECT(streammux), "batch-size", 2, "batched-push-timeout", 33000, "attach-sys-ts", true, NULL);

here is function for rtsp sources, this connects to sterammux:


void Pipeline::create_configure_rtsp_sources() {
  if (source_urls.empty()) return;

  for (size_t i = 0; i < source_urls.size(); ++i) {
    const std::string &url = source_urls[i];

    if (url == "main_cam") {
      // handled by nvarguscamerasrc
      continue;
    }

    guint slot = i;

    g_print("Creating RTSP pipeline for slot %u : %s\n", slot, url.c_str());

    // --- Create elements ---
    GstElement *rtspsrc = gst_element_factory_make("rtspsrc", NULL);
    GstElement *depay = gst_element_factory_make("rtph264depay", NULL);
    GstElement *parser = gst_element_factory_make("h264parse", NULL);
    GstElement *decoder = gst_element_factory_make("nvv4l2decoder", NULL);
    GstElement *q_post = gst_element_factory_make("queue", NULL);

    if (!rtspsrc || !depay || !parser || !decoder || !q_post) {
      g_printerr("Failed creating RTSP decode elements for source %zu\n", i);
      continue;
    }

    // --- Configure rtspsrc ---
    g_object_set(G_OBJECT(rtspsrc), "location", url.c_str(), "protocols", (guint)GST_RTSP_LOWER_TRANS_TCP, NULL);

    g_object_set(G_OBJECT(decoder), "enable-max-performance", TRUE, NULL);

    g_object_set(G_OBJECT(q_post), "max-size-buffers", 0,  // Unlimited buffers (let time control it)
                 "max-size-bytes", 0,                      // Unlimited bytes
                 "max-size-time", 0,               // 500ms buffer (holds ~15 frames @ 30fps)
                 NULL);

    // --- Add them to the pipeline ---
    gst_bin_add_many(GST_BIN(pipeline), rtspsrc, depay, parser, decoder, q_post, NULL);

    // --- Link static parts: depay → parse → decoder → queue ---
    if (!gst_element_link_many(depay, parser, decoder, q_post, NULL)) {
      g_printerr("Failed to link decode chain for RTSP source %zu\n", i);
      continue;
    }

    // --- Dynamic pad: rtspsrc.pad-added → depay.sink ---
    g_signal_connect(rtspsrc, "pad-added", G_CALLBACK(+[](GstElement *src, GstPad *pad, gpointer user_data) {
                       GstElement *depay = GST_ELEMENT(user_data);
                       GstPad *sinkpad = gst_element_get_static_pad(depay, "sink");
                       if (gst_pad_link(pad, sinkpad) != GST_PAD_LINK_OK) {
                         g_printerr("Failed linking rtspsrc → depay\n");
                       } else {
                         g_print("Linked rtspsrc → rtph264depay\n");
                       }
                       gst_object_unref(sinkpad);
                     }),
                     depay);

    // --- Connect decoded NVMM frames to streammux ---
    gchar padname[16];
    g_snprintf(padname, sizeof(padname), "sink_%u", slot);

    GstPad *mux_sink_pad = gst_element_request_pad_simple(streammux, padname);
    GstPad *queue_src_pad = gst_element_get_static_pad(q_post, "src");

    if (gst_pad_link(queue_src_pad, mux_sink_pad) != GST_PAD_LINK_OK) {
      g_printerr("Failed linking queue → streammux.%s\n", padname);
    } else {
      g_print("Linked decoder → streammux.%s\n", padname);
    }

    gst_object_unref(queue_src_pad);
    gst_object_unref(mux_sink_pad);
  }
}

Removed rtsp branch but it didnt help. I couldn’t succesfully connect 3dsink though so I checked the videos instead.

12

I have tried with deepstream-app -c config.txt where config is:

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5

[tiled-display]
enable=1
rows=1
columns=2
width=1920
height=540

[streammux]
batch-size=2
batched-push-timeout=40000
width=1280
height=720
live-source=1
enable-padding=1

[source0]
enable=1
type=5
camera-width=1280
camera-height=720
camera-fps-n=30
camera-fps-d=1

[source1]
enable=1
type=4
uri=rtsp://admin:noviTest_42@192.168.88.181/cam/realmonitor?channel=1&subtype=0
latency=200

[primary-gie]
enable=1
batch-size=1
config-file=/home/jensen/deepstream-factorymonitor/configs/config_infer_primary_rtdetr.txt
interval=0

[tracker]
enable=1
ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
ll-config-file=/home/jensen/deepstream-factorymonitor/configs/config_nvtracker.yml
gpu-id=0
tracker-width=640
tracker-height=640

[osd]
enable=1

[sink0]
enable=1
type=2
sync=0

here is a video:


this is not the problem Im facing with my own custom pipeline. Just sharing because I couldn’t find a working sample for both of these together. In my custom pipeline using nvurisrcbin oppose to rtspsrc would result in similiar glitch.

I can share my custom pipeline with dm too if you like.

13

I used nv3dsink and commented all things after demux and rtsp and arguscamerasrc worked fine, with a few rtspsrc queue parameters settings that is. No micro freezes on argus and rtsp had no delay on the display.

14

from your latest test, the issue should be related to remaining elements in RTSP branch, that is, " nvv4l2h265enc → rtspclientsink". To narrow down the nvv4l2h265enc’ issue, you can use "queue->nvv4l2h265enc->nvv4l2decoder-> nv3dsink’ in RTSP branch, if the output video is fine(no freeze, no delay), the issue should be related tortspclientsink and rtspserver.

15

I removed all rtsp stream output elements to test. but micro freezes in the video continues to happen. Could it be related to image saving in probe or video saving logic?

16
  1. what do you mean by " image saving in probe or video saving logic"? To narrow down this issue, if removing this code, will the issue peresit?
  2. could you share the result of test in my last comment? noticing “I used nv3dsink and commented all things after demux and rtsp and arguscamerasrc worked fine”.
17
    1. encode_probe: Post-tracker/PGIE probe for capturing full-frame JPEG snapshots on detection/FOV requests
      so I have this probe attacching to tracker:

GstPadProbeReturn Pipeline::encode_probe(GstPad *pad, GstPadProbeInfo *info) {
  if (shutting_down) return GST_PAD_PROBE_OK;
  GstBuffer *buf = static_cast<GstBuffer *>(info->data);
  if (!buf || !obj_ctx_handle) {
    g_printerr("[encode_probe] Buffer or JPEG context is null\n");
    return GST_PAD_PROBE_OK;
  }

  GstMapInfo inmap = {0};
  if (!gst_buffer_map(buf, &inmap, GST_MAP_READ)) {
    g_printerr("[encode_probe] Failed to map buffer\n");
    return GST_PAD_PROBE_OK;
  }

  // <-- changed here to reinterpret_cast -->
  NvBufSurface *surf = reinterpret_cast<NvBufSurface *>(inmap.data);
  if (!surf) {
    g_printerr("[encode_probe] NvBufSurface is null\n");
    gst_buffer_unmap(buf, &inmap);
    return GST_PAD_PROBE_OK;
  }

  NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta(buf);
  if (!batch_meta) {
    g_printerr("[encode_probe] Batch meta is null\n");
    gst_buffer_unmap(buf, &inmap);
    return GST_PAD_PROBE_OK;
  }

  for (NvDsMetaList *l_frame = batch_meta->frame_meta_list; l_frame; l_frame = l_frame->next) {
    NvDsFrameMeta *frame_meta = static_cast<NvDsFrameMeta *>(l_frame->data);
    if (!frame_meta) continue;

    guint source_id = frame_meta->source_id;

    // --- Full‐frame FOV snapshot logic (mutex‐protected) ---
    SnapshotInfo snapshot_info;
    {
      std::lock_guard<std::mutex> lock(fov_mutex);
      if (capture_flags[source_id]) {
        capture_flags[source_id] = false;
        snapshot_info = fov_snapshots[source_id];
      } else if (detection_capture_flags[source_id]) {
        detection_capture_flags[source_id] = false;
        snapshot_info = detection_snapshots[source_id];
      }
    }
    if (!snapshot_info.path.empty()) {
      NvDsObjEncUsrArgs frame_args = {0};
      frame_args.isFrame = 1;
      frame_args.saveImg = TRUE;
      frame_args.attachUsrMeta = TRUE;
      frame_args.quality = 90;
      std::snprintf(frame_args.fileNameImg, sizeof(frame_args.fileNameImg), "%s", snapshot_info.path.c_str());

      nvds_obj_enc_process(obj_ctx_handle, &frame_args, surf, nullptr, frame_meta);
      nvds_obj_enc_finish(obj_ctx_handle);
      if (!snapshot_info.is_fov_req) {
        on_file_save_completed(snapshot_info.path.c_str(), callsigns[source_id], snapshot_info.is_init);
        if (snapshot_info.is_init) {
          is_fov_saved[source_id] = true;
        }
      } else {
        is_fov_saved[source_id] = true;  // Only signal Redis-waiting code
      }
    }
  }
  gst_buffer_unmap(buf, &inmap);
  return GST_PAD_PROBE_OK;
}

and I also have video saving after demux. Demux split into 2 branches. main_cam has rtsp stream output (for live stream) and recording.

rtsp branches has:

nvdsosd_N → queue → nvv4l2h264enc → h264parse → Smart Record bin

I removed the demux and probes. instead put a nvidia tiler and nv3dsink. I saw that both streams were real time with stable fps. Arguscam also had no those unstable fps micro slowing issue.

18

regarding "I removed all rtsp stream output elements to test. but micro freezes in the video continues to happen. ", how did you obeserve the freeze? do you mean playing rtsp://127.0.0.1:8554/main_cam generating byrtspclientsink have freeze? do the files generated by smart recording have freeze issue? If playing rtsp://127.0.0.1:8554/main_cam on the machine running the deepstream app, will the issue persist? wondering if the issue is related to the network.