Nvstreamdemux does not read the output

I have a custom deepstream camera element that outputs 3 images in a batched buffer.
I create my output buffer and metadata like this.

NvBufSurface *frames_surf[framesCount];
for (int i = 0; i < framesCount; i++)
{
    NvArgusFrameInfo *consumerFrameInfo = consumerFrameInfos->at(i);
    if (&consumerFrameInfo->fd == NULL)
    {
        goto done;
    }
    retn = NvBufSurfaceFromFd(consumerFrameInfo->fd, (void **)(&frames_surf[i]));
    if (retn != 0)
    {
        GST_ERROR_OBJECT(src, "NvBufSurfaceFromFd Failed");
        goto done;
    }
}

GstMapInfo outmap = GST_MAP_INFO_INIT;
gst_buffer_map(buffer, &outmap, GST_MAP_WRITE);
NvBufSurface *out_surf = (NvBufSurface *)outmap.data;

GST_INFO_OBJECT(src, "NvBufSurface + batch size %d\n", out_surf->batchSize);
GST_HAKOB_PRINT("NvBufSurface + batch size %d\n", out_surf->batchSize);

g_assert(out_surf->batchSize == src->sensors_num);
g_assert(out_surf->batchSize >= consumerFrameInfos->size());

NvBufSurface in_surf = {0};
NvBufSurfaceParams surfaceList[out_surf->batchSize];
in_surf.surfaceList = &surfaceList[0];
in_surf.batchSize = out_surf->batchSize;
in_surf.numFilled = framesCount;
for (int i = 0; i < framesCount; i++)
{
    in_surf.surfaceList[i] = frames_surf[i]->surfaceList[0];
}

retn = NvBufSurfTransform(&in_surf, out_surf, &src->transform_params);
if (retn != 0)
{
    GST_ERROR_OBJECT(src, "NvBufSurfTransform Failed");
    /* TODO: Check if need to set ->stop_requested flag in error condition */
    goto done;
}
NvDsBatchMeta *batch_meta = nvds_create_batch_meta(src->sensors_num);
g_assert(batch_meta);
NvDsMeta *nvds_meta = gst_buffer_add_nvds_meta(buffer, batch_meta, NULL,
                                               nvds_batch_meta_copy_func, nvds_batch_meta_release_func);
g_assert(nvds_meta);
nvds_meta->meta_type = NVDS_BATCH_GST_META;
batch_meta->base_meta.batch_meta = batch_meta;
batch_meta->base_meta.copy_func = nvds_batch_meta_copy_func;
batch_meta->base_meta.release_func = nvds_batch_meta_release_func;
batch_meta->max_frames_in_batch = src->sensors_num;
// batch_meta->num_frames_in_batch = framesCount;



for (int i = 0; i < framesCount; i++)
{
    GST_DEBUG_OBJECT(src, "consumer fill frame meta %d", i);
    NvArgusFrameInfo *frameInfo = consumerFrameInfos->at(i);
    // attach frame meta
    NvDsFrameMeta *frame_meta = nvds_acquire_frame_meta_from_pool(batch_meta);
    g_assert(frame_meta);
    g_assert(frame_meta->base_meta.batch_meta == batch_meta);
    frame_meta->pad_index = i;
    frame_meta->batch_id = i;
    frame_meta->frame_num = frameInfo->frameNum;
    frame_meta->buf_pts = frameInfo->frameTime;
    frame_meta->ntp_timestamp = get_ntp_timestamp();
    frame_meta->source_id = src->sensors[i];
    frame_meta->num_surfaces_per_frame = 1;
    // TODO : remove hardcoding
    frame_meta->source_frame_width = 4032;
    frame_meta->source_frame_height = 3040;

    nvds_add_frame_meta_to_batch(batch_meta, frame_meta);
}
if (batch_meta->num_frames_in_batch != framesCount)
{
    GST_ERROR_OBJECT(src, "consumer fill output batch meta failed");
    break;
}

This element is attached to nvstreamdemux, but for some reason the output from nvstreamdemux is empty.

I got this error message from the logs I am getting this from my nvstreamdemux element

GST_CAPS gstpad.c:2704:gst_pad_has_current_caps:demux:src_0 check current pad caps (NULL)
GST_CAPS gstpad.c:2704:gst_pad_has_current_caps:demux:src_1 check current pad caps (NULL)
GST_CAPS gstpad.c:2704:gst_pad_has_current_caps:demux:src_2 check current pad caps (NULL)

Here is my full pipeline, where nvarguscamerahakob is my custom element/

gst-launch-1.0 -e nvarguscamerahakob silent=true ! "video/x-raw(memory:NVMM),format=(string)RGBA,width=(int)4032,height=(int)3040,framerate=(fraction)29/1" !  queue ! \
nvstreamdemux name=demux \
demux.src_0 ! nvv4l2h264enc bitrate=60000000 insert-sps-pps=true ! h264parse ! qtmux ! filesink location=video1.mp4 \
demux.src_1 ! nvv4l2h264enc bitrate=60000000 insert-sps-pps=true ! h264parse ! qtmux ! filesink location=video2.mp4 \
demux.src_2 ! nvv4l2h264enc bitrate=60000000 insert-sps-pps=true ! h264parse ! qtmux ! filesink location=video3.mp4

For testing purposes, I created another custom element and was able to fully read the output, which proves the images are going to the downstream component.

Can you please help me to understand how to debug this and find out what is going wrong?

Thanks

I don’t think the nvv4l2h264enc can encode RGBA videos. https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvvideo4linux2.html#encoder

Please add nvvideoconvert https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvvideoconvert.html before nvv4l2h264enc. And the queue before nvstreamdemux should be removed and please add queue after the “demux.src_x”.

Thank you for your quick response.

I tried this, and still the output is empty.

gst-launch-1.0 -e nvarguscamerahakob silent=true ! "video/x-raw(memory:NVMM),format=(string)NV12,width=(int)4032,height=(int)3040,framerate=(fraction)29/1" !   \
nvstreamdemux name=demux \
demux.src_0 ! queue ! nvvideoconvert ! "video/x-raw(memory:NVMM),format=(string)NV12" ! nvv4l2h264enc bitrate=60000000 insert-sps-pps=true ! h264parse ! qtmux ! filesink location=video1.mp4 \
demux.src_1 ! queue ! nvvideoconvert ! "video/x-raw(memory:NVMM),format=(string)NV12" ! nvv4l2h264enc bitrate=60000000 insert-sps-pps=true ! h264parse ! qtmux ! filesink location=video2.mp4 \
demux.src_2 ! queue ! nvvideoconvert ! "video/x-raw(memory:NVMM),format=(string)NV12" ! nvv4l2h264enc bitrate=60000000 insert-sps-pps=true ! h264parse ! qtmux ! filesink location=video3.mp4

I also tried this

gst-launch-1.0 -e nvarguscamerahakob silent=true ! "video/x-raw(memory:NVMM),format=(string)NV12,width=(int)4032,height=(int)3040,framerate=(fraction)29/1" !   \
nvstreamdemux name=demux \
demux.src_0 ! queue ! fakevideosink \
demux.src_1 ! queue ! fakevideosink \
demux.src_2 ! queue ! fakevideosink

And I still see these debug messages.

GST_CAPS gstpad.c:2704:gst_pad_has_current_caps:demux:src_0 check current pad caps (NULL)
GST_CAPS gstpad.c:2704:gst_pad_has_current_caps:demux:src_1 check current pad caps (NULL)
GST_CAPS gstpad.c:2704:gst_pad_has_current_caps:demux:src_2 check current pad caps (NULL)

I think something goes wrong in the cap negotiations, but I can’t find any way to debug what exactly is the reason for failure from nvstreamdemux should be.

The “nvarguscamera + nvstreammux” is already supported by DeepStream.

You need to debug your own customization by yourself.

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks