DeepStream 6.1 new streammux CUDA unified vs. default problem


I am starting to migrate from DeepStream 6.0.1 to 6.1.
new nvstreammux doesn’t do any scaling anymore.
My pipeline is like this:

rtspsrc →
decodebin →
nvvideoconvert (nvbuf-memory-type=nvbuf-mem-cuda-unified, NV12->RGBA, NVMM, 1280x720) →
nvstreammux →
nvinfer (detector) →
nvinfer (classifier) -

My problem is that with 6.0.1, I got a CUDA unified buffer, because I set it in nvvideoconvert and nvstreammux.
With 6.1 nvstreammux does not have nvbuf-memory-type property anymore.
So I thought that setting nvbuf-memory-type=nvbuf-mem-cuda-unified in nvvideoconvert was enough to receive it as CUDA unified in appsink.

I always get the buffer as NVBUF_MEM_DEFAULT (that is NVBUF_MEM_CUDA_DEVICE).
So I cannot use NvBufSurfaceMap in appsink. (NvBufSurfaceMap according to the documentation only works with NVBUF_MEM_CUDA_UNIFIED for dGPU).

Is it normal to receive DEFAULT, when I set it to CUDA unified in the nvvideoconvert?
Normally new nvstreammux doesn’t touch the frame, is that correct?

I guess I have 2 solutions, use another nvvideoconvert after nvstreammux (don’t know if it’s possible), or use the NvBufSurface API to convert the buffer from DEFAULT to CUDA unified in the appsink.
Am I correct?

Thanks for the help!

nvbuf-memory-type is not supported by new nvstreammux. Gst-nvstreammux New — DeepStream 6.1.1 Release documentation

Why do you need the buffer?

You can remove the nvvideoconvert after decodebin and before the nvstreammux because nvstreammux can mux NV12 videos.

You can add nvvideoconvert after nvinfer.

I need the buffer to be RGBA, that’s why I put it before nvstreammux, the nvinfer models need the frame to be RGBA.
I want the buffer to do some stuff in appsink, that’s why I want to use CUDA unified, so I can get it in CPU memory easily.
What I don’t understand is why I set it to CUDA unified in nvvideoconvert before nvstreammux, and in appsink I receive a DEFAULT one. At some point the buffer type is changed, and that’s what I’d like to understand.

One question aside from the doubt I have, if I put nvvideoconvert just after nvstreammux, and before the first nvinfer, if I want to convert from NV12 to RGBA, will it convert all the frames in the batch?

About my last question, just saw the documentation:

  • Inputs
    • Gst Buffer batched buffer
    • NvDsBatchMeta
    • Format: NV12, I420, P010_10LE, BGRx, RGBA (NVMM/RAW)

Do you mean your model input layer needs RGBA but not RGB?
nvinfer supports NV12 input, if the model needs RGB/BGR input, just set “model-color-format” parameter in the nvinfer configuration file to correct value is enough. It has nothing to do with the nvinfer input format. Gst-nvinfer — DeepStream 6.1.1 Release documentation

You must make sure nvstreammux’s “width” and “height” properties are the same as the video resolution. Gst-nvstreammux New — DeepStream 6.1.1 Release documentation
Sometimes when there are multiple input streams with different resolutions, the nvstreammux must scale all streams to the same resolution and make the batch, so it is a new buffer but not the one from video decoder.

You don’t have to convert the format to RGBA before nvinfer. If you do so, it will convert all frames in the batch.

Model input is RGB yes.
I cannot use width and height, in the new streammux they are absent, and theoretically new streammux does not scale.

Do you know why the buffer comes as NVBUF_MEM_DEFAULT in my appsink, when I set it to nvbuf-mem-cuda-unified in nvvideoconvert before streammux. Does new streammux modify the buffer?

For testing I created a nvvideoconvert after streammux like this:

And I still get NVBUF_MEM_DEFAULT and so NvBufSurfaceMap() is failing.

The same happens if I put the nvvideoconvert with CUDA unified just before the appsink:

Do you have any idea of how I can have a CUDA unified buffer in the appsink?

Just tried with old streammux, I get correctly a CUDA Unified buffer of all the frames in the batch:

I think I’ve give the solution in my previous post, you need to remove the nvvideoconvert after decodebin and move it after nvstreammux.

For the nvvideoconvert, the input resolution and format are just the same as the output resolution and format, the buffer(data) will pass by the plugin.

I tried to move it after streammux.
I also tried to move it just before appsink.
And in appsink I cannot receive the buffer as CUDA Unified. I always receive it as NVBUF_MEM_DEFAULT, that is CUDA device in my case as I’m using dGPU.

Can you post the code?

I am preparing a simple program then I’ll post you the code.