Custom network input using nvdspreprocess

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 6.0
• JetPack Version (valid for Jetson only)
• TensorRT Version 8.2.3.0
• NVIDIA GPU Driver Version (valid for GPU only) 470.141.03
• Issue Type( questions, new requirements, bugs) Question

Hi,

I’m trying to feed a six-channel input tensor to nvinfer using nvdspreprocess custom lib. I tried to understand how tensors are prepared and attached as metadata running the folloing example pipeline:
gst-launch-1.0 filesrc location = /opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvvideoconvert ! nvdspreprocess config-file= /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/config_preprocess.txt ! nvinfer config-file-path= /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt input-tensor-meta=1 batch-size=7 ! nvmultistreamtiler width=1920 height=1080 ! nvvideoconvert ! nvdsosd ! nveglglessink

In the function NvDsPreProcessTensorImpl::prepare_tensor of the provided nvdspreprocess_lib, I tried to visualize batch->units[i].converted_frame_ptr to understand it better but what I found weird is that for batch->units[i].converted_frame_ptr to be interpreted correctly, I have to consider it to be of pitch of batch->pitch (2028 in my case) which is different from what I expected m_NetworkSize.width * m_NetworkSize.channels (640 * 3 = 1920). Can you please provide some insights on why this is the case?

Also, in the same function, for the pointer void*& devBuf, which carries the output tensors after processing, I can see it comes form buf = acquirer->acquire() but what I don’t understand is where its allocation size is defined and how it is managed.

Thanks!

3

1.====>Can you please provide some insights on why this is the case?
The batch structure comes from gstnvdspreprocess.cpp. It ralated with the nvbufsurface.

batch->pitch = memory->surf->surfaceList[0].planeParams.pitch[0];

2.====>buf = acquirer->acquire()
You can refer the gstnvdspreprocess.cpp

NvDsPreProcessCustomBuf* NvDsPreProcessAcquirerImpl::acquire()
4

Hi,

Thanks for your reply,

I see that the batch structure comes from nvstreammux, but what makes it counterintuitive for me is what is mentioned in /../../gst-nvdspreprocess/include/nvdspreprocess_interface.h about converted_frame_ptr, it says:

Pointer to the converted frame memory. This memory contains the frame converted to RGB/RGBA and scaled to network resolution.
 This memory is given to Output loop as input for mean subtraction and normalization and Tensor Buffer formation for inferencing.

This means to me that data pointed by converted_frame_ptr has to assume the network input shape and consequently its pitch can be calculated given the tensor shape and its datatype. How the pitch is different than expected and the image still displays correctly?

Thanks!

5

The pitch is related to the surface. You can refer the nvbufsuface.h. It uses different algorithms according to different color spaces of specific images to get the value. This part isn’t open source so far. You can just refer the structure in the head file.

6

I see, I traced image format back and it was RGB, so what I’m expecting is it has 3 channels with each pixel requiring 1 byte(unsigned char) * 3, this gives a pitch of 3 * m_NetworkSize.width which is differnt than from batch->pitch. My issue is that I need to create a similar pointer to converted_frame_ptr which will hold another frame, I need both frames to be of the same format to be able to stack them together.

In other words, if I have and opencv image for example, how I can convert it to a converted_frame_ptr-like pointer?

Thanks!

7

There is no update from you for a period, assuming this is not an issue anymore.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

As you can see from the source code, we use gstbufferpool for unified memory management. So we just get memory from the pool. It doesn’t exactly match the format. You should calculate by the pitch and the image format when you use the memory. You can refer the link below to learn how to convert the opencv memory with nvbufsurface.
https://forums.developer.nvidia.com/t/deepstream-sample-code-snippet/142683/3

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.