• Hardware Platform (Jetson / GPU) : NVIDIA Jetson AGX Orin
• DeepStream Version : 7.1
• JetPack Version (valid for Jetson only) : 6.1
• TensorRT Version : 8.6.2.3
• Issue Type( questions, new requirements, bugs) : question
Hello,
I have a DeepStream pipeline performing inference using a classifier model. In my nvinfer
configuration file, I specified: infer-dims=3;224;224
. The input image to the network is of shape (3, 1440, 1440), and I expect nvinfer
element to resize it to (3, 224, 224) before inference. The input is 1440x1440 because before the nvinfer
plugin there is nvvideoconvert
which uses src-crop
to crop 4k image to 1440x1440. However, when extracting frames from the nvinfer
output in my pad probe function (src), I retrieve frames of shape (1440, 1440, 4) instead of (224, 224, 4).
Here is the function I use to extract frames:
def get_frame(gst_buffer: Gst.Buffer, batch_id: int) -> np.ndarray:
"""
Get the frame from the gst_buffer
"""
# Get the NvBufSurface from the gst_buffer (Surface containing image data)
n_frame = pyds.get_nvds_buf_surface(hash(gst_buffer), batch_id)
# Convert to numpy array
frame_image = np.array(n_frame, copy=True, order="C")
# Free up the memory associated with the buffer
pyds.unmap_nvds_buf_surface(hash(gst_buffer), batch_id)
return frame_image
Since the extracted frame retains the original 1440x1440 resolution, my subsequent processing becomes inefficient. For example, my latter operations are adding an alpha channel:
alpha = np.full((img.shape[0], img.shape[1]), 255, dtype=np.uint8)
img = np.dstack((img, alpha))
This operation takes 2-3 ms for an image of 224x224, but 40 ms for 1440x1440, making it a major bottleneck.
Questions:
- How can I efficiently extract the resized 224x224 frame that the inference model already operates on?
- Is there a way to access the preprocessed image directly instead of the original input size?
- Are there any special DeepStream/Cuda functions that will make it efficient to operate on arrays instead of using numpy which I believe uses CPU instead of using GPU?