How to Split nvinfer Output Tensor (batch_size, n, n, channels, height, width) into batch_size * n * n Sub-Tensors for Subsequent Inference

[john_c]

• Hardware Platform (Jetson / GPU) Jetson AGX Orin
• DeepStream Version 7.1
• JetPack Version (valid for Jetson only) 6.2.1
• TensorRT Version 10.3
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs) questions

Hi,

In my DeepStream pipeline, the output tensor of an nvinfer model has the shape (batch_size, n, n, channels, height, width). I need to split this tensor into batch_size * n * n sub-tensors of shape (channels, height, width) to feed into a subsequent model for inference. Is there any DeepStream component or plugin that can handle this tensor reshaping and splitting process directly? If not, what is the recommended approach to achieve this within the DeepStream framework?

Any guidance or examples would be greatly appreciated!

Thanks,

[junshengy]

You can refer to deepstream_pose_classification_app in deepstream_tao_apps.

apps/tao_others/deepstream-pose-classification/deepstream_pose_classification_app.cpp

For this sample:
In the parse_25dpose_from_tensor_meta function, first parse the NVDSINFER_TENSOR_OUTPUT_META as NVDS_OBJ_META ,Then nvdspreprocess will process NVDS_OBJ_META into NVDS_PREPROCESS_BATCH_META as the input of subsequent model.

For your application, you can split NVDSINFER_TENSOR_OUTPUT_META into sub-tensors in the custom processing library of nvdspreprocess

[john_c]

Thank you for your timely reply! I looked at the reference code you posted. In the sgie src probe, it parses NVDSINFER_TENSOR_OUTPUT_META as NVDS_OBJ_META, then in nvdspreprocess, it processes NVDS_OBJ_META into NVDS_PREPROCESS_BATCH_META as the input for sgie1. Like this:

sgie -> sgie_src_pad_buffer_probe -> preprocess1 -> sgie1

But in my case, the sgie output is (batch_size, n, n, channels, height, width), and batch_size x n x n might be relatively large, for example, 1 x 9 x 9 = 81, which exceeds the maximum batch size of sgie1 (possibly 8). In this case, how should I handle it?

[junshengy]

@john_c

Thank you for your timely reply! I looked at the reference code you posted. In the sgie src probe, it parses NVDSINFER_TENSOR_OUTPUT_META as NVDS_OBJ_META, then in nvdspreprocess, it processes NVDS_OBJ_META into NVDS_PREPROCESS_BATCH_META as the input for sgie1. Like this:

You can directly process NVDSINFER_TENSOR_OUTPUT_META into NVDS_PREPROCESS_BATCH_META in the custom library of nvdspreprocess

@john_c

But in my case, the sgie output is (batch_size, n, n, channels, height, width), and batch_size x n x n might be relatively large, for example, 1 x 9 x 9 = 81, which exceeds the maximum batch size of sgie1 (possibly 8). In this case, how should I handle it?

Generally speaking, there will be no problem. You can refer to gst_nvinfer_process_tensor_input function in /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvinfer/gstnvinfer.cpp. Alternatively, you can increase the batch-size in the sgie1 configuration file.

[john_c]

@junshengy

Alternatively, you can increase the batch-size in the sgie1 configuration file.

Thank you very much! However, the maximum batch size for sgie1 can only be set to 8 due to memory constraints when exporting the engine file using trtexec.

[junshengy]

Refer to this topic for large models, This might help.

Zram swap on Jetson Orin - efficient enough?

@paaabl0 in the case of these large models, I prefer to mount the swap on NVME and disable ZRAM, as shown in the setup docs: As @linuxdev mentioned, this can cause premature wear on the storage media, particularly noticeable with SD cards (use high-endurance SD cards if not NVME)