How to Test the nvdspreprocess Element in Standalone Mode to Verify custom_sequence_preprocess Lib Output?

Accelerated Computing Intelligent Video Analytics DeepStream SDK
, , , ,
1

• Hardware Platform Quadro RTX 5000
• DeepStream Version 7.0
• TensorRT Version 8.6.0.1
**• NVIDIA GPU Driver Version ** 550.120
• custom_sequence_preprocess

Hello DeepStream Community,

I am currently working with NVIDIA DeepStream SDK and have integrated the nvdspreprocess element into my pipeline for pre-processing tasks. I would like to verify the output of nvdspreprocess in a standalone setup which is using libnvds_custom_sequence_preprocess custom lib, ideally by inspecting the preprocessed data before it moves to other pipeline components.

Here’s what I’ve tried so far:

  1. Standalone Pipeline: Created a basic standalone pipeline with only appsrcbin → nvstreammux - > nvdspreprocess → fakesink to focus solely on the preprocessor output.
  2. Testing Parameters: Configured nvdspreprocess with parameters such as:
  • Channel Scaling factors
  • Resize
  • Normalization
  1. Output Analysis: Attempted to attach probes to inspect buffers directly after nvdspreprocess to check if preprocessing for these parameters were correctly applied.

Challenges:

It’s unclear how to validate the processed frame data. I am considering dumping buffer data post-preprocess and analyzing it. Currently, I am getting pipeline native source width and height in inspected probe results instead of set Resize parameters.

Seeking Advice:

  • Has anyone successfully validated the output of nvdspreprocess in standalone mode?
  • Are there any recommended methods for inspecting the preprocessed data, especially for verifying custom given preprocessing parameters?

Any insights on optimal buffer dumping or visualization techniques for debugging would be greatly appreciated.

3

Please refer the following code snippet in /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvdspreprocess/nvdspreprocess_lib/nvdspreprocess_impl.cpp

#ifdef DEBUG_LIB
    static int batch_num2 = 0;
    std::ofstream outfile2("impl_out_batch_" + std::to_string(batch_num2) + ".bin");
    outfile2.write((char*) outPtr, 4*m_NetworkSize.channels*m_NetworkSize.width*m_NetworkSize.height);
    outfile2.close();
    batch_num2 ++;
#endif

You also can refer to /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-preprocess-test/deepstream_preprocess_test.cpp

user_meta = (NvDsUserMeta *)(l_user_meta->data);
      if (user_meta->base_meta.meta_type == NVDS_PREPROCESS_BATCH_META) 
      {
        GstNvDsPreProcessBatchMeta *preprocess_batchmeta =
            (GstNvDsPreProcessBatchMeta *) (user_meta->user_meta_data);
        if (preprocess_batchmeta->tensor_meta->raw_tensor_buffer) {
          g_print("received preprocess meta\n");
        }
      }

All tensors are saved to GstNvDsPreProcessBatchMeta

/**
 * tensor meta containing prepared tensor and related info
 * inside preprocess user meta which is attached at batch level
 */
typedef struct
{
  /** raw tensor buffer preprocessed for infer */
  void *raw_tensor_buffer;

  /** size of raw tensor buffer */
  guint64 buffer_size;

The raw_tensor_buffer is GPU buffer， You may need to use cudamemcpy to copy it to the CPU to dump it

4

Hi @junshengy,

Thank you for sharing the details and the pointed code snippets!

I have a few questions and would appreciate some clarification to proceed test further:

  1. Availability of Enums in PyDS
    The code snippet mentions the NVDS_PREPROCESS_BATCH_META meta type. Could you confirm if this enum is available in the PyDS bindings? If yes, could you point me to any relevant documentation or examples? I have tested the same with PyDS but seems like there is no as such given enum.

(Got: AttributeError: type object 'pyds.NvDsMetaType' has no attribute 'NVDS_PREPROCESS_BATCH_META') for below checks,

if user_meta and user_meta.base_meta.meta_type == pyds.NvDsMetaType.NVDS_PREPROCESS_BATCH_META:
            # if user_meta and user_meta.base_meta.meta_type == 27:
                preprocess_batch_meta = pyds.NvDsPreProcessBatchMeta.cast(user_meta.user_meta_data)
                print("Successfully retrieved GstNvDsPreProcessBatchMeta")
  1. Testing with PyDS
    For testing purposes, how can I access and validate the GstNvDsPreProcessBatchMeta meta using PyDS? Specifically:
  • How do I retrieve the raw_tensor_buffer from the batch-level user meta?
  • Are there any utility functions in PyDS for handling GPU buffers, or should I use CUDA APIs like cudaMemcpy within Python to copy the buffer to CPU for inspection?

Looking forward to your insights to proceed with testing efficiently.

Thank you!

5

Accessing GstNvDsPreProcessBatchMeta from Python is not currently supported. You can add the binding yourself.

If you just want to dump the tensor output by nvdspreprocess, it is recommended to modify the native code directly

raw_tensor_buffer is only used as nvinfer input, and there is usually no need to access it in pyds.

Refer to /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/apps/deepstream-imagedata-multistream-cupy/deepstream_imagedata-multistream_cupy.py

6

Thank you @junshengy for the clarifications and suggestions. We are able to add custom python binding to access GstNvDsPreProcessBatchMeta and able to test nvdspreprocess tensor output for process-on-frame mode:1 using above mentioned Standalone Pipeline: appsrcbin → nvstreammux→ nvdspreprocess → fakesink

Now, we want to validate nvdspreprocess tensor output for mode:0 (process on objects) to test clip objects for ourcustom config properties.

I have a few follow-up questions and requests for guidance:

  1. Test Clip Objects
  • Could you provide more details about the test clip objects used for verifying nvdspreprocess tensor output for process-on-frame mode:0 ?
  • How can we simulate clip objects to test nvdspreprocess tensor output for mode:0 (process on objects) using mentioned Standalone Pipeline?
  • Are there any recommendations that can be used for validating the tensor outputs for clip objects to test custom config properties?
  1. Validating Tensor Outputs
    To validate the tensor outputs for clip objects:
  • Is there any reference implementation or expected output data I can compare against for nvdspreprocess tensor output?
  • If modifications to the native code are required, could you suggest specific areas to focus on for dumping the tensor data?

Your input will greatly assist in setting up the test pipeline and ensuring comprehensive validation of nvdspreprocess.

Thank you for your support!

7

There is no a separate property for object preprocess mode, but you only need to modify the configuration file.
Please refer to /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-preprocess-test/config_preprocess_sgie.txt

This is a complete command line for nvdspreprocess object processing.

gst-launch-1.0 uridecodebin uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264 name=dec0 dec0. ! queue ! mux.sink_0 uridecodebin uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264 name=dec1 dec1. ! queue ! mux.sink_1 uridecodebin uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264 name=dec2 dec2. ! queue ! mux.sink_2 uridecodebin uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264 name=dec3 dec3. ! queue ! mux.sink_3 nvstreammux name=mux batch-size=4 width=1280 height=720 ! nvinfer config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-preprocess-test/config_infer.txt ! nvdspreprocess config-file=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-preprocess-test/config_preprocess_sgie.txt ! nvinfer input-tensor-meta=1 name=sgie1 config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test2/dstest2_sgie2_config.txt ! nvvideoconvert ! nvmultistreamtiler ! nvdsosd ! nveglglessink

What are your expected data? Compared with OpenCV? These data are processed by GPU/VIC, which is a little different from OpenCV, but usually has little effect on the results. Of course, this is why nvdspreprocess exists, to facilitate user customization.

Dumping frame/ROI/object tensor outputs uses the same native code. You can refer to the previous answer.

8

Hi @junshengy . Yes, there is /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-preprocess-test/config_preprocess_sgie.txt

[property]
enable=1
target-unique-ids=1

# 0=process on objects 1=process on frames
process-on-frame=0

I am referring the requested clip object tests with 0=process on objects for process-on-frame=0 using mentioned Standalone Pipeline.

9

I have provided a command line.

10

Hello @junshengy, we are not looking for using nvinfer plaugging, we are working on component test for nvpreprocessor plugin and want to supply mock input to nvdspreprocess plugin.

we were able to test in full frame mode using appsrc → streammux but we now want to test process on objects and we are unable to simulate input to nvpreprocessor plugin, if you can guide us how can we do that would helpful.

11

I don’t quite understand what you mean. When nvdspreprocess processes an object, it will traverse the objects in obj_meta_list, but obj_meta_list is generated by nvinfer after detecting the frame. It cannot be used alone.

nvdspreprocess is used to generate GstNvDsPreProcessBatchMeta，I think using nvdspreprocess alone doesn’t make sense, what is your goal, maybe I have a better suggestion.

Please create a new topic for your question.

12

Adding obj_meta_list is not issue, that I can ingest using frame_meta and object_meta ingestion api into batch_meta structure.

my question, is how can I attach video external buffer to DeepStream pipeline when my plugin expect clip object. we can ingest external frame using appsrc as an example. Full frame we can push to get_buffer but how can we do preprocess input.

13

nvdspreprocess processes the batch generated by nvstreammux.
You need to feed the data fromappsrc to nvstreammux.

Please open a new topic