* URGENT * How to convert Deepstream tensor to Numpy?

saransh661 · June 19, 2020, 1:00am

• Hardware Platform : GPU
• DeepStream 5.0
• TensorRT 7
• NVIDIA GPU Driver Version :440.59

So I’m migrating my tensorrt based custom model to PyDS, and I need to post-process on the whole output tensor, which is big so get_detections(layer.buffer, index) wont be efficient (Also couldn’t find get_detections() in the documentation)

I got my tensor output using
layer = pyds.get_nvds_LayerInfo(*params)

then inspecting layer.buffer, it was a pycapsule object
so I decoded it to byte-buffer using

byte_buffer = ctypes.pythonapi.PyCapsule_GetPointer(layer.buffer, None)

Now when I use
nd_arr = np.frombuffer(byte_buffer)

numpy throws the following error
ValueError: buffer size must be a multiple of element size

I’m suspecting either of these :

It needs to be transferred to host memory first?
Wrong datatype when converting buffer to numpy
Don’t remember the third

Then there’s a binding of CudaToNumpy in jetson-utils that I can try to port but It’s kinda urgent.
Is this right approach? Please guide how to take this forward.

AastaLLL · June 19, 2020, 4:58am

Hi,

Are you using this sample?

Based on the page, the numpy binding should already be done.
Please find the following document and deepstream_ssd_parser.py for more information:

github.com

NVIDIA-AI-IOT/deepstream_python_apps/blob/master/apps/deepstream-ssd-parser/custom_parser_guide.md

This guide shows how to use custom deep learning models and parse their inference output in a Python application. Custom model support is provided by the Triton Inference Server plugin included in the DeepStream SDK. The raw inference output can be parsed in a Python application via access to the NvDsInferTensorMeta structure. Sample code is provided in the deepstream-ssd-parser Python application.

**1 - Inference via Triton**

To perform inference using Triton and access the raw output, the general steps are:

1. Include Triton plugin in the pipeline. The deepstream-ssd-parser app shows how to construct a pipeline with Triton as the primary detector.

2. Configure the Triton plugin to attach raw output tensors to the metadata, and allow for custom post-processing. See details below.

3. Add a probe function to intercept output tensors for parsing. See the deepstream-ssd-parser app for example.

<b>Configuring Triton for custom parsing</b>:

1. Select custom post-processing:

   In ds_test_ssd_no_postprocess.txt, edit the `infer_config -> postprocess` section 
   to specify "other" as the parsing function and point to the custom labels file used for parsing:

   ```

This file has been truncated. show original

Thanks.

saransh661 · June 19, 2020, 5:09am

Thanks for the fast response.

Yes, I’m mainly using ssd_parser.py for reference only. Sorry If its naive but I was unable to find any tensor to numpy convertor, can you please specifically point it out.

Afaik, Numpy bindings are only available for converting buf_surface(input matrix) into opencv frame, which is very high abstraction, expects/supports a certain format of dimension

Custom parser guide gives a good idea of the workflow but not the exact function to get it.

saransh661 · June 19, 2020, 6:33am

To be more precise, the output of my network can be thought like a 2d mask for each image(but in float).
So the conventional functions that we use in these examples retrieve detection/classes by iterating one element a time would be really inefficient

saransh661 · June 19, 2020, 7:35am

I inspected further, putting dtype=uint8, when using np.frombuffer() outputs values, albeit wrong dimensions and values.
The model is FP32 for testing, the numpy output dimensions are 285 when using uint8/int8

The actual output dimensions are 4800 and I verified that this is the size received right upto layer.buffer using get_detections()

So its prolly an issue of using a right datatype? If you could suggest which datatype to use for this conversion

AastaLLL · June 22, 2020, 6:34am

Hi,

Would you mind to share a source for the issue?
So we can check and give a further suggestion directly.

Thanks.

saransh661 · June 22, 2020, 7:51am

@AastaLLL I can, sure, but instead, I’ll ask with reference to ssd example itself, so it’d be faster for you too.
So in ssd python example, if you open ssd_parser.py file, if you can navigate to
res.detectionConfidence = pyds.get_detections(score_layer.buffer, index)
inside make_nodi function.

So in this line of code, score_layer.buffer is basically flattened 1-d version of the output tensor “score_layer” and the function get_detections() returns a single element from this tensor corresponding to the index .

So basically, if you can show how to convert whole score_layer.buffer into a numpy array at once, without looping through each index one by one, the solution will solve my issue too.

I need the output array converted to numpy all at once because my output is too big, (80x60)

zhliunycm2 · June 24, 2020, 9:51pm

Hi saransh661, you can try getting the layer datatype from NvDsInferLayerInfo. The dataType field is an enum with these values:
FLOAT FP32 format.
HALF FP16 format.
INT8 INT8 format.
INT32 INT32 format.

We’ll look into returning the output tensors as a numpy array but I understand that your need is urgent. Hope with the right dtype and shape, you can get your numpy array with existing bindings.

freget · September 11, 2020, 8:32pm

Hi together.

I just found a working solution:

ptr = ctypes.cast(pyds.get_ptr(layer.buffer), ctypes.POINTER(ctypes.c_float))
v = np.ctypeslib.as_array(ptr, shape=(4096,))

Of course, the data type of the pointer and the shape have to be modified.

AastaLLL · September 16, 2020, 5:35am

Thanks for the sharing.

anasmk · November 16, 2020, 9:32am

Thank you a lot that worked for me.

shubham.shah09 · February 13, 2021, 8:48am

I cannot print the value of my tensor by printing this pointer out_buf_ptrs_host[i]; how can i print this tensor in c++

shubham.shah09 · February 16, 2021, 4:54am

How do we get this tensor in c++

Rob_Royce · July 1, 2021, 2:02am

Is it possible to get this to work with FP16? As far as I can tell, ctypes doesn’t have such an option

kayccc · July 6, 2021, 3:42am