I wish to extract the exact input (and output) of the model inside the nvinfer module.
As far as I know, nvinfer executes some preprocessing:
Normalization + Mean Substraction
Resizing
Is it possible to extract the resulting tensor in Python ?
Also, I am currently trying to setup a probe for nvinfer, but I am struggling to understand the underlying structure of the data, is there a way or documentation which shows how to access this data?
def nvinfer_probe(pad, info, u_data):
global cnt
gst_buffer = info.get_buffer()
if not gst_buffer:
print("Unable to get GstBuffer ")
return Gst.PadProbeReturn.OK
# Retrieve batch metadata from the GstBuffer
batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
if not batch_meta:
return Gst.PadProbeReturn.OK
l_frame = batch_meta.frame_meta_list
while l_frame is not None:
print("L_FRAME")
print("Count: ", cnt)
cnt+=1
try:
frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
except StopIteration:
break
l_user = frame_meta.frame_user_meta_list
while l_user is not None:
print("L_USER")
try:
user_meta = pyds.NvDsUserMeta.cast(l_user.data)
except StopIteration:
break
if user_meta.base_meta.meta_type != pyds.NvDsMetaType.NVDSINFER_TENSOR_OUTPUT_META:
l_user = l_user.next
continue
tensor_meta = pyds.NvDsInferTensorMeta.cast(user_meta.user_meta_data)
for i in range(tensor_meta.num_output_layers):
layer = pyds.get_nvds_LayerInfo(tensor_meta, i)
if layer.layerName == "conv2d_2":
print(f"Processing Tensor output for layer: {layer.layerName}")
#@TODO ----
l_user = l_user.next
l_frame = l_frame.next
return Gst.PadProbeReturn.OK
I’m adding the probe as follows:
srcpad_nvinfer = nvinfer.get_static_pad("src")
if not srcpad_nvinfer:
sys.stderr.write("Unable to get src pad of nvinfer\n")
else:
srcpad_nvinfer.add_probe(Gst.PadProbeType.BUFFER, nvinfer_probe, 0)
In this code, I am never able to reach l_user, as it’s allways “NULL”, I am obviously doing something wrong, could you guide me as to where to find this information or how to correct it?
• Hardware Platform (Jetson / GPU) Jetson Xavier AGX • DeepStream Version 6.3 • JetPack Version (valid for Jetson only) 5.1.2 • Issue Type( questions, new requirements, bugs) Question
Actually, my issue arises when I set input-tensor-from-meta=1
I think my dataflow gets broken at that point.
I understand, that this makes that the preprocessing (inside nvinfer) has to be made manually and explicitly (using NvDsPreprocess).
Before I start making changes to it and customizing it; could you tell me if the default behaviour of NvDsPreprocessis exactly the same as what happens inside nvinfer when input-tensor-from-meta=0?
NvInfer has 3 modes infer: input-tensor-from-meta, input fullframe, and input from object of pgie (detection).
When you set input-tensor-from-meta=1, you must have nvdspreprocess plugin before NvInfer to create data structure suite for nvinfer with mode input-tensor-from-meta. And if you don’t have nvdspreprocess. NvInfer just continue and don’t infer anything
it does some scaling such as to fit the models requirements (which algorithm is being used?)
Where can I find details on this?
Transforms the color-format, which can be set in the config file
For the output:
Transforms the color-format back, I think ?
I printed out the pipeline graph and it seems the color format gets transformed back
Any other post-processing steps happening inside nvinfer?
My goal:
Extract the exact tensors that come in and out of the model
What’s the best way of doing this ?
Also, using output-tensor-meta=1 - I am unable to find the location of the tensor data from nvinfer buffer, I am able to do it in nvdsosd after nvvideoconvertthough, any help?
If you read this code, you will see that input of the model is datatype called: NvBufSurface, maybe you can add code to NvInfer to get this input data
Model will infer based on model-color-format and will transform it back to the pipeline (You can search NvBufSurfTransform in code of nvinfer)
For the output: You can see the output of the model by using a probe function and set mode in config-file for NvInfer is output-tensor-meta=1
When you set output-tensor-meta=1, as my knowledge, it will create usermeta with meta_type is: NVDSINFER_TENSOR_OUTPUT_META and attach output meta to user_meta->user_meta_data->out_buf_ptrs_host[i] (i is number of output). So you can get output-tensor-meta from user meta.
You must to set probe function behind NvInfer to get data.
def nvinfer_probe(pad, info, u_data):
gst_buffer = info.get_buffer()
if not gst_buffer:
print("Unable to get GstBuffer ")
return Gst.PadProbeReturn.OK
# Retrieve batch metadata from the GstBuffer
batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
if not batch_meta:
return Gst.PadProbeReturn.OK
l_frame = batch_meta.frame_meta_list
while l_frame is not None:
try:
frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
print("### Frame Number --- ", frame_meta.frame_num)
except StopIteration:
break
l_user = frame_meta.frame_user_meta_list
while l_user is not None:
try:
user_meta = pyds.NvDsUserMeta.cast(l_user.data)
except StopIteration:
break
if user_meta.base_meta.meta_type != pyds.NvDsMetaType.NVDSINFER_TENSOR_OUTPUT_META:
l_user = l_user.next
continue
tensor_meta = pyds.NvDsInferTensorMeta.cast(user_meta.user_meta_data)
for i in range(tensor_meta.num_output_layers):
layer = pyds.get_nvds_LayerInfo(tensor_meta, i)
if layer.layerName == "conv2d_2":
# --------
#CONTENT
l_user = l_user.next
try:
l_frame = l_frame.next
except StopIteration:
break
return Gst.PadProbeReturn.OK
Now, to extract the input, I can either customize nvinfer and/or NvDsPreprocess as far as I understood; otherwise I was thinking of having a tee before nvinfer and replicate the pre-processing steps in a parallel branch, as a quick dirty solution.
But maybe customizing nvinfer might not be too complex / long?
I’m building an anomaly detection solution, for which I’m comparing inputs to the network, to it’s outputs (reconstruction).
Because I’m unsure if I can access the exact input to the model inside nvinfer as of now, if not I need to replicate it.
I didn’t familiarize myself with DeepStreams C/C++ code, neither did I work with the language for a while; as such, before investing so much time into getting into it - for now - I would like to see how far I can go with the python bindings.
I already started looking into the C/C++ source code, but I can see that I need longer to achieve my goals, so that will take time, but I would like to test my solutions parallely on python if possible!
Also we have put the Gst-Nvinfer source code diagram on the FAQ. If you are interested, you can read our open source code directly by referring to this diagram.
Thank you, yes, looking at the diagram, I’m interested into reading either the gst_nvinfer_queue_loop or the CudaDeviceBuffer. If you got any leads how to do this effectively that would be awesome !
Meanwhile I’ll start looking inthe nvinfers’ source code and see if I can work with it.
There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks
OK. You can also refer to our FAQ when reading the source code to dump the Inference input and output.
