• Hardware Platform GPU
• DeepStream Version 7.1
• TensorRT Version 10.3
• NVIDIA GPU Driver Version 560.35.03
• Issue Type question
I use the Docker container nvcr.io/nvidia/deepstream:7.1-gc-triton-devel and can successfully run the C/C++ and Python samples.
My goal is to get nvinfer raw tensor output of a simple custom model. It is not a classifier, detector or anything specific, but just a model which takes a 128×128 grayscale image as input and outputs 10 coefficients I want to simply get directly without any post-process.
The model is a ONNX file trained with TensorFlow and converted by tf2onnx.convert (Python). It has the following dimensions (inspected with netron):
- input:
tensor: float32[unk__8,1,128,128]- (originally it was
[unk__8,128,128]but it seems I am forced to add a dimension for DeepStream),
- (originally it was
- output:
tensor: float32[unk__9,10], - note: I have not set those
unk__xmyself.
I converted the ONNX model with trtexec --onnx=model.onnx --saveEngine=model.engine.
I can successfully make it inference correctly with TensorRT directly.
The DeepStream config is in this following minimal config.txt file:
[property]
gpu-id=0
model-engine-file=model.engine
network-type=100 # 0: Detector, 1: Classifier, 2: Segmentation, 100: Other
model-color-format=2 # 0:RGB, 1:BGR, 2:GRAY
output-tensor-meta=1
gie-unique-id=1
The DeepStream runtime is this following minimal app.py file:
import gi
gi.require_version('Gst', '1.0')
from gi.repository import Gst, GLib
import pyds
Gst.init(None)
pipeline = Gst.parse_launch(
"pylonsrc pfs-location=/root/workspace/camera-settings/config.pfs ! "
"video/x-raw, width=128, height=128, format=GRAY8 ! "
"nvvideoconvert ! "
"video/x-raw(memory:NVMM) ! "
"mux.sink_0 nvstreammux name=mux batch-size=1 width=128 height=128 ! "
"nvinfer name=nvinfer_element config-file-path=config.txt ! "
"fakesink"
)
loop = GLib.MainLoop()
nvinfer_sink_pad = pipeline.get_by_name('nvinfer_element').get_static_pad("sink")
def nvinfer_sink_pad_buffer_probe(pad, info, u_data):
gst_buffer = info.get_buffer()
batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
l_frame = batch_meta.frame_meta_list
while l_frame is not None:
frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
user_meta_list = frame_meta.frame_user_meta_list
if not user_meta_list:
print("user_meta_list is None")
l_frame = l_frame.next
return Gst.PadProbeReturn.OK
nvinfer_sink_pad.add_probe(Gst.PadProbeType.BUFFER, nvinfer_sink_pad_buffer_probe, 0)
pipeline.set_state(Gst.State.PLAYING)
try: loop.run()
except KeyboardInterrupt: pass
pipeline.set_state(Gst.State.NULL)
The pipeline runs (and can also successfully show the image going into the nvstreammux with an alternative pipeline with a tee). I have no idea if the inferences are really made. The problem appears when the called probe is trying to get the user_meta_list which appears to be empty:
# python3 app.py
0:00:03.021292477 189234 0x64e9de736f10 INFO nvinfer gstnvinfer.cpp:684:gst_nvinfer_logger:<nvinfer_element> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:2092> [UID = 1]: deserialized trt engine from :/root/workspace/project/model.engine
Implicit layer support has been deprecated
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:327 [Implicit Engine Info]: layers num: 0
0:00:03.021332513 189234 0x64e9de736f10 INFO nvinfer gstnvinfer.cpp:684:gst_nvinfer_logger:<nvinfer_element> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2195> [UID = 1]: Use deserialized engine model: /root/workspace/project/model.engine
0:00:03.030667933 189234 0x64e9de736f10 INFO nvinfer gstnvinfer_impl.cpp:343:notifyLoadModelStatus:<nvinfer_element> [UID 1]: Load new model:config_nvinfer.txt sucessfully
user_meta_list is None
user_meta_list is None
user_meta_list is None
[...and so on...]
Any idea what’s wrong ?