Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU): Ubuntu 24.04 RTX 3060
• DeepStream Version: 8.0
• TensorRT Version: 10.3
• NVIDIA GPU Driver Version 580.95.05
• Issue Type: questions
Hi NVIDIA team,
I’m currently using DeepStream (with nvinferserver) connected to Triton Inference Server (gRPC) to run an ONNX model as a SGIE model.
dets: float32[batch_size, N, 5]
labels: int64[batch_size, N]
polygons: float32[batch_size, N, 8]
colors: int64[batch_size, N]
types: int64[batch_size, N]
When I test the model directly using the Triton Python client, all outputs are correct.
However, when the same model is loaded via nvinferserver, the tensor metadata (tensor_meta in probe function) is not correct.
Here’s a my config:
infer_config {
backend {
triton {
model_name: "plate_detector"
version: 1
grpc {
url: "triton:8001"
enable_cuda_buffer_sharing: true
}
}
}
preprocess {
network_format: IMAGE_FORMAT_RGB
normalize {
scale_factor: 0.017507
channel_offsets: [123.675, 116.28, 103.53]
}
}
}
output_control {
output_tensor_meta: true
}
My triton config:
name: "license_plate_detector",
platform: "tensorrt_plan"
default_model_filename: "model.plan"
max_batch_size: 16
version_policy: { specific: { versions: [1]}}
input: [
{
name: "input",
data_type: TYPE_FP32,
dims: [3, 512, 512],
}
]
output: [
{
name: "dets",
data_type: TYPE_FP32,
dims: [-1, 5]
},
{
name: "polygons",
data_type: TYPE_FP32,
dims: [-1, 8]
},
{
name: "labels",
data_type: TYPE_INT64,
dims: [-1]
},
{
name: "colors",
data_type: TYPE_INT64,
dims: [-1]
},
{
name: "types",
data_type: TYPE_INT64,
dims: [-1]
}
]
optimization: {
priority: PRIORITY_DEFAULT,
input_pinned_memory: {
enable: true
},
output_pinned_memory: {
enable: true
},
gather_kernel_buffer_threshold: 0,
eager_batching: false
}
instance_group: [
{
name: "plate_detector",
kind: KIND_GPU,
count: 2,
host_policy: ""
}
]
model_warmup: []
dynamic_batching {
max_queue_delay_microseconds: 100
}
Here the simple code that I use to parse tensor from user_meta data:
def get_layer(
layers_info: list[pyds.NvDsInferLayerInfo],
name: str,
) -> Optional[pyds.NvDsInferLayerInfo]:
"""Find a layer by name"""
for layer in layers_info:
if layer.layerName and layer.layerName == name:
return layer
return None
def ds_parse_plate_detector(layers_info: list[Any]):
"""Parse Triton outputs from license_plate_detector"""
def np_from_layer(layer: pyds.NvDsInferLayerInfo) -> np.ndarray:
if layer.dataType == pyds.NvDsInferDataType.FLOAT:
dtype = ctypes.c_float
elif layer.dataType == pyds.NvDsInferDataType.INT32:
dtype = ctypes.c_int32
# Rebuilt support for INT64 data type from pyds
elif layer.dataType == pyds.NvDsInferDataType.INT64:
dtype = ctypes.c_int64
else:
raise ValueError(f"Unsupported data type: {layer.dataType}")
shape = [layer.inferDims.d[i] for i in range(layer.inferDims.numDims)]
# Convert to numpy array
ptr = ctypes.cast(pyds.get_ptr(layer.buffer), ctypes.POINTER(dtype))
layer_array = np.ctypeslib.as_array(ptr, shape=shape)
return layer_array
det_layer = get_layer(layers_info, "dets")
label_layer = get_layer(layers_info, "labels")
poly_layer = get_layer(layers_info, "polygons")
color_layer = get_layer(layers_info, "colors")
type_layer = get_layer(layers_info, "types")
if (
det_layer is None
or label_layer is None
or poly_layer is None
or color_layer is None
or type_layer is None
):
raise ValueError("Missing required layers in plate detector output")
dets = np_from_layer(det_layer)
labels = np_from_layer(label_layer)
polygons = np_from_layer(poly_layer)
colors = np_from_layer(color_layer)
multilines = np_from_layer(type_layer)
# post-process to get patches
# score thresholding
bboxes = dets[:, :-1]
scores = dets[:, -1]
plate_info = dict(
bboxes=bboxes.tolist(),
scores=scores.tolist(),
labels=labels.tolist(),
polygons=polygons.tolist(),
colors=colors.tolist(),
multilines=multilines.tolist(),
)
return plate_info
I can successfully parse the tensor, here the output shapes:
colors: [32], dtype=NvDsInferDataType.INT64
dets: [32, 5], dtype=NvDsInferDataType.FLOAT
labels: [32], dtype=NvDsInferDataType.INT64
polygons: [32, 8], dtype=NvDsInferDataType.FLOAT
types: [32], dtype=NvDsInferDataType.INT64
but the values are not correct.
Thanks in advance for any help or insights!