Please provide complete information as applicable to your setup.
• Hardware Platform: GPU
• DeepStream Version: 6.2-triton (container of docker)
• Issue Type: bugs
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
I use yolov8 of ultralytics for pgie for detection. The current model I use is the pretrain model on the COCO set which includes 80 classes. I am facing a problem that if I use nvinfer it gives correct bounding boxes and classes results but using nvinferserver with triton it gives wrong results.
I see that both are using the same preprocess but the results are different. Can you tell me why and how to solve it?
The image below is the result I printed out in nvinfer’s parse bbox func:
And below is the print result from parse bbox func of nvinferserver:
I find it problematic because the probability is only in the range 0-1 but it is larger and the classid is only 0. Both nvinfer and nvinferserver use the same parse bbox func.
Here is the nvinfer config for pgie:
And here is the config of nvinferserver for pgie:
No. We can’t tell you why according to the information you provided.
Let’s guess, you are using marcoslucianops/DeepStream-Yolo: NVIDIA DeepStream SDK 6.3 / 6.2 / 6.1.1 / 6.1 / 6.0.1 / 6.0 / 5.1 implementation for YOLO models (github.com), right?
You Triton configuration is not complete, please tell us which model you are using? “tensorrt_plan”, “tensorflow_graphdef”, “tensorflow_savedmodel”, “onnxruntime_onnx” or “pytorch_libtorch”?
Can you also provide the config.pbtxt file of the model?
Can you provide the yolov8.onnx too?
1 Like
of course, this is my model.onnx
model.onnx (12.1 MB)
Sorry, did you receive the model i sent? I’m a new user so I don’t know if my response is correct for you to receive.
I’ve got the model. The output layers are dynamic dimensions. The output format may not meet the postprocess algorithm.
Can you change to tensorrt_plan backend? The engine can be generated by trtexec
E.G. the batch size 1 engine:
trtexec --onnx=model.onnx --workspace=10000 --saveEngine=model.onnx_b1.engine --minShapes=input:1x3x640x640 --maxShapes=input:1x3x640x640 --optShapes=input:1x3x640x640
The other configurations can refer to DeepStream triton inferencing samples.
1 Like
I will try it then get back to you.
I tried converting model to trt but the result is still the same. Here is the config serving model on my triton:
config.pbtxt (1.3 KB)
Did you convert it to trt and see that it gives the correct result?
I can reproduce the problem. And I found the native Triton inferencing can output correct bboxes. We will investigate the issue internally. Will the native Triton configuration meet your reqirement?
config.pbtxt (555 Bytes)
config_infer_primary_yoloV8_triton.yml (1.0 KB)
deepstream_app_config.txt (1.2 KB)
1 Like
I tried and the result is correct but why is it possible to use triton in deepstream container to serve model but the same config to create a separate service triton so calling localhost:8001 to use grpc can’t work? If using model_repo, how to use grpc ?
It is a bug of nvinferserver. We will investigate the issue internally.
1 Like
Thank you for your support. If you have information on how to fix this bug, please let me know.
We have investigated the nvinferserver and the project marcoslucianops/DeepStream-Yolo: NVIDIA DeepStream SDK 6.3 / 6.2 / 6.1.1 / 6.1 / 6.0.1 / 6.0 / 5.1 implementation for YOLO models (github.com) you refer to, we find that the bbox parsing function get the output layers in order. But in Triton grpc mode, the network transferring can not guarantee the data reach in order. The layer data may reach randomly. So you need to get the layer by the name.
The following patch to DeepStream-Yolo/nvdsinfer_custom_impl_Yolo/nvdsparsebbox_Yolo.cpp at master · marcoslucianops/DeepStream-Yolo (github.com) can resolve the output parsing failure.
diff --git a/nvdsinfer_custom_impl_Yolo/nvdsparsebbox_Yolo.cpp b/nvdsinfer_custom_impl_Yolo/nvdsparsebbox_Yolo.cpp
index e2ff987..85dbe89 100644
--- a/nvdsinfer_custom_impl_Yolo/nvdsparsebbox_Yolo.cpp
+++ b/nvdsinfer_custom_impl_Yolo/nvdsparsebbox_Yolo.cpp
@@ -137,16 +137,27 @@ NvDsInferParseCustomYolo(std::vector<NvDsInferLayerInfo> const& outputLayersInfo
return false;
}
+ auto layerFinder = [&outputLayersInfo](const std::string &name)
+ -> const NvDsInferLayerInfo *{
+ for (auto &layer : outputLayersInfo) {
+ if (layer.dataType == FLOAT &&
+ (layer.layerName && name == layer.layerName)) {
+ return &layer;
+ }
+ }
+ return nullptr;
+ };
+
std::vector<NvDsInferParseObjectInfo> objects;
- const NvDsInferLayerInfo& boxes = outputLayersInfo[0];
- const NvDsInferLayerInfo& scores = outputLayersInfo[1];
- const NvDsInferLayerInfo& classes = outputLayersInfo[2];
+ const NvDsInferLayerInfo * boxes = layerFinder("boxes");
+ const NvDsInferLayerInfo * scores = layerFinder("scores");
+ const NvDsInferLayerInfo * classes = layerFinder("classes");
- const uint outputSize = boxes.inferDims.d[0];
+ const uint outputSize = boxes->inferDims.d[0];
- std::vector<NvDsInferParseObjectInfo> outObjs = decodeTensorYolo((const float*) (boxes.buffer),
- (const float*) (scores.buffer), (const float*) (classes.buffer), outputSize, networkInfo.width, networkInfo.height,
+ std::vector<NvDsInferParseObjectInfo> outObjs = decodeTensorYolo((const float*) (boxes->buffer),
+ (const float*) (scores->buffer), (const float*) (classes->buffer), outputSize, networkInfo.width, networkInfo.height,
detectionParams.perClassPreclusterThreshold);
objects.insert(objects.end(), outObjs.begin(), outObjs.end());
The marcoslucianops/DeepStream-Yolo: NVIDIA DeepStream SDK 6.3 / 6.2 / 6.1.1 / 6.1 / 6.0.1 / 6.0 / 5.1 implementation for YOLO models (github.com) project is a third party open source project. We’d prefer you to refer to the DeepStream sample apps to avoid such issue.
Thank you very much it worked. Let me ask, my model has 80 classes but I only want to draw the osd classes [0,1,2], in nvinfer there is a filter-out-class-ids field to do this, and is there a similar field in nvinferserver? ? Or do we have to filter it directly in the parse bbox function?
No. There is no such parameter in nvinferserver. A replacement may be you can set per class parameter “pre_threshold” to a value larger than 1 for the classes you don’t need. Gst-nvinferserver — DeepStream 6.3 Release documentation
Or you can implement the filter in the customized bbox parser function.
