Deepstream with own model

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Besides the supplied nvidia models and some exact examples of for Yolo, I can not seem to find how to take to take a generic ONNX model and get it running under DS6.0.

I noticed that for example Yolo has a custom library, that i think is processing the output of the Yolo model and converts it into deepstream specific format ? (Can anyone confirm this? )

Say i find a way to convert the latest detector from RoboFlow (RF-DETR) to ONNX, what would be the next steps to getting it to run under DS6.0 ?
custom-lib-path=libnvdsinfer_custom_impl_rfdetr.so (in the nvinfer config)

How would I figure out what this library needs to do ? Who made the one for the Yolo one, for example, and how did they find out what this is supposed to do ? Is there documentation for this ?

then I think there is a parse bbox libray… and the same questions as above.
parse-bbox-func-name=NvDsInferParseCustomRFDETR

Thanks,
Tom

4

That’s right

Users need to implement this shared library by themselves. This shared library is used to parse the output of the model. If this is a detection model, it is responsible for parsing the tensor output by the model into bbox.

If this is a detection model, nvinfer will parse the parse-bbox-func-name configuration item in the configuration file, use dlopen/dlsym to call the user’s custom parsing library, and parse the output tensor into NvDsObjectMeta.

The function prototype is

bool (*)(std::vector<NvDsInferLayerInfo> const &outputLayersInfo,
                                   NvDsInferNetworkInfo  const &networkInfo,
                                   NvDsInferParseDetectionParams const &detectionParams,
                                   std::vector<NvDsInferObjectDetectionInfo> &objectList)

This is sample code

If your device support latest version of deepstream, please update to latest verison. Generally speak, we don’t support legacy version.

5

Thank you for your answers !!

Sorry, we still are running on a lot of TX2’s

6

Were you able to figure this out? I am also interested in running RF-DETR in Nvidia Deepstream

7

sorry, no. I’ve gone back to basics first and have created a very simple onnx and am trying to write a custom lib for that just for practice. but i have to do this inbetween jobs and haven’t gotten anything to work yet.. this stuff is not easy