yolo custom model file and error from primary-nvinference-engine

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

hi
i use deepstream4.0 jetson xavier

I made yolo custom model with darknet. and put it in (config_infer_primary_yoloV3_tiny.txt)
It works well. so I put it in model-file of(dstest2_pgie_config.txt).

original deepstream-test2 model-file is resnet10 caffemodel
Can’t yolo model file replace resnet10 caffemodel?
Since I have a YOLO dataset, I’m trying to use YOLO rather than training resnet with tlt, and using the dtest2 sample cause I need the object-id.

This is my terminal result

[NvDCF] Initialized
0:02:25.115191955 14820 0x55b2836a30 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger: NvDsInferContext[UID 1]:initialize(): Trying to create engine from model files
Yolo type is not defined from config file name:
0:02:25.115876334 14820 0x55b2836a30 ERROR nvinfer gstnvinfer.cpp:511:gst_nvinfer_logger: NvDsInferContext[UID 1]:generateTRTModel(): Failed to create network using custom network creation function
0:02:25.115930672 14820 0x55b2836a30 ERROR nvinfer gstnvinfer.cpp:511:gst_nvinfer_logger: NvDsInferContext[UID 1]:initialize(): Failed to create engine from model files
0:02:25.116230588 14820 0x55b2836a30 WARN nvinfer gstnvinfer.cpp:692:gst_nvinfer_start: error: Failed to create NvDsInferContext instance
0:02:25.116254941 14820 0x55b2836a30 WARN nvinfer gstnvinfer.cpp:692:gst_nvinfer_start: error: Config file path: dstest2_pgie_config.txt, NvDsInfer Error: NVDSINFER_CUSTOM_LIB_FAILED
Running…
ERROR from element primary-nvinference-engine: Failed to create NvDsInferContext instance
Error details: gstnvinfer.cpp(692): gst_nvinfer_start (): /GstPipeline:dstest2-pipeline/GstNvInfer:primary-nvinference-engine:
Config file path: dstest2_pgie_config.txt, NvDsInfer Error: NVDSINFER_CUSTOM_LIB_FAILED

2

Hi,

You need a custom lib to parse the output of yolo model that is required by nvinfer plugin. You can see this lib in the object detector yolo sample. It is enabled in the yolo sample primary infer config files as follows -

parse-bbox-func-name=NvDsInferParseCustomYoloV3Tiny
custom-lib-path=nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so

Please enable these settings in your “dstest2_pgie_config.txt” and you should be able to use yolo for inference in ds-test2 app.

3

Thanks for your answer
But I still get an same error from gstnvinfer.cpp

Error: Could not parse custom library path
Failed to parse group property
** ERROR: <gst_nvinfer_parse_config_file:943>: failed
Unknown key ‘gie-mode’ for group [property]
Unknown key ‘gie-mode’ for group [property]
Unknown key ‘gie-mode’ for group [property]
Now playing: rtsp://admin:1234@222.116.156.173/video1
.
.
.
[NvDCF] Initialized
0:01:32.486725207 23594 0x5582238c30 WARN nvinfer gstnvinfer.cpp:658:gst_nvinfer_start: error: Configuration file parsing failed
0:01:32.486794776 23594 0x5582238c30 WARN nvinfer gstnvinfer.cpp:658:gst_nvinfer_start: error: Config file path: dstest2_pgie_config.txt
Running…
ERROR from element primary-nvinference-engine: Configuration file parsing failed
Error details: gstnvinfer.cpp(658): gst_nvinfer_start (): /GstPipeline:dstest2-pipeline/GstNvInfer:primary-nvinference-engine:
Config file path: dstest2_pgie_config.txt
Returned, stopping playback
Deleting pipeline

The first line of errors seems to cause problems applying a custom model in gstnvinfer.cpp Is there anything else I should fix in other cpp files besides modifying dtest2’s pgie.cfg file?

Here is my dtest2 pgie config file
[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
model-file=yolov3-tiny_180000.weights
#proto-file=…/…/…/…/samples/models/Primary_Detector/resnet10.prototxt
labelfile-path=…/…/…/…/samples/models/Primary_Detector/labels.txt
int8-calib-file=yolov3-calibration.table.trt5.1
batch-size=1
network-mode=0
process-mode=1
model-color-format=0
num-detected-classes=5
interval=0
gie-unique-id=1
output-blob-names=conv2d_bbox;conv2d_cov/Sigmoid
parse-bbox-func-name=NvDsInferParseCustomYoloV3Tiny
custom-lib-path=//…/…/…/objectDetector_Yolo/nvdsinfer_custom_impl_Yolo/
libnvdsinfer_custom_impl_Yolo.so

[class-attrs-all]
threshold=0.2
eps=0.2
group-threshold=1

4 
Error: Could not parse custom library path

As the error message indicates, your path to custom library is wrong. Please correct it and try again.

5

Thank you.
We fixed the error.
But I faced a new situation.
The photo below compares the modified pgie.cfg(yolo) and the dstest2_pgie_config.txt in original deepstream-test2(caffemodel).
Red is what we fixed, green is the original.
There is no error and it runs but custom model doesn’t detect objects at all.
The below is a comparison of the cfg file and the result of the terminal window.

And from which file does class_id in nvdsmeta.h struct NvCsObjectMeta come from?

model-file? or proto-file? or is there another variable?

nvidia@nvidia-jetson:~/deepstream-4.0/sources/apps/sample_apps/deepstream-test2_rtsp$ ./deepstream-test2-app rtsp://admin:1234@222.116.156.173/video1
With tracker
Now playing: rtsp://admin:1234@222.116.156.173/video1

Using winsys: x11
Creating LL OSD context new
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-4.0/lib/libnvds_nvdcf.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is ON
[NvDCF] Initialized
0:00:00.939992944 16865 0x5598a54300 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger: NvDsInferContext[UID 1]:initialize(): Trying to create engine from model files
0:00:01.156771108 16865 0x5598a54300 WARN nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger: NvDsInferContext[UID 1]:generateTRTModel(): INT8 calibration file not specified/accessible. INT8 calibration can be done through setDynamicRange API in ‘NvDsInferCreateNetwork’ implementation
Loading pre-trained weights…
Loading complete!
Total Number of weights read : 8685484
layer inp_size out_size weightPtr
(1) conv-bn-leaky 3 x 416 x 416 16 x 416 x 416 496
(2) maxpool 16 x 416 x 416 16 x 208 x 208 496
(3) conv-bn-leaky 16 x 208 x 208 32 x 208 x 208 5232
(4) maxpool 32 x 208 x 208 32 x 104 x 104 5232
(5) conv-bn-leaky 32 x 104 x 104 64 x 104 x 104 23920
(6) maxpool 64 x 104 x 104 64 x 52 x 52 23920
(7) conv-bn-leaky 64 x 52 x 52 128 x 52 x 52 98160
(8) maxpool 128 x 52 x 52 128 x 26 x 26 98160
(9) conv-bn-leaky 128 x 26 x 26 256 x 26 x 26 394096
(10) maxpool 256 x 26 x 26 256 x 13 x 13 394096
(11) conv-bn-leaky 256 x 13 x 13 512 x 13 x 13 1575792
(12) maxpool 512 x 13 x 13 512 x 13 x 13 1575792
(13) conv-bn-leaky 512 x 13 x 13 1024 x 13 x 13 6298480
(14) conv-bn-leaky 1024 x 13 x 13 256 x 13 x 13 6561648
(15) conv-bn-leaky 256 x 13 x 13 512 x 13 x 13 7743344
(16) conv-linear 512 x 13 x 13 30 x 13 x 13 7758734
(17) yolo 30 x 13 x 13 30 x 13 x 13 7758734
(18) route - 256 x 13 x 13 7758734
(19) conv-bn-leaky 256 x 13 x 13 128 x 13 x 13 7792014
(20) upsample 128 x 13 x 13 128 x 26 x 26 -
(21) route - 384 x 26 x 26 7792014
(22) conv-bn-leaky 384 x 26 x 26 256 x 26 x 26 8677774
(23) conv-linear 256 x 26 x 26 30 x 26 x 26 8685484
(24) yolo 30 x 26 x 26 30 x 26 x 26 8685484
Output blob names :
yolo_17
yolo_24
Total number of layers: 50
Total number of layers on DLA: 0
Building the TensorRT Engine…
Building complete!
0:00:16.642073846 16865 0x5598a54300 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger: NvDsInferContext[UID 1]:generateTRTModel(): Storing the serialized cuda engine to file at /home/nvidia/deepstream-4.0/sources/apps/sample_apps/deepstream-test2_rtsp/model_b1_int8.engine
Deserialize yoloLayerV3 plugin: yolo_17
Deserialize yoloLayerV3 plugin: yolo_24
Decodebin child added: source
Running…
Decodebin child added: decodebin0
WARNING from element uri-decode-bin: No decoder available for type ‘application/x-rtp, media=(string)application, payload=(int)97, clock-rate=(int)90000, encoding-name=(string)VND.ONVIF.METADATA, a-tool=(string)“LIVE555\ Streaming\ Media\ v2015.10.12”, a-type=(string)broadcast, x-qt-text-nam=(string)“Video\ Session\ streamed\ by\ "TRUEN"”, x-qt-text-inf=(string)“Truen\ Streaming\ Media\ v2016.01.08”, clock-base=(uint)2621185694, seqnum-base=(uint)54006, npt-start=(guint64)0, play-speed=(double)1, play-scale=(double)1, ssrc=(uint)3804058159’.
Warning: No decoder available for type ‘application/x-rtp, media=(string)application, payload=(int)97, clock-rate=(int)90000, encoding-name=(string)VND.ONVIF.METADATA, a-tool=(string)“LIVE555\ Streaming\ Media\ v2015.10.12”, a-type=(string)broadcast, x-qt-text-nam=(string)“Video\ Session\ streamed\ by\ "TRUEN"”, x-qt-text-inf=(string)“Truen\ Streaming\ Media\ v2016.01.08”, clock-base=(uint)2621185694, seqnum-base=(uint)54006, npt-start=(guint64)0, play-speed=(double)1, play-scale=(double)1, ssrc=(uint)3804058159’.
Decodebin child added: decodebin1
Decodebin child added: rtph264depay0
Decodebin child added: h264parse0
Decodebin child added: capsfilter0
Decodebin child added: nvv4l2decoder0
Seting bufapi_version
Opening in BLOCKING MODE
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
In cb_newpad
Creating LL OSD context new
Frame Number = 0 Number of objects = 0 Vehicle Count = 0 Person Count = 0
Frame Number = 1 Number of objects = 0 Vehicle Count = 0 Person Count = 0
Frame Number = 2 Number of objects = 0 Vehicle Count = 0 Person Count = 0
Frame Number = 3 Number of objects = 0 Vehicle Count = 0 Person Count = 0
Frame Number = 4 Number of objects = 0 Vehicle Count = 0 Person Count = 0
Frame Number = 5 Number of objects = 0 Vehicle Count = 0 Person Count = 0

Screenshot from 2019-10-18 11-41-29.png
Screenshot from 2019-10-18 11-41-29.png2560×1080 391 KB

6

Hi, is the cfg of your yolo custom model with darknet different from yolov3-tiny?
if it is different, can you tell me how to inference with the different cfg and weights(train by darkenet,but cfg is not same to yolov3) on deep stream.

Thank you

7

Are you talking about cfg file in deepstream objectDetector_Yolo/yolov3-tiny.cfg?
I changed the classes and filters a bit when training on darknet, but the structure is the same.

8

Hi, the cfg file what I mean is in deepstream objectDetector_Yolo/yolov3-tiny.cfg.
If I want to add some layers like maxpoll in yolov3.cfg (also trained on darknet), is it ok? or what should I do for more? (Sorry my English is not good)

Thanks for your reply.

9

Hi

What are the changes you have made in your custom model compared to standard yolov3-tiny ? Can you please check if the anchors and masks of your custom model, matches the ones from the standard model here in function “NvDsInferParseCustomYoloV3Tiny” in “nvdsparsebbox_Yolo.cpp” ?

And from which file does class_id in nvdsmeta.h struct NvCsObjectMeta come from?
model-file? or proto-file? or is there another variable?

It’s from the labels file. You can check “label-file-path” variable in config file.

10

sorry for late reply.
As I said before, I didn’t change the structure of yolo, but I changed the filter number. If you put the maxpooling layer, adjust the weights accordingly, and then put it in the deepstream cfg file, I think Yolo will work with it.

11

Thanks for your reply.
In the nvdsparsebbox_Yolo.cpp file the masks and anchors are the same as my YOLO model. The mask is {{3,4,5},
{0,1,2}}
My cfg file does nothing but change the filters and classes. filter = 255 portion of my darknet to 30.

[net]

Testing

batch=1
subdivisions=1

Training

#batch=64
#subdivisions=2
width=416
height=416
channels=3
momentum=0.9
decay=0.0005
angle=0
saturation = 1.5
exposure = 1.5
hue=.1

learning_rate=0.001
burn_in=1000
max_batches = 60000
policy=steps
steps=400000,450000
scales=.1,.1

[convolutional]
batch_normalize=1
filters=16
size=3
stride=1
pad=1
activation=leaky

[maxpool]
size=2
stride=2

[convolutional]
batch_normalize=1
filters=32
size=3
stride=1
pad=1
activation=leaky

[maxpool]
size=2
stride=2

[convolutional]
batch_normalize=1
filters=64
size=3
stride=1
pad=1
activation=leaky

[maxpool]
size=2
stride=2

[convolutional]
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=leaky

[maxpool]
size=2
stride=2

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[maxpool]
size=2
stride=2

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

[maxpool]
size=2
stride=1

[convolutional]
batch_normalize=1
filters=1024
size=3
stride=1
pad=1
activation=leaky

###########

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

[convolutional]
size=1
stride=1
pad=1
filters=30
activation=linear

[yolo]
mask = 3,4,5
anchors = 10,14, 23,27, 37,58, 81,82, 135,169, 344,319
classes=5
num=6
jitter=.3
ignore_thresh = .7
truth_thresh = 1
random=1

[route]
layers = -4

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[upsample]
stride=2

[route]
layers = -1, 8

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[convolutional]
size=1
stride=1
pad=1
filters=30
activation=linear

[yolo]
mask = 0,1,2
anchors = 10,14, 23,27, 37,58, 81,82, 135,169, 344,319
classes=5
num=6
jitter=.3
ignore_thresh = .5
truth_thresh = 1
random=1

I recently achieved my goal because I can real time object detection with YOLOV3-tiny and tracking ID can be extracted without using the dtest2 sample, but I wonder why my model was not available in dtest2.

12

I recently achieved my goal because I can real time object detection with YOLOV3-tiny and tracking ID can be extracted without using the dtest2 sample, but I wonder why my model was not available in dtest2.

So are you saying you were able to get the correct results with objectDetector_Yolo sample but didnt see any detections with dstest2 app ?

13

Yes I checked the masks and anchors to use the yolo model file, changed from resnet10.caffemodel to yolov3-tiny_50000.weights custom model-file, changed the number of classes and made various changes but it didn’t work.

14

Have you been able to solve your issue ? If not, can you explain in more detail what you mean by ? -

"I recently achieved my goal because I can real time object detection with YOLOV3-tiny and tracking ID can be extracted without using the dtest2 sample, but I wonder why my model was not available in dtest2. "

15

Hi jhld8396

We haven’t heard back from you in a couple weeks, so marking this issue resolved by “accepting” this comment as answer.
Please open a new forum issue when you are ready and we’ll pick it up there.