[Tesla P4][Deepstream3] After changing the model to another caffe model, NPP runtime error appeared,...

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Hi,

I’am trying to implement Deepstream on Tesla P4 for real time decoding and inferencing.
The default test “deepstream-test3” is successfully running for multiple videos and default caffe model, so then I changed the model to my own caffe model, but encountered an error as below:

root@beb2e1ab2eaf:~/web_server/deepstream3/DeepStream_Release/sources/apps/sample_apps/deepstream-test3# ./deepstream-test3-app https://3.vod2.myqcloud.com/2330e8b5vodcq3/b3d3fd345285890782300872786/playlist.m3u8 https://3.vod2.myqcloud.com/2330e8b5vodcq3/a0f337765285890782300040031/playlist.m3u8 https://3.vod2.myqcloud.com/2330e8b5vodcq3/f25198f85285890782301204111/playlist.m3u8
WARNING: Overriding infer-config batch-size (1) with number of sources (3)
Now playing: https://3.vod2.myqcloud.com/2330e8b5vodcq3/b3d3fd345285890782300872786/playlist.m3u8, https://3.vod2.myqcloud.com/2330e8b5vodcq3/a0f337765285890782300040031/playlist.m3u8, https://3.vod2.myqcloud.com/2330e8b5vodcq3/f25198f85285890782301204111/playlist.m3u8,
>>> Generating new TRT model engine
Using FP32 data type.
Warning: Flatten layer ignored. TensorRT implicitly flattens input to FullyConnected layers, but in other circumstances this will result in undefined behavior.

 ***** Storing serialized engine file as /root/web_server/deepstream3/DeepStream_Release/sources/apps/sample_apps/deepstream-test3/../../../../../../../web_server/incepv3_experiment/incepv3.caffemodel_b3_fp32.engine batchsize = 3 *****

Running...
NPP runtime error -108 at line 836 in file nvbufconvert.cpp
Aborted (core dumped)

Then I gdb it and get:

$ gdb core.154 
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
[New LWP 187]
[New LWP 154]
[New LWP 156]
[New LWP 155]
[New LWP 157]
[New LWP 161]
[New LWP 159]
[New LWP 162]
[New LWP 163]
[New LWP 164]
[New LWP 166]
[New LWP 167]
[New LWP 165]
[New LWP 168]
[New LWP 169]
[New LWP 171]
[New LWP 172]
[New LWP 173]
[New LWP 170]
[New LWP 175]
[New LWP 174]
[New LWP 176]
[New LWP 177]
[New LWP 178]
[New LWP 180]
[New LWP 181]
[New LWP 183]
[New LWP 184]
[New LWP 179]
[New LWP 160]
[New LWP 185]
[New LWP 186]
[New LWP 182]
Missing separate debuginfo for the main executable file
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/9f/4e3a06fb38cae22f4b313f5f8452162b3c616f
Core was generated by `./deepstream-test3-app https://3.vod2.myqcloud.com/2330e8b5vodcq3/b3d3fd3452858'.
Program terminated with signal 6, Aborted.
#0  0x00007ff6abab6428 in ?? ()
"/home/web_server/deepstream3/DeepStream_Release/sources/apps/sample_apps/deepstream-test3/core.154" is a core file.
Please specify an executable to debug.
(gdb) bt
#0  0x00007ff6abab6428 in ?? ()
#1  0x00007ff6abab802a in ?? ()
#2  0x0000000000000020 in ?? ()
#3  0x0000000000000000 in ?? ()
(gdb)

Not sure whether this is related to the shape of input blob. The default model has input blob of (1 3 268 640) in NCHW, and my model has (10 3 299 299) in NCHW.

Is there an argument for this to change the input shape?

Any idea will be welcome.

Thanks,

2

How do you change the config file?
What’s the network of your caffe model?

3

I have changed the config file. Below is my config.
The network is inception v3

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
#model-file=../../../../samples/models/Primary_Detector/resnet10.caffemodel
#proto-file=../../../../samples/models/Primary_Detector/resnet10.prototxt
model-file=../../../../../../../web_server/incepv3_experiment/incepv3.caffemodel
proto-file=../../../../../../../web_server/incepv3_experiment/incepv3.prototxt
labelfile-path=../../../../samples/models/Primary_Detector/labels.txt
int8-calib-file=../../../../samples/models/Primary_Detector/cal_trt4.bin
input-object-min-width = 299
input-object-min-height = 299
input-object-max-width = 299
input-object-max-height = 299
batch-size=10
batch-size=16
network-mode=0
num-detected-classes=4
interval=0
gie-unique-id=1
parse-func=4
#output-blob-names=conv2d_bbox;conv2d_cov/Sigmoid
output-blob-names=prob

[class-attrs-all]
threshold=0.2
eps=0.2
group-threshold=1
4

Can you also comment
#int8-calib-file=…/…/…/…/samples/models/Primary_Detector/cal_trt4.bin
and replace your own labels file.
labelfile-path=…/…/…/…/samples/models/Primary_Detector/labels.txt

If there are layers thensorrT does not supported, it needs to implement tensorRT iplugin. Please refer to
DeepStream_Release/sources objectDetector_FasterRCNN objectDetector_SSD

5

I tried to comment the cat_trt4.bin but it’s still the same.
My model is converted from an Tensorflow inception-v3 model, so may be there exists some issues that can’t be suitable for tensorrt engine. I guess this is the point.
I will try to retrain a new model of resnet and see the performance.

Thanks

6

Yes. Resnet is OK.