Tao-converter error

h9945394143 · October 22, 2021, 6:52pm

NVIDIA-SMI 460.32.03
Driver Version: 460.32.03
CUDA Version: 11.2

i get the above error

command :

./tao-converter -k nvidia_tlt -p Input,1x3x384x1248,8x3x384x1248,16x3x384x1248 -d 3,384,1248 -o BatchedNMS -e trt.engine -m 1 -t fp16 -i nchw /app/tao-convertor/models/model//yolov4_resnet18_epoch_170.etlt

 /app/tao-convertor/models/model//yolov4_resnet18_epoch_170.etlt
[WARNING] onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[INFO] ModelImporter.cpp:135: No importer registered for op: BatchedNMSDynamic_TRT. Attempting to import as plugin.
[INFO] builtin_op_importers.cpp:3771: Searching for plugin: BatchedNMSDynamic_TRT, plugin_version: 1, plugin_namespace: 
[INFO] builtin_op_importers.cpp:3788: Successfully created plugin: BatchedNMSDynamic_TRT
[INFO] Detected input dimensions from the model: (-1, 3, 384, 1248)
[INFO] Model has dynamic shape. Setting up optimization profiles.
[INFO] Using optimization profile min shape: (1, 3, 384, 1248) for input: Input
[INFO] Using optimization profile opt shape: (8, 3, 384, 1248) for input: Input
[INFO] Using optimization profile max shape: (16, 3, 384, 1248) for input: Input
[INFO] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
[INTERNAL_ERROR] Assertion failed: status == STATUS_SUCCESS
/home/ubuntu/TensorRT/plugin/batchedNMSPlugin/batchedNMSPlugin.cpp:246
Aborting...




Aborted

Morganh · October 23, 2021, 10:09am

Is above correct?
How about /app/tao-convertor/models/model/yolov4_resnet18_epoch_170.etlt

h9945394143 · October 24, 2021, 4:46am

correct

Morganh · October 24, 2021, 6:05am

Please refer to Error while converting model using TAO
and Error while converting model using TAO

Make check

TRT OSS plugin is built and replaced
tao-converter version is compatible
Or use official model to test

h9945394143 · October 25, 2021, 5:11am

i was able to generate the engine file , but while running i get the below error message “ABORTED” on V100…

Creating sink bin
sink-bin-00
Creating nvvidconv pre

Creating nvosd

Creating nvvidconv

Creating H264 Encoder
Creating H264 rtppay

 updsink_port_num = %u 6001
linking srcpad to sinkpad # sinbin  0


 uri = %u /ds-test

 updsink_port_num = %u 6001

 *** DeepStream: Launched RTSP Streaming at rtsp:// 0.0.0.0 : 26001 /ds-test ***


Now playing...
1 :  file:///app/uploads/videos/tempUser/337817d77a7d76c251507f0c850be339
Starting pipeline

gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_mot_klt.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
gstnvtracker: Past frame output is OFF
0:00:05.253206092   694      0x1af0e90 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1702> [UID = 1]: deserialized trt engine from :/app/model/IndianVehicle_model/trt.engine
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:685 [FullDims Engine Info]: layers num: 5
0   INPUT  kFLOAT Input           3x384x1248      min: 1x3x384x1248    opt: 8x3x384x1248    Max: 16x3x384x1248
1   OUTPUT kINT32 BatchedNMS      1               min: 0               opt: 0               Max: 0
2   OUTPUT kFLOAT BatchedNMS_1    200x4           min: 0               opt: 0               Max: 0
3   OUTPUT kFLOAT BatchedNMS_2    200             min: 0               opt: 0               Max: 0
4   OUTPUT kFLOAT BatchedNMS_3    200             min: 0               opt: 0               Max: 0

0:00:05.253342517   694      0x1af0e90 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1806> [UID = 1]: Use deserialized engine model: /app/model/IndianVehicle_model/trt.engine
0:00:05.271375736   694      0x1af0e90 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary-inference> [UID 1]: Load new model:config/Vehicle/PoC_pgie_config.txt sucessfully
Decodebin child added: source

Decodebin child added: decodebin0

Decodebin child added: qtdemux0

Decodebin child added: multiqueue0

Decodebin child added: h264parse0

Decodebin child added: capsfilter0

Decodebin child added: aacparse0

Decodebin child added: avdec_aac0

Decodebin child added: nvv4l2decoder0

In cb_newpad

gstname= video/x-raw
features= <Gst.CapsFeatures object at 0x7f643981a3a8 (GstCapsFeatures at 0x7f62a403a600)>
In cb_newpad

gstname= audio/x-raw
ERROR: ../nvdsinfer/nvdsinfer_func_utils.cpp:33 [TRT]: Assertion failed: status == STATUS_SUCCESS
/home/ubuntu/TensorRT/plugin/batchedNMSPlugin/batchedNMSPlugin.cpp:246
Aborting...

Aborted

Morganh · October 25, 2021, 6:33am

I am afraid the libnvinfer_plugin.so is not correct. How did you build it?

h9945394143 · October 25, 2021, 6:36am

i changed fp16 to fp32 then i was able to generate engine file for tlt model,

./tao-converter -k nvidia_tlt -p Input,1x3x384x1248,8x3x384x1248,16x3x384x1248 -d 3,384,1248 -o BatchedNMS -e trt.engine -m 1 -t fp32 -i nchw /app/tao-convertor/models/model//yolov4_resnet18_epoch_170.etlt

and when i try to run , i get the above error …

"I’m in a process for moving code from one system to another system and this happened in between "

Morganh · October 25, 2021, 9:14am

What do you mean by “moving code” ? What is the code?

h9945394143 · October 25, 2021, 10:13am

from T4 machine to V100 machine

Morganh · October 25, 2021, 10:17am

What is the code?

h9945394143 · October 25, 2021, 10:18am

complete deepstream apps

Morganh · October 25, 2021, 10:21am

In V100, can you delete engine and try? Just let deepstream generate engine.
Please share the config file too.

h9945394143 · October 25, 2021, 10:27am

im using tao-convertor to generate the engine file , i genereated but am getting the above error while runnning the application

Morganh · October 25, 2021, 10:31am

In V100, did you build /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so ?

h9945394143 · October 25, 2021, 10:34am

NO , since we are using tao-convertor , we do not need usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so . Correct ?

Morganh · October 25, 2021, 10:37am

Refer to YOLOv4 — TAO Toolkit 3.22.05 documentation
Strongly recommend you go through
Error while converting model using TAO
and Error in Yolov4 engine conversion, - #22 by Morganh as well.

h9945394143 · October 25, 2021, 10:43am

Morganh · October 25, 2021, 10:51am

For dgpu , please check
/usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so

It is needed to build a new TRT OSS plugin and replace.

h9945394143 · October 26, 2021, 11:26am

below is ie config file , and i have regerated libnvds_infercustomparser_tlt.so file


[property]
gpu-id=0
net-scale-factor=1.0
offsets=103.939;116.779;123.68
model-color-format=1
labelfile-path=../../model/Helmet_model/labels.txt
model-engine-file=/app/tao-convertor/cuda11.0-trt7.1-20210820T231035Z-001/cuda11.0-trt7.1/trt.engine

#tlt-encoded-model=../../model/Helmet_model/yolov4_resnet18_epoch_050.etlt
tlt-model-key=nvidia_tlt
infer-dims=3;384;1248
maintain-aspect-ratio=1
uff-input-order=0
uff-input-blob-name=Input
batch-size=1
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=0
num-detected-classes=2
interval=0
gie-unique-id=1
is-classifier=0
#network-type=0
#no cluster
cluster-mode=3
output-blob-names=BatchedNMS
parse-bbox-func-name=NvDsInferParseCustomBatchedNMSTLT
custom-lib-path=/app/deepstream_tlt_apps/post_processor/libnvds_infercustomparser_tlt.so



[class-attrs-0]
post-cluster-threshold=1
group-threshold=1
## Set eps=0.7 and minBoxes for enable-dbscan=1
eps=0.7
dbscan-min-score=0.95
#minBoxes=3
roi-top-offset=0
roi-bottom-offset=0
detected-min-w=0
detected-min-h=0
detected-max-w=0
detected-max-h=0

below is the error while running :




Now playing...
1 :  file:///app/1080p_length_extension.mp4
Starting pipeline 

gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_mot_klt.so
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
gstnvtracker: Past frame output is OFF
0:00:03.214675083 26633      0x1de76a0 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1702> [UID = 1]: deserialized trt engine from :/app/tao-convertor/cuda11.0-trt7.1-20210820T231035Z-001/cuda11.0-trt7.1/trt.engine
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:685 [FullDims Engine Info]: layers num: 5
0   INPUT  kFLOAT Input           3x384x1248      min: 1x3x384x1248    opt: 8x3x384x1248    Max: 16x3x384x1248   
1   OUTPUT kINT32 BatchedNMS      1               min: 0               opt: 0               Max: 0               
2   OUTPUT kFLOAT BatchedNMS_1    200x4           min: 0               opt: 0               Max: 0               
3   OUTPUT kFLOAT BatchedNMS_2    200             min: 0               opt: 0               Max: 0               
4   OUTPUT kFLOAT BatchedNMS_3    200             min: 0               opt: 0               Max: 0               

0:00:03.214823575 26633      0x1de76a0 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1806> [UID = 1]: Use deserialized engine model: /app/tao-convertor/cuda11.0-trt7.1-20210820T231035Z-001/cuda11.0-trt7.1/trt.engine
0:00:03.232216943 26633      0x1de76a0 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary-inference> [UID 1]: Load new model:config/Helmet/PoC_pgie_config.txt sucessfully
Decodebin child added: source 

Decodebin child added: decodebin0 

Starting wsserver
WebSock Server running with port: %d 36201
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python3.6/asyncio/base_events.py", line 1062, in create_server
    sock.bind(sa)
OSError: [Errno 98] Address already in use

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "PoC_Helmet.py", line 1168, in wsserver
    loop.run_until_complete(start_server)
  File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
    return future.result()
  File "/usr/lib/python3.6/asyncio/tasks.py", line 537, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/usr/local/lib/python3.6/dist-packages/websockets/legacy/server.py", line 1071, in __await_impl__
    server = await self._create_server()
  File "/usr/lib/python3.6/asyncio/base_events.py", line 1066, in create_server
    % (sa, err.strerror.lower()))
OSError: [Errno 98] error while attempting to bind on address ('0.0.0.0', 36201): address already in use
Decodebin child added: qtdemux0 

Decodebin child added: multiqueue0 

Decodebin child added: h264parse0 

Decodebin child added: capsfilter0 

Decodebin child added: aacparse0 

Decodebin child added: avdec_aac0 

Decodebin child added: nvv4l2decoder0 

In cb_newpad

gstname= video/x-raw
features= <Gst.CapsFeatures object at 0x7f00a5fa32e8 (GstCapsFeatures at 0x7eff1002e7a0)>
In cb_newpad

gstname= audio/x-raw
ERROR: ../nvdsinfer/nvdsinfer_func_utils.cpp:33 [TRT]: Assertion failed: status == STATUS_SUCCESS
/home/ubuntu/TensorRT/plugin/batchedNMSPlugin/batchedNMSPlugin.cpp:246
Aborting...

Aborted

Morganh · October 26, 2021, 12:49pm

Did you build /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so according to deepstream_tao_apps/TRT-OSS/x86 at release/tao3.0 · NVIDIA-AI-IOT/deepstream_tao_apps · GitHub ?

Topic		Replies	Views
[ERROR] Model has dynamic shape but no optimization profile specified. Aborted (core dumped) TAO Toolkit	30	2370	December 13, 2021
Convert model to Jetson Error during model export step in TAO notebook TAO Toolkit	21	2261	February 15, 2022
Error in Yolov4 engine conversion, TAO Toolkit	43	2752	October 26, 2021
TAO converter TAO Toolkit cuda , tao	11	1016	August 11, 2023
Tao-converter failed to convert etlt to engine file due to could not find any implementation for node conv1/convolution + activate_1/Relu6 TAO Toolkit	9	954	April 26, 2022
The effect is very poor when converted to trt TAO Toolkit tensorrt , ubuntu	61	1983	September 11, 2023
Cannot run tao-converter on DS6.1 TAO Toolkit deepstream61	9	521	July 11, 2022
Tao-converter doesn't work for Deepstream 6.1 TAO Toolkit	7	854	July 14, 2022
TAO converter for linux Ubuntu 20.04 TAO Toolkit	14	772	August 2, 2023
Error in integrating Yolov4 in Deepstream 6, 6.1, 6.1.1, and 6.2 TAO Toolkit	14	975	March 21, 2023

Tao-converter error

Related topics