Tensorrt support for SSD_inception trained on custom dataset

Linux version : Ubuntu 16.04 LTS
GPU type : GeForce GTX 1080
nvidia driver version : 410.72
CUDA version : 9.0
CUDNN version : 7.0.5
Python version [if using python] : 3.5.2
Tensorflow version : tensorflow-gpu 1.9
TensorRT version : 5.0.2.6

Actual Problem,

I tried the example script under samples/python/uff_ssd folder. The Script downloads SSD_inception model, creates uff parser, builds engine and performs inference on Image.
These are the results of that:

Preparing pretrained model
Downloading /home/teai/TensorRT/TensorRT-5.0.2.6/targets/x86_64-linux-gnu/samples/python/uff_ssd/utils/../workspace/models/ssd_inception_v2_coco_2017_11_17.tar.gz
Download progress [==================================================] 100%
Download complete
Unpacking /home/teai/TensorRT/TensorRT-5.0.2.6/targets/x86_64-linux-gnu/samples/python/uff_ssd/utils/../workspace/models/ssd_inception_v2_coco_2017_11_17.tar.gz
Extracting complete
Removing /home/teai/TensorRT/TensorRT-5.0.2.6/targets/x86_64-linux-gnu/samples/python/uff_ssd/utils/../workspace/models/ssd_inception_v2_coco_2017_11_17.tar.gz
Model ready
WARNING: To create TensorRT plugin nodes, please use the `create_plugin_node` function instead.
UFF Version 0.5.5
=== Automatically deduced input nodes ===
[name: "Input"
op: "Placeholder"
attr {
  key: "dtype"
  value {
    type: DT_FLOAT
  }
}
attr {
  key: "shape"
  value {
    shape {
      dim {
        size: 1
      }
      dim {
        size: 3
      }
      dim {
        size: 300
      }
      dim {
        size: 300
      }
    }
  }
}
]
=========================================

Using output node NMS
Converting to UFF graph
Warning: No conversion function registered for layer: NMS_TRT yet.
Converting NMS as custom op: NMS_TRT
Warning: No conversion function registered for layer: FlattenConcat_TRT yet.
Converting concat_box_conf as custom op: FlattenConcat_TRT
Warning: No conversion function registered for layer: GridAnchor_TRT yet.
Converting GridAnchor as custom op: GridAnchor_TRT
Warning: No conversion function registered for layer: FlattenConcat_TRT yet.
Converting concat_box_loc as custom op: FlattenConcat_TRT
No. nodes: 563
UFF Output written to /home/teai/TensorRT/TensorRT-5.0.2.6/targets/x86_64-linux-gnu/samples/python/uff_ssd/utils/../workspace/models/ssd_inception_v2_coco_2017_11_17/frozen_inference_graph.uff
UFF Text Output written to /home/teai/TensorRT/TensorRT-5.0.2.6/targets/x86_64-linux-gnu/samples/python/uff_ssd/utils/../workspace/models/ssd_inception_v2_coco_2017_11_17/frozen_inference_graph.pbtxt
TensorRT inference engine settings:
  * Inference precision - DataType.FLOAT
  * Max batch size - 1

Building TensorRT engine. This may take few minutes.
TensorRT inference time: 4 ms
Detected car with confidence 97%
Total time taken for one image: 54 ms

Now, instead of downloading a pre-trained model, I trained my own object_detection on a custom datasetusing SSD_inception as architecture. I commented out the download part in the script and made the script to look for my trained .pb file. But am getting following errors:

WARNING: To create TensorRT plugin nodes, please use the `create_plugin_node` function instead.
UFF Version 0.5.5
=== Automatically deduced input nodes ===
[name: "Input"
op: "Placeholder"
attr {
  key: "dtype"
  value {
    type: DT_FLOAT
  }
}
attr {
  key: "shape"
  value {
    shape {
      dim {
        size: 1
      }
      dim {
        size: 3
      }
      dim {
        size: 300
      }
      dim {
        size: 300
      }
    }
  }
}
]
=========================================

Using output node NMS
Converting to UFF graph
Warning: No conversion function registered for layer: NMS_TRT yet.
Converting NMS as custom op: NMS_TRT
Warning: No conversion function registered for layer: FlattenConcat_TRT yet.
Converting concat_box_conf as custom op: FlattenConcat_TRT
Warning: No conversion function registered for layer: GridAnchor_TRT yet.
Converting GridAnchor as custom op: GridAnchor_TRT
Warning: No conversion function registered for layer: FlattenConcat_TRT yet.
Converting concat_box_loc as custom op: FlattenConcat_TRT
No. nodes: 781
UFF Output written to /home/teai/TensorRT/TensorRT-5.0.2.6/targets/x86_64-linux-gnu/samples/python/uff_ssd/utils/../workspace/models/ssd_inception_v2_coco_2017_11_17/frozen_inference_graph.uff
UFF Text Output written to /home/teai/TensorRT/TensorRT-5.0.2.6/targets/x86_64-linux-gnu/samples/python/uff_ssd/utils/../workspace/models/ssd_inception_v2_coco_2017_11_17/frozen_inference_graph.pbtxt
TensorRT inference engine settings:
  * Inference precision - DataType.FLOAT
  * Max batch size - 1

[TensorRT] ERROR: Parameter check failed at: ../builder/Layers.h::setAxis::315, condition: axis>=0
[TensorRT] ERROR: Concatenate/concat: all concat input tensors must have the same dimensions except on the concatenation axis
[TensorRT] ERROR: UFFParser: Parser error: BoxPredictor_0/ClassPredictor/BiasAdd: The input to the Scale Layer is required to have a minimum of 3 dimensions.
Building TensorRT engine. This may take few minutes.
[TensorRT] ERROR: Network must have at least one output
Traceback (most recent call last):
  File "detect_objects.py", line 193, in <module>
    main()
  File "detect_objects.py", line 166, in main
    batch_size=parsed['max_batch_size'])
  File "/home/teai/TensorRT/TensorRT-5.0.2.6/targets/x86_64-linux-gnu/samples/python/uff_ssd/utils/inference.py", line 69, in __init__
    engine_utils.save_engine(self.trt_engine, trt_engine_path)
  File "/home/teai/TensorRT/TensorRT-5.0.2.6/targets/x86_64-linux-gnu/samples/python/uff_ssd/utils/engine.py", line 83, in save_engine
    buf = engine.serialize()
AttributeError: 'NoneType' object has no attribute 'serialize'

Stuck with this issue for a long time. Could anyone help me with to resolve

Hello,

It looks like you have several layers not supported by Tensorrt (FlattenConcat_TRT , GridAnchor_TRT , etc…) . However, some of these layers (such as gridanchor_trt) are implemented as a pluging as part of the sampleUffSSD.

Very likely during your modifications of sampleUffSSD to use custom trained model, you “disconnected” the sampleUffSSD plugin infrastructure?

Hi,

I am using the same sampleUFFSSD folder, just made one modification in utils/model.py. The modification goes as follows,
-> Removed the download_model function.
-> Instead added the path of custom trained model(.pb).

Just for confirmation I am using python version of SSD : TensorRT-5.0.2.6/samples/python/uff_ssd
and not the c++ version(sampleUffSSD) which you are referring to.

Hello,

as described in comment #2 and confirmed in #3, looks like you are using a custom model. The custom plugins implemented for sampleUffSSD may no longer apply for your custom model. You’ll need to implement something specific for your model/.pb.

Hi, when I compiled FlattenConcat custom plugin layer,it is always failed in uff_ssd.
when i read the READ.ME:

  • You need to compile FlattenConcat custom plugin. To do that, enter sample
    directory, and run the following commands:
mkdir -p build
cd build
cmake ..
make
cd ..

I can’t execute the “cmake …” command correctly. My execution path is ‘~/samples/python/uff_ssd’. Can you help me ?

Hi,

I am using the uff_ssd example in tensorrt5.1.2.2.
I trained my model with ‘ssd_inception_v2_coco.config’ in tensorflow-api. When I used the script detect_objects.py in uff_ssd, the model was converted from ‘.pb’ to ‘.uff’ and also generated '.pbtxt 'File.
But when building an engine, there is always the same error. The error message is as below.
When I use the default model ‘ssd_inception_v2_coco_2017_11_17’ in the uff_ssd script ‘detect_objects.py’, everything works fine.
Any help will be appreciated!!!

My email is :zhangcunyi_02147@163.com

Linux version : Ubuntu 16.04 LTS
GPU type : GeForce GTX 1080
nvidia driver version : 410.93
CUDA version : 10.0
CUDNN version : 7.4.1
Python version [Anaconda] : 3.6.8
Tensorflow version : tensorflow-gpu 1.13.1
TensorRT version : 5.1.2.2

[libprotobuf FATAL /home/erisuser/p4sw/sw/gpgpu/MachineLearning/DIT/externals/protobuf/x86_64/10.0/include/google/protobuf/repeated_field.h:1408] CHECK failed: (index) < (current_size_): 
Traceback (most recent call last):
  File "detect_objects.py", line 245, in <module>
    main()
  File "detect_objects.py", line 219, in main
    batch_size=args.max_batch_size)
  File "/home/zcy/1_data_sets/TensorRT-5.1.2.2/targets/x86_64-linux-gnu/samples/python/uff_ssd/utils/inference.py", line 115, in __init__
    batch_size=batch_size)
  File "/home/zcy/1_data_sets/TensorRT-5.1.2.2/targets/x86_64-linux-gnu/samples/python/uff_ssd/utils/engine.py", line 75, in build_engine
    parser.parse(uff_model_path, network)
RuntimeError: CHECK failed: (index) < (current_size_):

Linux version : Ubuntu 16.04 LTS
GPU type : GeForce GTX 1080
nvidia driver version : 410.93
CUDA version : 10.0
CUDNN version : 7.4.1
Python version [Anaconda] : 3.6.8
Tensorflow version : tensorflow-gpu 1.13.1
TensorRT version : 5.1.2.2

[libprotobuf FATAL /home/erisuser/p4sw/sw/gpgpu/MachineLearning/DIT/externals/protobuf/x86_64/10.0/include/google/protobuf/repeated_field.h:1408] CHECK failed: (index) < (current_size_): 
Traceback (most recent call last):
  File "detect_objects.py", line 245, in <module>
    main()
  File "detect_objects.py", line 219, in main
    batch_size=args.max_batch_size)
  File "/home/zcy/1_data_sets/TensorRT-5.1.2.2/targets/x86_64-linux-gnu/samples/python/uff_ssd/utils/inference.py", line 115, in __init__
    batch_size=batch_size)
  File "/home/zcy/1_data_sets/TensorRT-5.1.2.2/targets/x86_64-linux-gnu/samples/python/uff_ssd/utils/engine.py", line 75, in build_engine
    parser.parse(uff_model_path, network)
RuntimeError: CHECK failed: (index) < (current_size_):

I am using the uff_ssd example in tensorrt5.1.2.2.
I trained my model with ‘ssd_inception_v2_coco.config’ in tensorflow-api. When I used the script detect_objects.py in uff_ssd, the model was converted from ‘.pb’ to ‘.uff’ and also generated '.pbtxt 'File.
But when building an engine, there is always the same error. The error message is as above.
When I use the default model ‘ssd_inception_v2_coco_2017_11_17’ in the uff_ssd script ‘detect_objects.py’, everything works fine.
Any help will be appreciated!!!

hey I’m having the same issue :(
did you solve it?

hey, same problem here :

When I use a pretrained ssd_inception model (ssd_inception_v2_coco_2018_01_28), I can transform .pb file to .uff file, and preform an inference using SampluUffSSD.

convert-to-uff ssd_inception_v2_coco_2018_01_28/frozen_inference_graph.pb -O NMS -p config_inception.py
cp ssd_inception_v2_coco_2018_01_28/frozen_inference_graph.uff ../../../../data/ssd/sample_ssd_relu6.uff
../../bin/sample_uff_ssd

yields :

Converting concat_box_loc as custom op: FlattenConcat_TRT
No. nodes: 558
UFF Output written to ssd_inception_v2_coco_2018_01_28/frozen_inference_graph.uff

[I] ../../../../data/ssd/sample_ssd_relu6.uff
[I] Begin parsing model...
[I] End parsing model...
[I] Begin building engine...
[I] End building engine...
[I]  Num batches  1
[I]  Data Size  270000
[I] *** deserializing
[I] Time taken for inference is 4.36768 ms.
[I]  KeepCount 100
[I] Detected dog in the image 0 (../../../../data/ssd/dog.ppm) with confidence 89.001 and coordinates (81.7568,23.1155),(295.041,298.62).
[I] Result stored in dog-0.890010.ppm.
[I] Detected dog in the image 0 (../../../../data/ssd/dog.ppm) with confidence 88.0681 and coordinates (1.39267,0),(118.431,237.262).
[I] Result stored in dog-0.880681.ppm.
&&&& PASSED TensorRT.sample_uff_ssd

When using an ssd inception model I trained ( from models/research/object_detection, using ssd_inception_v2_coco_2018_01_28 as init, and its config file, should be the same from my understanding ), I get an error with the same steps :

Converting concat_box_conf as custom op: FlattenConcat_TRT
No. nodes: 811
UFF Output written to model_inception/frozen_inference_graph.uff

[I] ../../../../data/ssd/sample_ssd_relu6.uff
[I] Begin parsing model...
[libprotobuf FATAL /home/erisuser/p4sw/sw/gpgpu/MachineLearning/DIT/externals/protobuf/x86_64/10.0/include/google/protobuf/repeated_field.h:1408] CHECK failed: (index) < (current_size_):
terminate called after throwing an instance of 'google_private::protobuf::FatalException'
  what():  CHECK failed: (index) < (current_size_):
Aborted (core dumped)

Any help appreciated

Perhaps try trt.init_libnvinfer_plugins(TRT_LOGGER, ‘’) .

hey, same problem.

How on earth should I train the model to use TensorRT?

I’m having the exact same issue. I’m using the repo suggested in the Jetson Zoo wiki: https://github.com/AastaNV/TRT_object_detection

After re-training mobilenet v1 (or v2 for that matter), and replacing the “ToFloat” with “Cast” in namespace_plugin_map that’s used with the graphsurgeon, i’m getting the same CHECK failed error:

[TensorRT] INFO: UFFParser: parsing GridAnchor
[libprotobuf FATAL /home/erisuser/p4sw/sw/gpgpu/MachineLearning/DIT/externals/protobuf/aarch64/10.0/include/google/protobuf/repeated_field.h:1408] CHECK failed: (index) < (current_size_): 
Traceback (most recent call last):
  File "main.py", line 41, in <module>
    parser.parse('tmp.uff', network)
RuntimeError: CHECK failed: (index) < (current_size_

There’s a number of threads on this forum and all without answers. Looking at tensorboard for both default tensorflow mobilenet and my re-trained one, I check that input/output dimensions match. I’m now suspecting it’s something to do with the GridAnchor plugin config, but can’t figure out what. Here’s what I have at least, which is just a copypaste from the github repo I linked above.

PriorBox = gs.create_plugin_node(
        name="GridAnchor",
        op="GridAnchor_TRT",
        minSize=0.2,
        maxSize=0.95,
        aspectRatios=[1.0, 2.0, 0.5, 3.0, 0.33],
        variance=[0.1,0.1,0.2,0.2],
        featureMapShapes=[19, 10, 5, 3, 2, 1],
        numLayers=6
    )

It’s frustrating as all hell not to be able to get this working properly, having basic example working is good and all, but barely useful when it comes to actually making any use of TensorRT.

System info for the record

$ python3
>>> tf.__version__
'1.14.0'

$ jtop
 - Board:
    * Name:           NVIDIA Jetson NANO/TX1
    * Type:           NANO/TX1
    * Jetpack:        4.2.1 [L4T 32.2.0]
    * GPU-Arch:       5.3
  - Libraries:
    * CUDA:           10.0.326
    * cuDNN:          7.5.0.56-1+cuda10.0
    * TensorRT:       5.1.6.1-1+cuda10.0
    * VisionWorks:    1.6.0.500n
    * OpenCV:         3.3.1 compiled CUDA: NO

WARNING: To create TensorRT plugin nodes, please use the create_plugin_node function instead.
UFF Version 0.6.5
=== Automatically deduced input nodes ===
[name: “Input”
op: “Placeholder”
attr {
key: “dtype”
value {
type: DT_FLOAT
}
}
attr {
key: “shape”
value {
shape {
dim {
size: 1
}
dim {
size: 3
}
dim {
size: 640
}
dim {
size: 640
}
}
}
}
]

Using output node NMS
Converting to UFF graph
Warning: No conversion function registered for layer: NMS_TRT yet.
Converting NMS as custom op: NMS_TRT
Warning: No conversion function registered for layer: FlattenConcat_TRT yet.
Converting concat_box_loc as custom op: FlattenConcat_TRT
WARNING:tensorflow:From /home/york/anaconda3/envs/tensorRT6.0/lib/python3.7/site-packages/uff/converters/tensorflow/converter.py:179: The name tf.AttrValue is deprecated. Please use tf.compat.v1.AttrValue instead.

Warning: No conversion function registered for layer: GridAnchor_TRT yet.
Converting GridAnchor as custom op: GridAnchor_TRT
Warning: No conversion function registered for layer: FlattenConcat_TRT yet.
Converting concat_box_conf as custom op: FlattenConcat_TRT
DEBUG [/home/york/anaconda3/envs/tensorRT6.0/lib/python3.7/site-packages/uff/converters/tensorflow/converter.py:96] Marking [‘NMS’] as outputs
No. nodes: 515
UFF Output written to /media/york/F/GitHub/tensorflow/train_model/ssd_mobilenet_v2_coco_focal_loss_trafficlight/export_train_bdd100k_baidu_truck_zl004_class4_wh640640_maxscale4_trainval299287_expansion_layer3_cosine_reducebox_v2_step320000-733/frozen_inference_graph.uff
UFF Text Output written to /media/york/F/GitHub/tensorflow/train_model/ssd_mobilenet_v2_coco_focal_loss_trafficlight/export_train_bdd100k_baidu_truck_zl004_class4_wh640640_maxscale4_trainval299287_expansion_layer3_cosine_reducebox_v2_step320000-733/frozen_inference_graph.pbtxt
[TensorRT] VERBOSE: Plugin Creator registration succeeded - GridAnchor_TRT
[TensorRT] VERBOSE: Plugin Creator registration succeeded - NMS_TRT
[TensorRT] VERBOSE: Plugin Creator registration succeeded - Reorg_TRT
[TensorRT] VERBOSE: Plugin Creator registration succeeded - Region_TRT
[TensorRT] VERBOSE: Plugin Creator registration succeeded - Clip_TRT
[TensorRT] VERBOSE: Plugin Creator registration succeeded - LReLU_TRT
[TensorRT] VERBOSE: Plugin Creator registration succeeded - PriorBox_TRT
[TensorRT] VERBOSE: Plugin Creator registration succeeded - Normalize_TRT
[TensorRT] VERBOSE: Plugin Creator registration succeeded - RPROI_TRT
[TensorRT] VERBOSE: Plugin Creator registration succeeded - BatchedNMS_TRT
[TensorRT] VERBOSE: Plugin Creator registration succeeded - FlattenConcat_TRT
TensorRT inference engine settings:

  • Inference precision - DataType.FLOAT
  • Max batch size - 1

[TensorRT] VERBOSE: UFFParser: BoxPredictor_2/Reshape/shape/3 ->
[TensorRT] VERBOSE: UFFParser: Applying order forwarding to: BoxPredictor_2/Reshape/shape/3
[TensorRT] VERBOSE: UFFParser: Parsing BoxPredictor_2/Reshape/shape[Op: Stack]. Inputs: BoxPredictor_2/strided_slice, BoxPredictor_2/Reshape/shape/1, BoxPredictor_2/Reshape/shape/2, BoxPredictor_2/Reshape/shape/3
[TensorRT] VERBOSE: UFFParser: Applying order forwarding to: BoxPredictor_2/Reshape/shape
[TensorRT] VERBOSE: UFFParser: Parsing BoxPredictor_2/Reshape[Op: Reshape]. Inputs: BoxPredictor_2/BoxEncodingPredictor/BiasAdd, BoxPredictor_2/Reshape/shape
[TensorRT] VERBOSE: UFFParser: BoxPredictor_2/Reshape -> [2400,1,4]
[TensorRT] VERBOSE: UFFParser: Applying order forwarding to: BoxPredictor_2/Reshape
[TensorRT] VERBOSE: UFFParser: Parsing concat_box_conf[Op: FlattenConcat_TRT]. Inputs: BoxPredictor_0/Reshape, BoxPredictor_1/Reshape, BoxPredictor_2/Reshape
[TensorRT] VERBOSE: UFFParser: Parsing Squeeze[Op: Squeeze]. Inputs: concat_box_conf
[TensorRT] VERBOSE: UFFParser: Squeeze -> [124800,1,1]
[TensorRT] VERBOSE: UFFParser: Applying order forwarding to: Squeeze
[TensorRT] VERBOSE: UFFParser: Parsing GridAnchor[Op: GridAnchor_TRT].
[libprotobuf FATAL /externals/protobuf/x86_64/10.0/include/google/protobuf/repeated_field.h:1408] CHECK failed: (index) < (current_size
):
Traceback (most recent call last):
File “/media/york/F/GitHub/tensorflow/models/research/auto_driving/uff_ssd-TensorRT-6.0.1.5/detect_objects_trafficlight_trt6.0.py”, line 266, in
main()
File “/media/york/F/GitHub/tensorflow/models/research/auto_driving/uff_ssd-TensorRT-6.0.1.5/detect_objects_trafficlight_trt6.0.py”, line 240, in main
batch_size=args.max_batch_size)
File “/media/york/F/GitHub/tensorflow/models/research/auto_driving/uff_ssd-TensorRT-6.0.1.5/utils/inference.py”, line 116, in init
batch_size=batch_size)
File “/media/york/F/GitHub/tensorflow/models/research/auto_driving/uff_ssd-TensorRT-6.0.1.5/utils/engine.py”, line 125, in build_engine
parser.parse(uff_model_path, network)
RuntimeError: CHECK failed: (index) < (current_size
):

Process finished with exit code 1

what‘ the problem??? tensorflow detection model to tensorRT