Tao-converter not working for custom pointpillars

[pamasi]

Hi, I’ve trained the PointPillarNet using a custom dataset with the following spec file
pointpillars_exp1_128.yaml (4.8 KB)

and I used the following weight
checkpoint_epoch_100.tlt (58.4 MB)

When I run the command

pointpillars export -m checkpoint_epoch_100.tlt -k perception -o tao_custom_128.etlt -e pointpillars_exp1_128.yaml

it works fine, but when I want to convert it to tensor_rt file using the command
tao-converter tao_custom_128.etlt -k perception -e trt.engine -p points,1x25000x4,1x25000x4,1x204800x4 -p num_points,1,1,1 it does not work and it says:

[ERROR] ModelImporter.cpp:743: — End node —
[ERROR] ModelImporter.cpp:746: ERROR: builtin_op_importers.cpp:5374 In function importFallbackPluginImporter:
[8] Assertion failed: plugin && “Could not create plugin”
Assertion failed: plugin && “Could not create plugin”
[ERROR] Failed to parse the model, please check the encoding key to make sure it’s correct
[INFO] Detected input dimensions from the model: (-1, 25000, 4)
[INFO] Detected input dimensions from the model: (-1)
[INFO] Model has dynamic shape. Setting up optimization profiles.
[INFO] Using optimization profile min shape: (1, 25000, 4) for input: points
[INFO] Using optimization profile opt shape: (1, 25000, 4) for input: points
[INFO] Using optimization profile max shape: (1, 204800, 4) for input: points
[INFO] Using optimization profile min shape: (1) for input: num_points
[INFO] Using optimization profile opt shape: (1) for input: num_points
[INFO] Using optimization profile max shape: (1) for input: num_points
[ERROR] 4: [network.cpp::validate::2738] Error Code 4: Internal Error (Network must have at least one output)

Does someone know how to fix it?
Just as info, I’m using the nvidia-docker image for tao-toolkit.

[_Supersu]

Try this comand
tao-converter tao_custom_128.etlt -k perception -e trt.engine -p points,1x25000x4,1x25000x4,1x25000x4 -p num_points,1,1,1

[pamasi]

It still doesn’t work, this what I get now:

[INFO] [MemUsageChange] Init CUDA: CPU +12, GPU +0, now: CPU 24, GPU 78 (MiB)
[INFO] [MemUsageChange] Init builder kernel library: CPU +264, GPU +74, now: CPU 340, GPU 152 (MiB)
[INFO] ----------------------------------------------------------------
[INFO] Input filename: /tmp/file0mo25a
[INFO] ONNX IR version: 0.0.8
[INFO] Opset version: 11
[INFO] Producer name: pytorch
[INFO] Producer version: 1.13.0
[INFO] Domain:
[INFO] Model version: 0
[INFO] Doc string:
[INFO] ----------------------------------------------------------------
[WARNING] onnx2trt_utils.cpp:377: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[INFO] No importer registered for op: VoxelGeneratorPlugin. Attempting to import as plugin.
[INFO] Searching for plugin: VoxelGeneratorPlugin, plugin_version: 1, plugin_namespace:
[INFO] Successfully created plugin: VoxelGeneratorPlugin
[INFO] No importer registered for op: PillarScatterPlugin. Attempting to import as plugin.
[INFO] Searching for plugin: PillarScatterPlugin, plugin_version: 1, plugin_namespace:
[INFO] Successfully created plugin: PillarScatterPlugin
[INFO] No importer registered for op: DecodeBbox3DPlugin. Attempting to import as plugin.
[INFO] Searching for plugin: DecodeBbox3DPlugin, plugin_version: 1, plugin_namespace:
[INTERNAL_ERROR] Validation failed: static_cast<size_t>(num_classes_) * 2 * 4 == anchors_.size()
/opt/trt_oss_src/TensorRT/plugin/decodeBbox3DPlugin/decodeBbox3D.cpp:79

[ERROR] std::exception
[ERROR] ModelImporter.cpp:740: While parsing node number 58 [DecodeBbox3DPlugin → “output_boxes”]:
[ERROR] ModelImporter.cpp:741: — Begin node —
[ERROR] ModelImporter.cpp:742: input: “cls_preds”
input: “box_preds”
input: “dir_cls_preds”
output: “output_boxes”
output: “num_boxes”
name: “DecodeBbox3DPlugin_0”
op_type: “DecodeBbox3DPlugin”
attribute {
name: “point_cloud_range”
floats: 0
floats: -32
floats: -0.6
floats: 32
floats: 32
floats: 0.6
type: FLOATS
}
attribute {
name: “num_dir_bins”
i: 2
type: INT
}
attribute {
name: “dir_offset”
f: 0.78539
type: FLOAT
}
attribute {
name: “dir_limit_offset”
f: 0
type: FLOAT
}
attribute {
name: “score_thresh”
f: 0.1
type: FLOAT
}
attribute {
name: “anchor_bottom_height”
floats: -0.139
type: FLOATS
}
attribute {
name: “anchors”
floats: 0.238
floats: 0.239
floats: 0.378
floats: 0
type: FLOATS
}

[ERROR] ModelImporter.cpp:743: — End node —
[ERROR] ModelImporter.cpp:746: ERROR: builtin_op_importers.cpp:5374 In function importFallbackPluginImporter:
[8] Assertion failed: plugin && “Could not create plugin”
Assertion failed: plugin && “Could not create plugin”
[ERROR] Failed to parse the model, please check the encoding key to make sure it’s correct
[INFO] Detected input dimensions from the model: (-1, 25000, 4)
[INFO] Detected input dimensions from the model: (-1)
[INFO] Model has dynamic shape. Setting up optimization profiles.
[INFO] Using optimization profile min shape: (1, 25000, 4) for input: points
[INFO] Using optimization profile opt shape: (1, 25000, 4) for input: points
[INFO] Using optimization profile max shape: (1, 25000, 4) for input: points
[INFO] Using optimization profile min shape: (1) for input: num_points
[INFO] Using optimization profile opt shape: (1) for input: num_points
[INFO] Using optimization profile max shape: (1) for input: num_points
[ERROR] 4: [network.cpp::validate::2738] Error Code 4: Internal Error (Network must have at least one output)
[ERROR] Unable to create engine
Segmentation fault (core dumped)

Just for clearness I’ve add the entire output from the command

[Morganh]

Which Tensorrt version did you sync?
Could you use

git clone -b 22.02 https://github.com/NVIDIA/TensorRT.git TensorRT

as well to double check?

[pamasi]

I used the one available from docker image from nvcr.io/nvidia/tao/tao-toolkit tag 4.0.0-pyt, which is 8.5.0

Is there any docker available with the TensorRT version you suggested or should I compile everything?
Because if I used the weight trained from KITTI dataseton the website and default configuration it works, but with mine (trained on a custom dataset) it doesn’t.

[_Supersu]

which tao-converter did you use?
In my case, I download TAO Converter | NVIDIA NGC
Also, there is no tao-converter in tao-toolkit docker image!

[pamasi]

Actually it is present in the image that I referred above and it’s the last version:

Anyway I’ve tried the conversion even using the tao-converter binary that you linked and it doesn’t work neither using the latest version nor oldest one and the error is still the same

[Morganh]

Hi,
As seen above, there is tao-converter inside the tao pytorch docker.

To narrow down the error

@pamasi

[INTERNAL_ERROR] Validation failed: static_cast<size_t>(num_classes_) * 2 * 4 == anchors_.size()
/opt/trt_oss_src/TensorRT/plugin/decodeBbox3DPlugin/decodeBbox3D.cpp:79

[ERROR] std::exception
[ERROR] ModelImporter.cpp:740: While parsing node number 58 [DecodeBbox3DPlugin → “output_boxes”]:
[ERROR] ModelImporter.cpp:741: — Begin node —
[ERROR] ModelImporter.cpp:742: input: “cls_preds”

Could you build and replace the libnvifer_plugin.so inside the tao pytorch docker ? Please refer to below step.

https://github.com/NVIDIA-AI-IOT/tao_toolkit_recipes/tree/main/tao_pointpillars/tensorrt_sample

git clone -b 22.02 https://github.com/NVIDIA/TensorRT.git TensorRT
cd TensorRT
git submodule update --init --recursive
mkdir -p build && cd build
cmake .. -DCUDA_VERSION=$CUDA_VERSION -DGPU_ARCHS=$GPU_ARCHS
make nvinfer_plugin -j$(nproc)
make nvinfer_plugin_static -j$(nproc)
cp libnvinfer_plugin.so.8.2.* /usr/lib/$ARCH-linux-gnu/libnvinfer_plugin.so.8.2.3
cp libnvinfer_plugin_static.a /usr/lib/$ARCH-linux-gnu/libnvinfer_plugin_static.a

More, please check if you run notebok successfully. Also, please make sure if you can use the official etlt file from ngc as well.

[pamasi]

@_Supersu

tao-converter tao_custom_128.etlt -k perception -e trt.engine -p points,1x25000x4,1x25000x4,1x25000x4 -p num_points,1,1,1

So guys, I’ve found the problem! Basically, in my setup I’ve decided to have one value for anchor_rotations since in my dataset there is no rotation, which leads to an array of 1 element, but the converter works only if there are two values specified.

I do not know whose responsible for the development of this framework, but for future release I highly suggest to permit to specific no rotation in even if it is a rare case. Also because I had to redone the training, which is not ideal.