ONNX and tensorRT: ERROR: Network must have at least one output

Hi,

I exported a model to ONNX from pytorch 1.0, and tried to load it to tensorRT using:

def build_engine_onnx(model_file):
    with trt.Builder(TRT_LOGGER) as builder, builder.create_network() as network, trt.OnnxParser(network, TRT_LOGGER) as parser:
        builder.max_workspace_size = common.GiB(1)
        # Load the Onnx model and parse it in order to populate the TensorRT network.
        with open(model_file, 'rb') as model:
            parser.parse(model.read())
        return builder.build_cuda_engine(network)
build_engine_onnx(model_path)

I have received the following error:

ERROR: Network must have at least one output.

When I tried to print the model, I get:

output {
    name: "output1"
    type {
      tensor_type {
        elem_type: FLOAT
        shape {
          dim {
            dim_value: 1
          }
          dim {
            dim_value: 17
          }
          dim {
            dim_value: 56
          }
          dim {
            dim_value: 64
          }
        }
      }
    }
  }
}
opset_import {
  version: 9
}

So it seems there is an output layer, but the opset is 9.
How can I resolve this?

Hello,

Can you share the onnx file with us? You can DM me if you’d like to keep the file off of the forum.

Also, can you provide details on the platforms you are using?

Linux distro and version
GPU type
nvidia driver version
CUDA version
CUDNN version
Python version [if using python]
Tensorflow version
TensorRT version

Hi,

I’m using Ubuntu 18.04, geforce 1070 (mobile), nvidia driver 410, cuda 10, cudnn 7.4.1, python 3.6, tensorRT 5.0.2.6.
I’ve sent the ONNX by a private message.

Hello,
Which version of onnx parser are you using? Or the one that came default with TRt 5.0.2.6?

Seems like there are more issues than simply the “Network must have at least one output” error.

(trt) kevinchen@kevinch-desktop:~/Downloads/TensorRT-5.0.2.6/trt/work$ python test_onnx.py
Unsupported ONNX data type: INT64 (7)
[TensorRT] ERROR: Parameter check failed at: ../builder/Layers.cpp::ConstantLayer::1570, condition: weights.type == DataType::kFLOAT || weights.type == DataType::kHALF || weights.type == DataType::kINT32
[TensorRT] ERROR: (Unnamed Layer* 13) [Constant]: invalid dimensions
Unsupported ONNX data type: INT64 (7)
[TensorRT] ERROR: Parameter check failed at: ../builder/Layers.cpp::ConstantLayer::1570, condition: weights.type == DataType::kFLOAT || weights.type == DataType::kHALF || weights.type == DataType::kINT32
[TensorRT] ERROR: (Unnamed Layer* 14) [Gather]: indices input must be Int32
[TensorRT] ERROR: (Unnamed Layer* 16) [Constant]: invalid dimensions
[TensorRT] ERROR: Network must have at least one output

Hi,

I have created the file using the ONNX using pyTorch 1.0 (also tried 0.4.1, same errors).
I have imported the file using tensorRT ONNX parser (the build-in one).

Any ideas what should I do next?

Ping…
Any hints?

Hello,

It seems that there is an issue in the convertor process that’s not properly flagging the output tensor. Engineering is still investigating.

In the meantime, a work-around is if you manually flag it with the API, i.e mark the output tensor of the last layer of the network as the output of the network with:

network.mark_output(network.get_layer(network.num_layers-1).get_output(0))

Then the TRT engine should be built OK.

1 Like

Thanks. I have modified the function build_engine_onnx to be:

def build_engine_onnx(model_file):
    with trt.Builder(TRT_LOGGER) as builder, builder.create_network() as network, trt.OnnxParser(network, TRT_LOGGER) as parser:
        builder.max_workspace_size = common.GiB(1)
        # Load the Onnx model and parse it in order to populate the TensorRT network.
        with open(model_file, 'rb') as model:
            parser.parse(model.read())
        network.mark_output(network.get_layer(network.num_layers - 1).get_output(0))
        return builder.build_cuda_engine(network)

The number of detected network layers is only 12 (network.num_layers=12), however, the ONNX file contains ~1000 entries.
Does that make sense?

i have the same probilem . my envirment is Ubuntu16.04, GTX 1070, nvidia driver 384.130 , cuda 9, cudnn 7.1 , python 2.7, tensorRT 5.0.2.6

@bloodymeli
Seems like your ONNX model is currently not supported within TensorRT.

Specifically:

(trt) kevinchen@kevinch-desktop:~/Downloads/TensorRT-5.0.2.6/trt/work$ python test_onnx.py
Unsupported ONNX data type: INT64 (7)
[TensorRT] ERROR: Parameter check failed at: ../builder/Layers.cpp::ConstantLayer::1570, condition: weights.type == DataType::kFLOAT || weights.type == DataType::kHALF || weights.type == DataType::kINT32
[TensorRT] ERROR: (Unnamed Layer* 13) [Constant]: invalid dimensions
Unsupported ONNX data type: INT64 (7)
[TensorRT] ERROR: Parameter check failed at: ../builder/Layers.cpp::ConstantLayer::1570, condition: weights.type == DataType::kFLOAT || weights.type == DataType::kHALF || weights.type == DataType::kINT32
[TensorRT] ERROR: (Unnamed Layer* 14) [Gather]: indices input must be Int32
[TensorRT] ERROR: (Unnamed Layer* 16) [Constant]: invalid dimensions
[TensorRT] ERROR: Network must have at least one output

Layer 13 is unsupported, which causes the network to only be built up to the 12th layer. Looks like you are using INT64 type, which TRT does not support. If you can recreate your model using a supported datatype (float32, float 16, int32) you may be able to go further in the parsing step.

thx, i will try

Hi,

I have removed all the int64 layers, but I’m still unable to go past layer 13. I’ve sent you an updated ONNX file with no int64 layers. The total number of layers is 1117, conversation breaks at layer 13.

Also, is there a way to closely debug the conversation process? In other words, is parser implemented in python? I can’t find this library.

Is there any official examples for transferring the pytorch model to tensorrt? With the workflow, pytorch–>(torch.onnx.export)–>onnx–>tensorrt?

I encountered this problem also, ans i have made a try to export onnx style files with pytorch torch.onnx.export function, while many operations(such as lstm, view and unfold…) encountered this problem: Network must have at least one output.

Just wonder if the problem is solved on your side? I had the same problem. The parser failed to parse the whole network, it breaks at a certain layer. Thanks.

Same problem. Why not TensorRT supports parser for turning PyTorch model to TensorRT model directly. There are so many problems with the intermediary ONXX

1 Like

Oh My God. Seems I cannot avoid using int64 in my PyTorch Model.
I am using PyTorch, and I want to use TensorRT for speeding up the inference of model.
This is my workflow: PyTorch model -> torch.onnx.export the onnx model -> tensorrt onnx_parser load the model.
Finally I get the same problem: INT64 is not supported.
This is my PyTorch Model definition:

class Mnist_CNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 16, kernel_size=3, stride=2, padding=1)
        self.conv2 = nn.Conv2d(16, 16, kernel_size=3, stride=2, padding=1)
        self.conv3 = nn.Conv2d(16, 10, kernel_size=3, stride=2, padding=1)

    def forward(self, xb):
        xb = F.relu(self.conv1(xb))
        xb = F.relu(self.conv2(xb))
        xb = F.relu(self.conv3(xb))
        xb = F.avg_pool2d(xb, 4)
        return xb.view(-1, xb.size(1))

And the below are the output information when using torch.onnx.export to export the onnx model:

graph(%0 : Float(4, 1, 28, 28)
%1 : Float(16, 1, 3, 3)
%2 : Float(16)
%3 : Float(16, 16, 3, 3)
%4 : Float(16)
%5 : Float(10, 16, 3, 3)
%6 : Float(10)) {
%7 : Float(4, 16, 14, 14) = onnx::Conv[dilations=[1, 1], group=1, kernel_shape=[3, 3], pads=[1, 1, 1, 1], strides=[2, 2]](%0, %1, %2), scope: Mnist_CNN/Conv2d[conv1]
%8 : Float(4, 16, 14, 14) = onnx::Relu(%7), scope: Mnist_CNN
%9 : Float(4, 16, 7, 7) = onnx::Conv[dilations=[1, 1], group=1, kernel_shape=[3, 3], pads=[1, 1, 1, 1], strides=[2, 2]](%8, %3, %4), scope: Mnist_CNN/Conv2d[conv2]
%10 : Float(4, 16, 7, 7) = onnx::Relu(%9), scope: Mnist_CNN
%11 : Float(4, 10, 4, 4) = onnx::Conv[dilations=[1, 1], group=1, kernel_shape=[3, 3], pads=[1, 1, 1, 1], strides=[2, 2]](%10, %5, %6), scope: Mnist_CNN/Conv2d[conv3]
%12 : Float(4, 10, 4, 4) = onnx::Relu(%11), scope: Mnist_CNN
%13 : Tensor = onnx::Padmode=“constant”, pads=[0, 0, 0, 0, 0, 0, 0, 0], value=0, scope: Mnist_CNN
%14 : Float(4, 10, 1, 1) = onnx::AveragePoolkernel_shape=[4, 4], pads=[0, 0, 0, 0], strides=[4, 4], scope: Mnist_CNN
%15 : Long() = onnx::Constantvalue={1}, scope: Mnist_CNN
%16 : Tensor = onnx::Shape(%14), scope: Mnist_CNN
%17 : Long() = onnx::Gather[axis=0](%16, %15), scope: Mnist_CNN
%18 : Long() = onnx::Constantvalue={-1}, scope: Mnist_CNN
%19 : Tensor = onnx::Unsqueezeaxes=[0]
%20 : Tensor = onnx::Unsqueezeaxes=[0]
%21 : Tensor = onnx::Concat[axis=0](%19, %20)
%22 : Float(4, 10) = onnx::Reshape(%14, %21), scope: Mnist_CNN
return (%22);
}

Let look at %17~%18. Long means torch.int64. When I delete the view operation in the forward function and rerun the code, I get :

graph(%0 : Float(4, 1, 28, 28)
%1 : Float(16, 1, 3, 3)
%2 : Float(16)
%3 : Float(16, 16, 3, 3)
%4 : Float(16)
%5 : Float(10, 16, 3, 3)
%6 : Float(10)) {
%7 : Float(4, 16, 14, 14) = onnx::Conv[dilations=[1, 1], group=1, kernel_shape=[3, 3], pads=[1, 1, 1, 1], strides=[2, 2]](%0, %1, %2), scope: Mnist_CNN/Conv2d[conv1]
%8 : Float(4, 16, 14, 14) = onnx::Relu(%7), scope: Mnist_CNN
%9 : Float(4, 16, 7, 7) = onnx::Conv[dilations=[1, 1], group=1, kernel_shape=[3, 3], pads=[1, 1, 1, 1], strides=[2, 2]](%8, %3, %4), scope: Mnist_CNN/Conv2d[conv2]
%10 : Float(4, 16, 7, 7) = onnx::Relu(%9), scope: Mnist_CNN
%11 : Float(4, 10, 4, 4) = onnx::Conv[dilations=[1, 1], group=1, kernel_shape=[3, 3], pads=[1, 1, 1, 1], strides=[2, 2]](%10, %5, %6), scope: Mnist_CNN/Conv2d[conv3]
%12 : Float(4, 10, 4, 4) = onnx::Relu(%11), scope: Mnist_CNN
%13 : Tensor = onnx::Padmode=“constant”, pads=[0, 0, 0, 0, 0, 0, 0, 0], value=0, scope: Mnist_CNN
%14 : Float(4, 10, 1, 1) = onnx::AveragePoolkernel_shape=[4, 4], pads=[0, 0, 0, 0], strides=[4, 4], scope: Mnist_CNN
return (%14);
}

But, how can I define a practical model without view operation?
Please help. Any suggestion?

Same problem here! Is there any solution ?


Input filename: /home/xiaoju/cloth_2_trt/cloth_0320/resnet101/model.ckpt.onnx
ONNX IR version: 0.0.3
Opset version: 7
Producer name: pytorch
Producer version: 0.4
Domain:
Model version: 0
Doc string:

Parsing model
Unsupported ONNX data type: INT64 (7)
[2019-03-26 10:10:38 ERROR] Parameter check failed at: …/builder/Layers.cpp::ConstantLayer::1438, condition: weights.type == DataType::kFLOAT || weights.type == DataType::kHALF || weights.type == DataType::kINT32
axis:-1, nbDims:1While parsing node number 346 [Gather -> “973”]:
ERROR: /home/xiaoju/cloth_2_trt/onnx-tensorrt/onnx2trt_utils.hpp:302 In function convert_axis:
[8] Assertion failed: axis >= 0 && axis < nbDims

You solve it? I have the same problem;

Unsupported ONNX data type: INT64 (7)
ERROR: Parameter check failed at: …/builder/Layers.cpp::ConstantLayer::1585, condition: weights.type == DataType::kFLOAT || weights.type == DataType::kHALF || weights.type == DataType::kINT32
While parsing node number 100 [Gather]:
ERROR: onnx2trt_utils.hpp:275 In function convert_axis:
[8] Assertion failed: axis >= 0 && axis < nbDims
ERROR: failed to parse onnx file

Hi,

I’ve solved the problem by:

x = x.view(int(x.size(0)), -1)

convert the size to int type.

Then re-export the model to onnx.

But I think this may cause errors if size is out of int’s range.

Hi love.infly,I am using faceboxes of pytorch (https://github.com/XiaXuehai/faceboxes),I use your ways ,but it dont work.

%312 : Long() = onnx::Constant[value={0}](), scope: FaceBox/MultiBoxLayer[multilbox]
  %313 : Tensor = onnx::Shape(%311), scope: FaceBox/MultiBoxLayer[multilbox]
  %314 : Long() = onnx::Gather[axis=0](%313, %312), scope: FaceBox/MultiBoxLayer[multilbox]
  %315 : Float(1, 16, 16, 4) = onnx::Transpose[perm=[0, 2, 3, 1]](%311), scope: FaceBox/MultiBoxLayer[multilbox]
  %316 : Long() = onnx::Constant[value={-1}](), scope: FaceBox/MultiBoxLayer[multilbox]
  %317 : Long() = onnx::Constant[value={4}](), scope: FaceBox/MultiBoxLayer[multilbox]
  %318 : Tensor = onnx::Unsqueeze[axes=[0]](%314)
  %319 : Tensor = onnx::Unsqueeze[axes=[0]](%316)
  %320 : Tensor = onnx::Unsqueeze[axes=[0]](%317)
  %321 : Tensor = onnx::Concat[axis=0](%318, %319, %320)
  %322 : Float(1, 256, 4) = onnx::Reshape(%315, %321), scope: FaceBox/MultiBoxLayer[multilbox]
  %323 : Float(1, 4, 8, 8) = onnx::Conv[dilations=[1, 1], group=1, kernel_shape=[3, 3], pads=[1, 1, 1, 1], strides=[1, 1]](%298, %194, %195), scope: FaceBox/MultiBoxLayer[multilbox]/Conv2d
  %324 : Long() = onnx::Constant[value={0}](), scope: FaceBox/MultiBoxLayer[multilbox]
  %325 : Tensor = onnx::Shape(%323), scope: FaceBox/MultiBoxLayer[multilbox]
  %326 : Long() = onnx::Gather[axis=0](%325, %324), scope: FaceBox/MultiBoxLayer[multilbox]
  %327 : Float(1, 8, 8, 4) = onnx::Transpose[perm=[0, 2, 3, 1]](%323), scope: FaceBox/MultiBoxLayer[multilbox]
  %328 : Long() = onnx::Constant[value={-1}](), scope: FaceBox/MultiBoxLayer[multilbox]
  %329 : Long() = onnx::Constant[value={4}](), scope: FaceBox/MultiBoxLayer[multilbox]
  %330 : Tensor = onnx::Unsqueeze[axes=[0]](%326)
  %331 : Tensor = onnx::Unsqueeze[axes=[0]](%328)
  %332 : Tensor = onnx::Unsqueeze[axes=[0]](%329)
  %333 : Tensor = onnx::Concat[axis=0](%330, %331, %332)
  %334 : Float(1, 64, 4) = onnx::Reshape(%327, %333), scope: FaceBox/MultiBoxLayer[multilbox]
  %335 : Float(1, 21824, 4) = onnx::Concat[axis=1](%310, %322, %334), scope: FaceBox/MultiBoxLayer[multilbox]
  %output : Float(1, 87296) = onnx::Flatten[axis=1](%335), scope: FaceBox
  return (%output);
}

look at 328,Long() may be int(64),tensorrt-onnx dont support it.