Trtexec Internal Error (Symbolic relation a.z >= 0 is always false. )

Description

Unable to convert an ONNX model to a trt engine using trtexec command. Would love to give tensorrt a try but have been unable to use it for quite some time now. The model architecture is called “Segformer”, it’s a vision transformer for image segmentation.

&&&& RUNNING TensorRT.trtexec [TensorRT v8403] # /usr/src/tensorrt/bin/trtexec --workspace=49140 --onnx=sidewalk3.onnx --saveEngine=sidewalk3.trt
[09/13/2022-01:47:31] [W] --workspace flag has been deprecated by --memPoolSize flag.
[09/13/2022-01:47:31] [I] === Model Options ===
[09/13/2022-01:47:31] [I] Format: ONNX
[09/13/2022-01:47:31] [I] Model: sidewalk3.onnx
[09/13/2022-01:47:31] [I] Output:
[09/13/2022-01:47:31] [I] === Build Options ===
[09/13/2022-01:47:31] [I] Max batch: explicit batch
[09/13/2022-01:47:31] [I] Memory Pools: workspace: 49140 MiB, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default
[09/13/2022-01:47:31] [I] minTiming: 1
[09/13/2022-01:47:31] [I] avgTiming: 8
[09/13/2022-01:47:31] [I] Precision: FP32
[09/13/2022-01:47:31] [I] LayerPrecisions: 
[09/13/2022-01:47:31] [I] Calibration: 
[09/13/2022-01:47:31] [I] Refit: Disabled
[09/13/2022-01:47:31] [I] Sparsity: Disabled
[09/13/2022-01:47:31] [I] Safe mode: Disabled
[09/13/2022-01:47:31] [I] DirectIO mode: Disabled
[09/13/2022-01:47:31] [I] Restricted mode: Disabled
[09/13/2022-01:47:31] [I] Build only: Disabled
[09/13/2022-01:47:31] [I] Save engine: sidewalk3.trt
[09/13/2022-01:47:31] [I] Load engine: 
[09/13/2022-01:47:31] [I] Profiling verbosity: 0
[09/13/2022-01:47:31] [I] Tactic sources: Using default tactic sources
[09/13/2022-01:47:31] [I] timingCacheMode: local
[09/13/2022-01:47:31] [I] timingCacheFile: 
[09/13/2022-01:47:31] [I] Input(s)s format: fp32:CHW
[09/13/2022-01:47:31] [I] Output(s)s format: fp32:CHW
[09/13/2022-01:47:31] [I] Input build shapes: model
[09/13/2022-01:47:31] [I] Input calibration shapes: model
[09/13/2022-01:47:31] [I] === System Options ===
[09/13/2022-01:47:31] [I] Device: 0
[09/13/2022-01:47:31] [I] DLACore: 
[09/13/2022-01:47:31] [I] Plugins:
[09/13/2022-01:47:31] [I] === Inference Options ===
[09/13/2022-01:47:31] [I] Batch: Explicit
[09/13/2022-01:47:31] [I] Input inference shapes: model
[09/13/2022-01:47:31] [I] Iterations: 10
[09/13/2022-01:47:31] [I] Duration: 3s (+ 200ms warm up)
[09/13/2022-01:47:31] [I] Sleep time: 0ms
[09/13/2022-01:47:31] [I] Idle time: 0ms
[09/13/2022-01:47:31] [I] Streams: 1
[09/13/2022-01:47:31] [I] ExposeDMA: Disabled
[09/13/2022-01:47:31] [I] Data transfers: Enabled
[09/13/2022-01:47:31] [I] Spin-wait: Disabled
[09/13/2022-01:47:31] [I] Multithreading: Disabled
[09/13/2022-01:47:31] [I] CUDA Graph: Disabled
[09/13/2022-01:47:31] [I] Separate profiling: Disabled
[09/13/2022-01:47:31] [I] Time Deserialize: Disabled
[09/13/2022-01:47:31] [I] Time Refit: Disabled
[09/13/2022-01:47:31] [I] Inputs:
[09/13/2022-01:47:31] [I] === Reporting Options ===
[09/13/2022-01:47:31] [I] Verbose: Disabled
[09/13/2022-01:47:31] [I] Averages: 10 inferences
[09/13/2022-01:47:31] [I] Percentile: 99
[09/13/2022-01:47:31] [I] Dump refittable layers:Disabled
[09/13/2022-01:47:31] [I] Dump output: Disabled
[09/13/2022-01:47:31] [I] Profile: Disabled
[09/13/2022-01:47:31] [I] Export timing to JSON file: 
[09/13/2022-01:47:31] [I] Export output to JSON file: 
[09/13/2022-01:47:31] [I] Export profile to JSON file: 
[09/13/2022-01:47:31] [I] 
[09/13/2022-01:47:31] [I] === Device Information ===
[09/13/2022-01:47:31] [I] Selected Device: NVIDIA RTX A6000
[09/13/2022-01:47:31] [I] Compute Capability: 8.6
[09/13/2022-01:47:31] [I] SMs: 84
[09/13/2022-01:47:31] [I] Compute Clock Rate: 1.8 GHz
[09/13/2022-01:47:31] [I] Device Global Memory: 48685 MiB
[09/13/2022-01:47:31] [I] Shared Memory per SM: 100 KiB
[09/13/2022-01:47:31] [I] Memory Bus Width: 384 bits (ECC disabled)
[09/13/2022-01:47:31] [I] Memory Clock Rate: 8.001 GHz
[09/13/2022-01:47:31] [I] 
[09/13/2022-01:47:31] [I] TensorRT version: 8.4.3
[09/13/2022-01:47:31] [I] [TRT] [MemUsageChange] Init CUDA: CPU +328, GPU +0, now: CPU 336, GPU 446 (MiB)
[09/13/2022-01:47:32] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +327, GPU +104, now: CPU 682, GPU 550 (MiB)
[09/13/2022-01:47:32] [I] Start parsing network model
[09/13/2022-01:47:35] [I] [TRT] ----------------------------------------------------------------
[09/13/2022-01:47:35] [I] [TRT] Input filename:   sidewalk3.onnx
[09/13/2022-01:47:35] [I] [TRT] ONNX IR version:  0.0.8
[09/13/2022-01:47:35] [I] [TRT] Opset version:    16
[09/13/2022-01:47:35] [I] [TRT] Producer name:    pytorch
[09/13/2022-01:47:35] [I] [TRT] Producer version: 1.12.1
[09/13/2022-01:47:35] [I] [TRT] Domain:           
[09/13/2022-01:47:35] [I] [TRT] Model version:    0
[09/13/2022-01:47:35] [I] [TRT] Doc string:       
[09/13/2022-01:47:35] [I] [TRT] ----------------------------------------------------------------
[09/13/2022-01:47:35] [W] [TRT] onnx2trt_utils.cpp:369: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[09/13/2022-01:47:48] [I] Finish parsing network model
[09/13/2022-01:48:32] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +839, GPU +360, now: CPU 1927, GPU 910 (MiB)
[09/13/2022-01:48:32] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +129, GPU +60, now: CPU 2056, GPU 970 (MiB)
[09/13/2022-01:48:32] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored.
[09/13/2022-01:51:40] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:51:40] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:51:40] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:51:40] [W] [TRT] Skipping tactic 0 due to insufficient memory on requested size of 68719476736 detected for tactic 0x0000000000000018.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:51:40] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:51:40] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:51:40] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:51:40] [W] [TRT] Skipping tactic 1 due to insufficient memory on requested size of 68719476736 detected for tactic 0x0000000000000019.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:51:40] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:51:40] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:51:40] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:51:40] [W] [TRT] Skipping tactic 2 due to insufficient memory on requested size of 68719476736 detected for tactic 0x000000000000001a.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:51:41] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:51:41] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:51:41] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:51:41] [W] [TRT] Skipping tactic 3 due to insufficient memory on requested size of 68719476736 detected for tactic 0x000000000000001b.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:51:41] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:51:41] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:51:41] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:51:41] [W] [TRT] Skipping tactic 4 due to insufficient memory on requested size of 68719476736 detected for tactic 0x000000000000001f.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:53:09] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:09] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:09] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:53:09] [W] [TRT] Skipping tactic 0 due to insufficient memory on requested size of 68719476736 detected for tactic 0x0000000000000018.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:53:09] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:09] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:09] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:53:09] [W] [TRT] Skipping tactic 1 due to insufficient memory on requested size of 68719476736 detected for tactic 0x0000000000000019.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:53:09] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:09] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:09] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:53:09] [W] [TRT] Skipping tactic 2 due to insufficient memory on requested size of 68719476736 detected for tactic 0x000000000000001a.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:53:09] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:09] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:09] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:53:09] [W] [TRT] Skipping tactic 3 due to insufficient memory on requested size of 68719476736 detected for tactic 0x000000000000001b.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:53:09] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:09] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:09] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:53:09] [W] [TRT] Skipping tactic 4 due to insufficient memory on requested size of 68719476736 detected for tactic 0x000000000000001f.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:53:10] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:10] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:10] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:53:10] [W] [TRT] Skipping tactic 0 due to insufficient memory on requested size of 68719476736 detected for tactic 0x0000000000000018.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:53:10] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:10] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:10] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:53:10] [W] [TRT] Skipping tactic 1 due to insufficient memory on requested size of 68719476736 detected for tactic 0x0000000000000019.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:53:10] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:10] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:10] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:53:10] [W] [TRT] Skipping tactic 2 due to insufficient memory on requested size of 68719476736 detected for tactic 0x000000000000001a.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:53:10] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:10] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:10] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:53:10] [W] [TRT] Skipping tactic 3 due to insufficient memory on requested size of 68719476736 detected for tactic 0x000000000000001b.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:53:10] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:10] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:10] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:53:10] [W] [TRT] Skipping tactic 4 due to insufficient memory on requested size of 68719476736 detected for tactic 0x000000000000001f.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:55:29] [E] Error[2]: [elementwiseBuilder.cpp::splitVectorizedHub::188] Error Code 2: Internal Error (Symbolic relation a.z >= 0 is always false. )
[09/13/2022-01:55:29] [E] Error[2]: [builder.cpp::buildSerializedNetwork::636] Error Code 2: Internal Error (Assertion engine != nullptr failed. )
[09/13/2022-01:55:29] [E] Engine could not be created from network
[09/13/2022-01:55:29] [E] Building engine failed
[09/13/2022-01:55:29] [E] Failed to create engine from model or file.
[09/13/2022-01:55:29] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8403] # /usr/src/tensorrt/bin/trtexec --workspace=49140 --onnx=sidewalk3.onnx --saveEngine=sidewalk3.trt

Environment

TensorRT Version: 8.4.3
GPU Type: NVIDIA RTX A6000
Nvidia Driver Version: 510.47.03
CUDA Version: 11.6
CUDNN Version: 8.5.0
Operating System + Version: Ubuntu 20.04
Python Version (if applicable): 3.8.10
TensorFlow Version (if applicable): N/A
PyTorch Version (if applicable): 1.12.1
Baremetal or Container (if container which image + tag): N/A

Relevant Files

sidewalk3.onnx sidewalk3.onnx - Google Drive
sidewalk4.onnx sidewalk4.onnx - Google Drive

Steps To Reproduce

Both of the following errors produces the same error:
trtexec --workspace=49140 --onnx=sidewalk3.onnx --saveEngine=sidewalk3.trt
trtexec --workspace=49140 --onnx=sidewalk4.onnx --saveEngine=sidewalk4.trt

Please include:

  • Exact steps/commands to build your repro
  • Exact steps/commands to run your repro
  • Full traceback of errors encountered

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.

In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

Hello! When using the onnx checker nothing comes out so I assume that’s good but yes there is an error when attempting to use trtexec to convert the model to trt format. I do see the memory errors but the model is able to do inference on a Jetson Xavier NX so I kind of doubt converting it for TensorRT use would require something bigger than a 48gb A6000. My goal is to run an optimized or quantized version of the model for inference on a Jetson Xavier NX. Thank you.

Hi @wesley3,

We are working on the fix for this issue.
This issue will be fixed in the 8.5.1 GA release.

Thank you.