Trtexec Internal Error (Symbolic relation a.z >= 0 is always false. )

wesley3 · September 13, 2022, 2:10am

Description

Unable to convert an ONNX model to a trt engine using trtexec command. Would love to give tensorrt a try but have been unable to use it for quite some time now. The model architecture is called “Segformer”, it’s a vision transformer for image segmentation.

&&&& RUNNING TensorRT.trtexec [TensorRT v8403] # /usr/src/tensorrt/bin/trtexec --workspace=49140 --onnx=sidewalk3.onnx --saveEngine=sidewalk3.trt
[09/13/2022-01:47:31] [W] --workspace flag has been deprecated by --memPoolSize flag.
[09/13/2022-01:47:31] [I] === Model Options ===
[09/13/2022-01:47:31] [I] Format: ONNX
[09/13/2022-01:47:31] [I] Model: sidewalk3.onnx
[09/13/2022-01:47:31] [I] Output:
[09/13/2022-01:47:31] [I] === Build Options ===
[09/13/2022-01:47:31] [I] Max batch: explicit batch
[09/13/2022-01:47:31] [I] Memory Pools: workspace: 49140 MiB, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default
[09/13/2022-01:47:31] [I] minTiming: 1
[09/13/2022-01:47:31] [I] avgTiming: 8
[09/13/2022-01:47:31] [I] Precision: FP32
[09/13/2022-01:47:31] [I] LayerPrecisions: 
[09/13/2022-01:47:31] [I] Calibration: 
[09/13/2022-01:47:31] [I] Refit: Disabled
[09/13/2022-01:47:31] [I] Sparsity: Disabled
[09/13/2022-01:47:31] [I] Safe mode: Disabled
[09/13/2022-01:47:31] [I] DirectIO mode: Disabled
[09/13/2022-01:47:31] [I] Restricted mode: Disabled
[09/13/2022-01:47:31] [I] Build only: Disabled
[09/13/2022-01:47:31] [I] Save engine: sidewalk3.trt
[09/13/2022-01:47:31] [I] Load engine: 
[09/13/2022-01:47:31] [I] Profiling verbosity: 0
[09/13/2022-01:47:31] [I] Tactic sources: Using default tactic sources
[09/13/2022-01:47:31] [I] timingCacheMode: local
[09/13/2022-01:47:31] [I] timingCacheFile: 
[09/13/2022-01:47:31] [I] Input(s)s format: fp32:CHW
[09/13/2022-01:47:31] [I] Output(s)s format: fp32:CHW
[09/13/2022-01:47:31] [I] Input build shapes: model
[09/13/2022-01:47:31] [I] Input calibration shapes: model
[09/13/2022-01:47:31] [I] === System Options ===
[09/13/2022-01:47:31] [I] Device: 0
[09/13/2022-01:47:31] [I] DLACore: 
[09/13/2022-01:47:31] [I] Plugins:
[09/13/2022-01:47:31] [I] === Inference Options ===
[09/13/2022-01:47:31] [I] Batch: Explicit
[09/13/2022-01:47:31] [I] Input inference shapes: model
[09/13/2022-01:47:31] [I] Iterations: 10
[09/13/2022-01:47:31] [I] Duration: 3s (+ 200ms warm up)
[09/13/2022-01:47:31] [I] Sleep time: 0ms
[09/13/2022-01:47:31] [I] Idle time: 0ms
[09/13/2022-01:47:31] [I] Streams: 1
[09/13/2022-01:47:31] [I] ExposeDMA: Disabled
[09/13/2022-01:47:31] [I] Data transfers: Enabled
[09/13/2022-01:47:31] [I] Spin-wait: Disabled
[09/13/2022-01:47:31] [I] Multithreading: Disabled
[09/13/2022-01:47:31] [I] CUDA Graph: Disabled
[09/13/2022-01:47:31] [I] Separate profiling: Disabled
[09/13/2022-01:47:31] [I] Time Deserialize: Disabled
[09/13/2022-01:47:31] [I] Time Refit: Disabled
[09/13/2022-01:47:31] [I] Inputs:
[09/13/2022-01:47:31] [I] === Reporting Options ===
[09/13/2022-01:47:31] [I] Verbose: Disabled
[09/13/2022-01:47:31] [I] Averages: 10 inferences
[09/13/2022-01:47:31] [I] Percentile: 99
[09/13/2022-01:47:31] [I] Dump refittable layers:Disabled
[09/13/2022-01:47:31] [I] Dump output: Disabled
[09/13/2022-01:47:31] [I] Profile: Disabled
[09/13/2022-01:47:31] [I] Export timing to JSON file: 
[09/13/2022-01:47:31] [I] Export output to JSON file: 
[09/13/2022-01:47:31] [I] Export profile to JSON file: 
[09/13/2022-01:47:31] [I] 
[09/13/2022-01:47:31] [I] === Device Information ===
[09/13/2022-01:47:31] [I] Selected Device: NVIDIA RTX A6000
[09/13/2022-01:47:31] [I] Compute Capability: 8.6
[09/13/2022-01:47:31] [I] SMs: 84
[09/13/2022-01:47:31] [I] Compute Clock Rate: 1.8 GHz
[09/13/2022-01:47:31] [I] Device Global Memory: 48685 MiB
[09/13/2022-01:47:31] [I] Shared Memory per SM: 100 KiB
[09/13/2022-01:47:31] [I] Memory Bus Width: 384 bits (ECC disabled)
[09/13/2022-01:47:31] [I] Memory Clock Rate: 8.001 GHz
[09/13/2022-01:47:31] [I] 
[09/13/2022-01:47:31] [I] TensorRT version: 8.4.3
[09/13/2022-01:47:31] [I] [TRT] [MemUsageChange] Init CUDA: CPU +328, GPU +0, now: CPU 336, GPU 446 (MiB)
[09/13/2022-01:47:32] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +327, GPU +104, now: CPU 682, GPU 550 (MiB)
[09/13/2022-01:47:32] [I] Start parsing network model
[09/13/2022-01:47:35] [I] [TRT] ----------------------------------------------------------------
[09/13/2022-01:47:35] [I] [TRT] Input filename:   sidewalk3.onnx
[09/13/2022-01:47:35] [I] [TRT] ONNX IR version:  0.0.8
[09/13/2022-01:47:35] [I] [TRT] Opset version:    16
[09/13/2022-01:47:35] [I] [TRT] Producer name:    pytorch
[09/13/2022-01:47:35] [I] [TRT] Producer version: 1.12.1
[09/13/2022-01:47:35] [I] [TRT] Domain:           
[09/13/2022-01:47:35] [I] [TRT] Model version:    0
[09/13/2022-01:47:35] [I] [TRT] Doc string:       
[09/13/2022-01:47:35] [I] [TRT] ----------------------------------------------------------------
[09/13/2022-01:47:35] [W] [TRT] onnx2trt_utils.cpp:369: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[09/13/2022-01:47:48] [I] Finish parsing network model
[09/13/2022-01:48:32] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +839, GPU +360, now: CPU 1927, GPU 910 (MiB)
[09/13/2022-01:48:32] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +129, GPU +60, now: CPU 2056, GPU 970 (MiB)
[09/13/2022-01:48:32] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored.
[09/13/2022-01:51:40] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:51:40] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:51:40] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:51:40] [W] [TRT] Skipping tactic 0 due to insufficient memory on requested size of 68719476736 detected for tactic 0x0000000000000018.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:51:40] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:51:40] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:51:40] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:51:40] [W] [TRT] Skipping tactic 1 due to insufficient memory on requested size of 68719476736 detected for tactic 0x0000000000000019.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:51:40] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:51:40] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:51:40] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:51:40] [W] [TRT] Skipping tactic 2 due to insufficient memory on requested size of 68719476736 detected for tactic 0x000000000000001a.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:51:41] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:51:41] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:51:41] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:51:41] [W] [TRT] Skipping tactic 3 due to insufficient memory on requested size of 68719476736 detected for tactic 0x000000000000001b.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:51:41] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:51:41] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:51:41] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:51:41] [W] [TRT] Skipping tactic 4 due to insufficient memory on requested size of 68719476736 detected for tactic 0x000000000000001f.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:53:09] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:09] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:09] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:53:09] [W] [TRT] Skipping tactic 0 due to insufficient memory on requested size of 68719476736 detected for tactic 0x0000000000000018.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:53:09] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:09] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:09] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:53:09] [W] [TRT] Skipping tactic 1 due to insufficient memory on requested size of 68719476736 detected for tactic 0x0000000000000019.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:53:09] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:09] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:09] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:53:09] [W] [TRT] Skipping tactic 2 due to insufficient memory on requested size of 68719476736 detected for tactic 0x000000000000001a.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:53:09] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:09] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:09] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:53:09] [W] [TRT] Skipping tactic 3 due to insufficient memory on requested size of 68719476736 detected for tactic 0x000000000000001b.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:53:09] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:09] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:09] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:53:09] [W] [TRT] Skipping tactic 4 due to insufficient memory on requested size of 68719476736 detected for tactic 0x000000000000001f.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:53:10] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:10] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:10] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:53:10] [W] [TRT] Skipping tactic 0 due to insufficient memory on requested size of 68719476736 detected for tactic 0x0000000000000018.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:53:10] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:10] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:10] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:53:10] [W] [TRT] Skipping tactic 1 due to insufficient memory on requested size of 68719476736 detected for tactic 0x0000000000000019.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:53:10] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:10] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:10] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:53:10] [W] [TRT] Skipping tactic 2 due to insufficient memory on requested size of 68719476736 detected for tactic 0x000000000000001a.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:53:10] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:10] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:10] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:53:10] [W] [TRT] Skipping tactic 3 due to insufficient memory on requested size of 68719476736 detected for tactic 0x000000000000001b.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:53:10] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:10] [E] Error[2]: [virtualMemoryBuffer.cpp::resizePhysical::144] Error Code 2: OutOfMemory (no further information)
[09/13/2022-01:53:10] [W] [TRT] Requested amount of GPU memory (68719476736 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[09/13/2022-01:53:10] [W] [TRT] Skipping tactic 4 due to insufficient memory on requested size of 68719476736 detected for tactic 0x000000000000001f.
Try decreasing the workspace size with IBuilderConfig::setMemoryPoolLimit().
[09/13/2022-01:55:29] [E] Error[2]: [elementwiseBuilder.cpp::splitVectorizedHub::188] Error Code 2: Internal Error (Symbolic relation a.z >= 0 is always false. )
[09/13/2022-01:55:29] [E] Error[2]: [builder.cpp::buildSerializedNetwork::636] Error Code 2: Internal Error (Assertion engine != nullptr failed. )
[09/13/2022-01:55:29] [E] Engine could not be created from network
[09/13/2022-01:55:29] [E] Building engine failed
[09/13/2022-01:55:29] [E] Failed to create engine from model or file.
[09/13/2022-01:55:29] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8403] # /usr/src/tensorrt/bin/trtexec --workspace=49140 --onnx=sidewalk3.onnx --saveEngine=sidewalk3.trt

Environment

TensorRT Version: 8.4.3
GPU Type: NVIDIA RTX A6000
Nvidia Driver Version: 510.47.03
CUDA Version: 11.6
CUDNN Version: 8.5.0
Operating System + Version: Ubuntu 20.04
Python Version (if applicable): 3.8.10
TensorFlow Version (if applicable): N/A
PyTorch Version (if applicable): 1.12.1
Baremetal or Container (if container which image + tag): N/A

Relevant Files

sidewalk3.onnx sidewalk3.onnx - Google Drive
sidewalk4.onnx sidewalk4.onnx - Google Drive

Steps To Reproduce

Both of the following errors produces the same error:
trtexec --workspace=49140 --onnx=sidewalk3.onnx --saveEngine=sidewalk3.trt
trtexec --workspace=49140 --onnx=sidewalk4.onnx --saveEngine=sidewalk4.trt

Please include:

Exact steps/commands to build your repro
Exact steps/commands to run your repro
Full traceback of errors encountered

NVES · September 13, 2022, 2:37am

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.

In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

wesley3 · September 13, 2022, 3:10am

Hello! When using the onnx checker nothing comes out so I assume that’s good but yes there is an error when attempting to use trtexec to convert the model to trt format. I do see the memory errors but the model is able to do inference on a Jetson Xavier NX so I kind of doubt converting it for TensorRT use would require something bigger than a 48gb A6000. My goal is to run an optimized or quantized version of the model for inference on a Jetson Xavier NX. Thank you.

spolisetty · September 29, 2022, 12:42pm

Hi @wesley3,

We are working on the fix for this issue.
This issue will be fixed in the 8.5.1 GA release.

Thank you.

wesley3 · October 4, 2022, 7:46pm

Sounds amazing! I tried to find a timeline for that online and was unable to. Is there a release date planned yet, or at least a vague idea on whether this is days, weeks, or months away? If not I totally understand. Thank you for your help!

wesley3 · December 3, 2022, 9:22pm

Hello @spolisetty I saw that TensorRT 8.5.1 was released. Do I need to wait until it is added to JetPack or would I be able to convert the model then run in on my Jetson with 8.4 TensorRT?

Thank you!

spolisetty · December 5, 2022, 11:17am

Hi @wesley3,

Could you please create a new post on the Jetson forum with reference to this post to get better help.