Trtexec "Failed to sort the model topologically."

Description

I’m trying to create an engine from a onnx model using trtexec. The model is the ViT-H/14, which has been fine tuned to the CIFAR-10 dataset using the following: https://github.com/bwconrad/vit-finetune

I transform the checkpoint to onnx with the following:

from src.model import ClassificationModel
import torch
import torchvision
import torchvision.transforms as transforms
from torch.utils.data import DataLoader
import os

home_dir = os.path.expanduser("~")

ckpt_path = "output/default/version_0/checkpoints/best-step-step=97000-val_acc=0.9847.ckpt"
model = ClassificationModel.load_from_checkpoint(ckpt_path)
model = model.to("cpu")
model.eval()
dummy_input = torch.randn(1, 3, 224, 224, device="cpu")

onnx_model_path = os.path.join(home_dir, "model/big/vith14.onnx")
torch.onnx.export(model.net,
                dummy_input,
                onnx_model_path,
                export_params=True,
                opset_version=17,
                do_constant_folding=True
                input_names=['input'],
                output_names=['output'],
)

Given that the model is fairly big, 2.4G, it creates a huge list of files, like onnx__MatMul_3541, vit.embeddings.cls_token vit.encoder.layer.20.layernorm_after.weight, etc. I am aware that trtexec can only be used with models up to 2GB, so I’ve tried to use fp16 precision to compress it’s size. For example, using the command: python3 -m onnxruntime.transformers.optimizer --input ${HOME}/model/big/vith14.onnx --output ${HOME}/model/opt/vith14.onnx --hidden_size 1280 --num_heads 16 --opt_level 0 --float16 --use_gpu gives the result:

apply: Fused Gelu: 32
               apply: Fused SkipLayerNormalization: 64
               apply: Fused Attention: 31
         prune_graph: Removed 155 nodes
               apply: Fused BiasGelu: 32
               apply: Fused SkipLayerNormalization(add bias): 64
            optimize: opset version: 17
get_operator_statistics: Operators:[('MatMul', 101), ('SkipLayerNormalization', 64), ('BiasGelu', 32), ('Attention', 31), ('Constant', 13), ('Reshape', 5), ('Transpose', 5), ('Add', 4), ('Cast', 2), ('Concat', 2), ('ConstantOfShape', 1), ('Conv', 1), ('Div', 1), ('Equal', 1), ('Expand', 1), ('Gather', 1), ('Gemm', 1), ('LayerNormalization', 1), ('Mul', 1), ('Shape', 1), ('Slice', 1), ('Softmax', 1), ('Where', 1)]
get_fused_operator_statistics: Optimized operators: {'EmbedLayerNormalization': 0, 'Attention': 31, 'MultiHeadAttention': 0, 'Gelu': 0, 'FastGelu': 0, 'BiasGelu': 32, 'GemmFastGelu': 0, 'LayerNormalization': 1, 'SimplifiedLayerNormalization': 0, 'SkipLayerNormalization': 64, 'SkipSimplifiedLayerNormalization': 0, 'RotaryEmbedding': 0, 'QOrderedAttention': 0, 'QOrderedGelu': 0, 'QOrderedLayerNormalization': 0, 'QOrderedMatMul': 0}
                main: The model has been optimized.
  save_model_to_file: Sort graphs in topological order
  save_model_to_file: Model saved to /home/jose/model/opt/vith14.onnx

I have chosen no graph optimization options with --opt_level 0 because I saw somewhere in a discussion someone saying that one should not attempt to optimize the model with onnxruntime if it was to be used with trtexec, which would apply it’s own optimizations. After this, the output model weights 1.26GB. If I make a docker container with docker run --gpus all --rm -ti nvcr.io/nvidia/tensorrt:24.01-py3 and run trtexec with that optimized model, I get:

[02/19/2024-13:47:35] [I] === Model Options ===
[02/19/2024-13:47:35] [I] Format: ONNX
[02/19/2024-13:47:35] [I] Model: /workspace/model/vith14.onnx
[02/19/2024-13:47:35] [I] Output:
[02/19/2024-13:47:35] [I] === Build Options ===
[02/19/2024-13:47:35] [I] Max batch: explicit batch
[02/19/2024-13:47:35] [I] Memory Pools: workspace: default, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default
[02/19/2024-13:47:35] [I] minTiming: 1
[02/19/2024-13:47:35] [I] avgTiming: 8
[02/19/2024-13:47:35] [I] Precision: FP32
[02/19/2024-13:47:35] [I] LayerPrecisions: 
[02/19/2024-13:47:35] [I] Layer Device Types: 
[02/19/2024-13:47:35] [I] Calibration: 
[02/19/2024-13:47:35] [I] Refit: Disabled
[02/19/2024-13:47:35] [I] Version Compatible: Disabled
[02/19/2024-13:47:35] [I] TensorRT runtime: full
[02/19/2024-13:47:35] [I] Lean DLL Path: 
[02/19/2024-13:47:35] [I] Tempfile Controls: { in_memory: allow, temporary: allow }
[02/19/2024-13:47:35] [I] Exclude Lean Runtime: Disabled
[02/19/2024-13:47:35] [I] Sparsity: Disabled
[02/19/2024-13:47:35] [I] Safe mode: Disabled
[02/19/2024-13:47:35] [I] Build DLA standalone loadable: Disabled
[02/19/2024-13:47:35] [I] Allow GPU fallback for DLA: Disabled
[02/19/2024-13:47:35] [I] DirectIO mode: Disabled
[02/19/2024-13:47:35] [I] Restricted mode: Disabled
[02/19/2024-13:47:35] [I] Skip inference: Disabled
[02/19/2024-13:47:35] [I] Save engine: 
[02/19/2024-13:47:35] [I] Load engine: 
[02/19/2024-13:47:35] [I] Profiling verbosity: 0
[02/19/2024-13:47:35] [I] Tactic sources: Using default tactic sources
[02/19/2024-13:47:35] [I] timingCacheMode: local
[02/19/2024-13:47:35] [I] timingCacheFile: 
[02/19/2024-13:47:35] [I] Heuristic: Disabled
[02/19/2024-13:47:35] [I] Preview Features: Use default preview flags.
[02/19/2024-13:47:35] [I] MaxAuxStreams: -1
[02/19/2024-13:47:35] [I] BuilderOptimizationLevel: -1
[02/19/2024-13:47:35] [I] Input(s)s format: fp32:CHW
[02/19/2024-13:47:35] [I] Output(s)s format: fp32:CHW
[02/19/2024-13:47:35] [I] Input build shapes: model
[02/19/2024-13:47:35] [I] Input calibration shapes: model
[02/19/2024-13:47:35] [I] === System Options ===
[02/19/2024-13:47:35] [I] Device: 0
[02/19/2024-13:47:35] [I] DLACore: 
[02/19/2024-13:47:35] [I] Plugins:
[02/19/2024-13:47:35] [I] setPluginsToSerialize:
[02/19/2024-13:47:35] [I] dynamicPlugins:
[02/19/2024-13:47:35] [I] ignoreParsedPluginLibs: 0
[02/19/2024-13:47:35] [I] 
[02/19/2024-13:47:35] [I] === Inference Options ===
[02/19/2024-13:47:35] [I] Batch: Explicit
[02/19/2024-13:47:35] [I] Input inference shapes: model
[02/19/2024-13:47:35] [I] Iterations: 10
[02/19/2024-13:47:35] [I] Duration: 3s (+ 200ms warm up)
[02/19/2024-13:47:35] [I] Sleep time: 0ms
[02/19/2024-13:47:35] [I] Idle time: 0ms
[02/19/2024-13:47:35] [I] Inference Streams: 1
[02/19/2024-13:47:35] [I] ExposeDMA: Disabled
[02/19/2024-13:47:35] [I] Data transfers: Enabled
[02/19/2024-13:47:35] [I] Spin-wait: Disabled
[02/19/2024-13:47:35] [I] Multithreading: Disabled
[02/19/2024-13:47:35] [I] CUDA Graph: Disabled
[02/19/2024-13:47:35] [I] Separate profiling: Disabled
[02/19/2024-13:47:35] [I] Time Deserialize: Disabled
[02/19/2024-13:47:35] [I] Time Refit: Disabled
[02/19/2024-13:47:35] [I] NVTX verbosity: 0
[02/19/2024-13:47:35] [I] Persistent Cache Ratio: 0
[02/19/2024-13:47:35] [I] Inputs:
[02/19/2024-13:47:35] [I] === Reporting Options ===
[02/19/2024-13:47:35] [I] Verbose: Enabled
[02/19/2024-13:47:35] [I] Averages: 10 inferences
[02/19/2024-13:47:35] [I] Percentiles: 90,95,99
[02/19/2024-13:47:35] [I] Dump refittable layers:Disabled
[02/19/2024-13:47:35] [I] Dump output: Disabled
[02/19/2024-13:47:35] [I] Profile: Disabled
[02/19/2024-13:47:35] [I] Export timing to JSON file: 
[02/19/2024-13:47:35] [I] Export output to JSON file: 
[02/19/2024-13:47:35] [I] Export profile to JSON file: 
[02/19/2024-13:47:35] [I] 
[02/19/2024-13:47:35] [I] === Device Information ===
[02/19/2024-13:47:35] [I] Selected Device: Tesla T4
[02/19/2024-13:47:35] [I] Compute Capability: 7.5
[02/19/2024-13:47:35] [I] SMs: 40
[02/19/2024-13:47:35] [I] Device Global Memory: 14929 MiB
[02/19/2024-13:47:35] [I] Shared Memory per SM: 64 KiB
[02/19/2024-13:47:35] [I] Memory Bus Width: 256 bits (ECC enabled)
[02/19/2024-13:47:35] [I] Application Compute Clock Rate: 1.59 GHz
[02/19/2024-13:47:35] [I] Application Memory Clock Rate: 5.001 GHz
[02/19/2024-13:47:35] [I] 
[02/19/2024-13:47:35] [I] Note: The application clock rates do not reflect the actual clock rates that the GPU is currently running at.
[02/19/2024-13:47:35] [I] 
[02/19/2024-13:47:35] [I] TensorRT version: 8.6.1
[02/19/2024-13:47:35] [I] Loading standard plugins
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::BatchedNMS_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::BatchTilePlugin_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::Clip_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::CoordConvAC version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::CropAndResizeDynamic version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::CropAndResize version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::DecodeBbox3DPlugin version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::DetectionLayer_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::EfficientNMS_Explicit_TF_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::EfficientNMS_Implicit_TF_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::EfficientNMS_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::FlattenConcat_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::GenerateDetection_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::GridAnchor_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::GridAnchorRect_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 2
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::LReLU_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::ModulatedDeformConv2d version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::MultilevelCropAndResize_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::MultilevelProposeROI_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::MultiscaleDeformableAttnPlugin_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::NMSDynamic_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::NMS_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::Normalize_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::PillarScatterPlugin version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::PriorBox_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::ProposalDynamic version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::ProposalLayer_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::Proposal version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::PyramidROIAlign_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::Region_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::Reorg_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::ResizeNearest_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::ROIAlign_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::RPROI_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::ScatterND version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::SpecialSlice_TRT version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::Split version 1
[02/19/2024-13:47:35] [V] [TRT] Registered plugin creator - ::VoxelGeneratorPlugin version 1
[02/19/2024-13:47:35] [I] [TRT] [MemUsageChange] Init CUDA: CPU +2, GPU +0, now: CPU 22, GPU 109 (MiB)
[02/19/2024-13:47:36] [V] [TRT] Trying to load shared library libnvinfer_builder_resource.so.8.6.1
[02/19/2024-13:47:36] [V] [TRT] Loaded shared library libnvinfer_builder_resource.so.8.6.1
[02/19/2024-13:47:54] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +889, GPU +174, now: CPU 987, GPU 283 (MiB)
[02/19/2024-13:47:54] [V] [TRT] CUDA lazy loading is enabled.
[02/19/2024-13:47:54] [I] Start parsing network model.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:604] Reading dangerously large protocol message.  If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:81] The total number of bytes read was 1261667396
[02/19/2024-13:47:56] [I] [TRT] ----------------------------------------------------------------
[02/19/2024-13:47:56] [I] [TRT] Input filename:   /workspace/model/vith14.onnx
[02/19/2024-13:47:56] [I] [TRT] ONNX IR version:  0.0.8
[02/19/2024-13:47:56] [I] [TRT] Opset version:    17
[02/19/2024-13:47:56] [I] [TRT] Producer name:    onnxruntime.transformers
[02/19/2024-13:47:56] [I] [TRT] Producer version: 1.17.0
[02/19/2024-13:47:56] [I] [TRT] Domain:           
[02/19/2024-13:47:56] [I] [TRT] Model version:    0
[02/19/2024-13:47:56] [I] [TRT] Doc string:       
[02/19/2024-13:47:56] [I] [TRT] ----------------------------------------------------------------
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:604] Reading dangerously large protocol message.  If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:81] The total number of bytes read was 1261667396
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::BatchedNMSDynamic_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::BatchedNMS_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::BatchTilePlugin_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::Clip_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::CoordConvAC version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::CropAndResizeDynamic version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::CropAndResize version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::DecodeBbox3DPlugin version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::DetectionLayer_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::EfficientNMS_Explicit_TF_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::EfficientNMS_Implicit_TF_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::EfficientNMS_ONNX_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::EfficientNMS_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::FlattenConcat_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::GenerateDetection_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::GridAnchor_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::GridAnchorRect_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::InstanceNormalization_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::InstanceNormalization_TRT version 2
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::LReLU_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::ModulatedDeformConv2d version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::MultilevelCropAndResize_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::MultilevelProposeROI_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::MultiscaleDeformableAttnPlugin_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::NMSDynamic_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::NMS_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::Normalize_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::PillarScatterPlugin version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::PriorBox_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::ProposalDynamic version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::ProposalLayer_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::Proposal version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::PyramidROIAlign_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::Region_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::Reorg_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::ResizeNearest_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::ROIAlign_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::RPROI_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::ScatterND version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::SpecialSlice_TRT version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::Split version 1
[02/19/2024-13:48:03] [V] [TRT] Plugin creator already registered - ::VoxelGeneratorPlugin version 1
[02/19/2024-13:48:03] [V] [TRT] Adding network input: input with dtype: float32, dimensions: (1, 3, 224, 224)
[02/19/2024-13:48:03] [V] [TRT] Registering tensor: input for ONNX tensor: input
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.embeddings.cls_token
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.embeddings.position_embeddings
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.embeddings.patch_embeddings.projection.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.embeddings.patch_embeddings.projection.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.0.attention.attention.query.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.0.attention.attention.key.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.0.attention.attention.value.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.0.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.0.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.0.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.0.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.0.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.0.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.0.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.1.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.1.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.1.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.1.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.1.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.1.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.1.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.2.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.2.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.2.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.2.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.2.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.2.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.2.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.3.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.3.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.3.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.3.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.3.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.3.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.3.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.4.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.4.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.4.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.4.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.4.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.4.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.4.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.5.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.5.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.5.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.5.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.5.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.5.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.5.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.6.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.6.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.6.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.6.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.6.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.6.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.6.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.7.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.7.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.7.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.7.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.7.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.7.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.7.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.8.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.8.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.8.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.8.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.8.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.8.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.8.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.9.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.9.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.9.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.9.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.9.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.9.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.9.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.10.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.10.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.10.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.10.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.10.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.10.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.10.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.11.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.11.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.11.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.11.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.11.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.11.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.11.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.12.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.12.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.12.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.12.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.12.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.12.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.12.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.13.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.13.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.13.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.13.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.13.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.13.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.13.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.14.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.14.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.14.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.14.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.14.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.14.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.14.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.15.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.15.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.15.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.15.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.15.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.15.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.15.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.16.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.16.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.16.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.16.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.16.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.16.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.16.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.17.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.17.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.17.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.17.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.17.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.17.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.17.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.18.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.18.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.18.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.18.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.18.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.18.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.18.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.19.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.19.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.19.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.19.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.19.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.19.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.19.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.20.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.20.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.20.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.20.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.20.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.20.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.20.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.21.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.21.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.21.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.21.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.21.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.21.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.21.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.22.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.22.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.22.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.22.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.22.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.22.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.22.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.23.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.23.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.23.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.23.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.23.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.23.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.23.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.24.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.24.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.24.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.24.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.24.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.24.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.24.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.25.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.25.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.25.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.25.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.25.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.25.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.25.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.26.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.26.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.26.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.26.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.26.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.26.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.26.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.27.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.27.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.27.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.27.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.27.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.27.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.27.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.28.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.28.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.28.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.28.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.28.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.28.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.28.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.29.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.29.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.29.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.29.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.29.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.29.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.29.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.30.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.30.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.30.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.30.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.30.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.30.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.30.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.31.attention.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.31.intermediate.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.31.output.dense.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.31.layernorm_before.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.31.layernorm_before.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.31.layernorm_after.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.encoder.layer.31.layernorm_after.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.layernorm.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: vit.layernorm.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: classifier.weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: classifier.bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3284
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3285
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3291
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3306
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3307
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3308
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3331
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3332
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3333
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3356
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3357
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3358
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3381
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3382
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3383
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3406
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3407
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3408
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3431
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3432
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3433
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3456
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3457
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3458
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3481
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3482
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3483
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3506
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3507
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3508
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3531
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3532
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3533
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3556
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3557
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3558
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3581
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3582
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3583
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3606
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3607
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3608
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3631
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3632
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3633
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3656
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3657
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3658
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3681
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3682
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3683
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3706
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3707
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3708
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3731
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3732
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3733
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3756
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3757
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3758
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3781
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3782
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3783
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3806
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3807
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3808
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3831
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3832
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3833
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3856
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3857
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3858
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3881
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3882
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3883
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3906
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3907
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3908
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3931
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3932
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3933
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3956
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3957
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3958
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3981
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3982
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_3983
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_4006
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_4007
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_4008
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_4031
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_4032
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_4033
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_4056
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_4057
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_4058
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_4081
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_4082
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: onnx::MatMul_4083
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_0_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_0_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_1_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_1_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_2_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_2_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_3_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_3_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_4_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_4_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_5_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_5_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_6_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_6_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_7_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_7_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_8_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_8_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_9_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_9_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_10_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_10_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_11_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_11_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_12_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_12_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_13_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_13_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_14_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_14_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_15_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_15_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_16_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_16_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_17_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_17_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_18_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_18_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_19_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_19_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_20_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_20_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_21_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_21_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_22_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_22_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_23_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_23_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_24_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_24_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_25_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_25_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_26_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_26_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_27_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_27_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_28_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_28_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_29_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_29_qkv_bias
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_30_qkv_weight
[02/19/2024-13:48:03] [V] [TRT] Importing initializer: Attention_30_qkv_bias
ERROR: Output name is not unique: 
[02/19/2024-13:48:03] [E] [TRT] ModelImporter.cpp:777: ERROR: ModelImporter.cpp:121 In function parseGraph:
[5] Assertion failed: toposort(graph.node(), &topoOrder) && "Failed to sort the model topologically."
[02/19/2024-13:48:04] [E] Failed to parse onnx file
[02/19/2024-13:48:04] [I] Finished parsing network model. Parse time: 10.1709
[02/19/2024-13:48:04] [E] Parsing model failed
[02/19/2024-13:48:04] [E] Failed to create engine from model or file.
[02/19/2024-13:48:04] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8601] # trtexec --onnx=/workspace/model/vith14.onnx --verbose

I find it weird that the error message Assertion failed: toposort(graph.node(), &topoOrder) && "Failed to sort the model topologically." appears if during the optimization with onnxruntime it said save_model_to_file: Sort graphs in topological order. The thing is, if I load the program with

home_dir = os.path.expanduser("~")
model_path = os.path.join(home_dir, "model/opt/vith14.onnx")
model = onnx.load(model_path)
onnx.checker.check_model(model)

it doesn’t throw an exception. I’ve also performed inference with onnxruntime and the accuracy on CIFAR-10 test set is the same as the checkpoint announces. I also don’t understand why there is an error "ERROR: Output name is not unique: ", but it does not matter if I change the output name when converting the checkpoint to onnx, the error will be the same. The model I am using is:

vith14.onnx

And the checkpoint:

checkpoint.ckpt

Any help you can give is much appreciated.

Environment

TensorRT Version: 8.6.1
GPU Type: Tesla T4
Nvidia Driver Version: 545.23.08
CUDA Version: 12.3
CUDNN Version: 9
Operating System + Version: Ubuntu 22.04.3 LTS (GNU/Linux 5.15.0-94-generic x86_64)
Python Version: 3.11.7
PyTorch Version: 2.2.0

Hi @jogadel137 ,
polygraphy surgeon sanitize has a --toposort option.
This error means that some tensors in the graph (not necessarily the graph outputs) have duplicate names.
If you have it loaded in ONNX-GS, then you can do graph.tensors(check_duplicates=True) to find the duplicated tensors more easily.

Thanks

Hi @AakankshaS . Thanks for your help. I’ve taken a look into it and, as suggested, I did:

import onnx_graphsurgeon as gs
import onnx

graph = gs.import_onnx(onnx.load("vith14.onnx"))
tensors = graph.tensors(check_duplicates=True)

According to the documentation, this should rise an exception if multiple distinct tensors in the graph share the same name, but for this model it doesn’t. I’ve also tried to list every tensor name with

import onnx
model = onnx.load("vith14_mod.onnx")

for tensor in model.graph.initializer:
    print(tensor.name)

for tensor in model.graph.input:
    print(tensor.name)

for tensor in model.graph.output:
    print(tensor.name)

for node in model.graph.node:
    for output_name in node.output:
        print(output_name)

And in that case model.graph.output does indeed show two tensors named “output”. However, changing their names so they are distinct won’t work. Instead, what I did was saving the model as external data with:

import onnx
onnx_model = onnx.load(onnx_model_path)
onnx.save_model(onnx_model, "../model/opt/vith14.onnx", save_as_external_data=True, all_tensors_to_one_file=True, location="filename", size_threshold=1024*3, convert_attribute=False)

this time without performing any optimizations using onnxruntime.transformers.optimizer as I did last time. With that, I’m able to compile the engine using trtexec or using onnxruntime with the tensorrt EP, but the accuracy is 10% so, given there are 10 classes in the CIFAR-10 dataset, I think is not loading the external data. The only way I’m able to perform inference on this model is using onnxruntime with CUDA EP.