Benchmark tensorrt model in UFF extension on jetson AGX Orin

I downloaded the ssd_mobilenet_v2_coco.uff from
https://nvidia.box.com/shared/static/jcdewxep8vamzm71zajcovza938lygre.gz, I’d like to benchmark this tensorrt model on Jetson AGX Orin.

Using model (unet-segmentation.uff) in jetson_benchmarks/benchmark_csv/orin-benchmarks.csv at master · NVIDIA-AI-IOT/jetson_benchmarks · GitHub as an example, I saved the ssd_mobilenet_v2_coco.uff under models dir, I updated orin-benchmarks.csv and benchmark.py to execute the ssd_mobilenet_v2_coco.uff, but it failed.

My question here: does the tensorrt model (ssd_mobilenet_v2_coco.uff) work on AGX Orin? if so, what might be the problem? I suspect the input (input,3,300, 300) and out (NMP) ) in orin_benchmark.csv, but run out of ideas.

Hi,

Would you mind sharing the error log so we can know more about the issue?

Thanks.

/usr/src/tensorrt/bin/trtexec --uff=ssd_mobilenet.uff --output=NMP --uffInput=input_1,31,300,300 --verbose

[06/11/2024-16:01:33] [I] === Model Options ===
[06/11/2024-16:01:33] [I] Format: UFF
[06/11/2024-16:01:33] [I] Model: ssd_mobilenet.uff
[06/11/2024-16:01:33] [I] Uff Inputs Layout: NCHW
[06/11/2024-16:01:33] [I] Input: input_1,31,300,300
[06/11/2024-16:01:33] [I] Output: NMP
[06/11/2024-16:01:33] [I] === Build Options ===
[06/11/2024-16:01:33] [I] Max batch: 1
[06/11/2024-16:01:33] [I] Memory Pools: workspace: default, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default
[06/11/2024-16:01:33] [I] minTiming: 1
[06/11/2024-16:01:33] [I] avgTiming: 8
[06/11/2024-16:01:33] [I] Precision: FP32
[06/11/2024-16:01:33] [I] LayerPrecisions:
[06/11/2024-16:01:33] [I] Calibration:
[06/11/2024-16:01:33] [I] Refit: Disabled
[06/11/2024-16:01:33] [I] Sparsity: Disabled
[06/11/2024-16:01:33] [I] Safe mode: Disabled
[06/11/2024-16:01:33] [I] DirectIO mode: Disabled
[06/11/2024-16:01:33] [I] Restricted mode: Disabled
[06/11/2024-16:01:33] [I] Build only: Disabled
[06/11/2024-16:01:33] [I] Save engine:
[06/11/2024-16:01:33] [I] Load engine:
[06/11/2024-16:01:33] [I] Profiling verbosity: 0
[06/11/2024-16:01:33] [I] Tactic sources: Using default tactic sources
[06/11/2024-16:01:33] [I] timingCacheMode: local
[06/11/2024-16:01:33] [I] timingCacheFile:
[06/11/2024-16:01:33] [I] Heuristic: Disabled
[06/11/2024-16:01:33] [I] Preview Features: Use default preview flags.
[06/11/2024-16:01:33] [I] Input(s)s format: fp32:CHW
[06/11/2024-16:01:33] [I] Output(s)s format: fp32:CHW
[06/11/2024-16:01:33] [I] Input build shapes: model
[06/11/2024-16:01:33] [I] Input calibration shapes: model
[06/11/2024-16:01:33] [I] === System Options ===
[06/11/2024-16:01:33] [I] Device: 0
[06/11/2024-16:01:33] [I] DLACore:
[06/11/2024-16:01:33] [I] Plugins:
[06/11/2024-16:01:33] [I] === Inference Options ===
[06/11/2024-16:01:33] [I] Batch: 1
[06/11/2024-16:01:33] [I] Input inference shapes: model
[06/11/2024-16:01:33] [I] Iterations: 10
[06/11/2024-16:01:33] [I] Duration: 3s (+ 200ms warm up)
[06/11/2024-16:01:33] [I] Sleep time: 0ms
[06/11/2024-16:01:33] [I] Idle time: 0ms
[06/11/2024-16:01:33] [I] Streams: 1
[06/11/2024-16:01:33] [I] ExposeDMA: Disabled
[06/11/2024-16:01:33] [I] Data transfers: Enabled
[06/11/2024-16:01:33] [I] Spin-wait: Disabled
[06/11/2024-16:01:33] [I] Multithreading: Disabled
[06/11/2024-16:01:33] [I] CUDA Graph: Disabled
[06/11/2024-16:01:33] [I] Separate profiling: Disabled
[06/11/2024-16:01:33] [I] Time Deserialize: Disabled
[06/11/2024-16:01:33] [I] Time Refit: Disabled
[06/11/2024-16:01:33] [I] NVTX verbosity: 0
[06/11/2024-16:01:33] [I] Persistent Cache Ratio: 0
[06/11/2024-16:01:33] [I] Inputs:
[06/11/2024-16:01:33] [I] === Reporting Options ===
[06/11/2024-16:01:33] [I] Verbose: Enabled
[06/11/2024-16:01:33] [I] Averages: 10 inferences
[06/11/2024-16:01:33] [I] Percentiles: 90,95,99
[06/11/2024-16:01:33] [I] Dump refittable layers:Disabled
[06/11/2024-16:01:33] [I] Dump output: Disabled
[06/11/2024-16:01:33] [I] Profile: Disabled
[06/11/2024-16:01:33] [I] Export timing to JSON file:
[06/11/2024-16:01:33] [I] Export output to JSON file:
[06/11/2024-16:01:33] [I] Export profile to JSON file:
[06/11/2024-16:01:33] [I]
[06/11/2024-16:01:33] [I] === Device Information ===
[06/11/2024-16:01:33] [I] Selected Device: Orin
[06/11/2024-16:01:33] [I] Compute Capability: 8.7
[06/11/2024-16:01:33] [I] SMs: 16
[06/11/2024-16:01:33] [I] Compute Clock Rate: 1.3 GHz
[06/11/2024-16:01:33] [I] Device Global Memory: 30588 MiB
[06/11/2024-16:01:33] [I] Shared Memory per SM: 164 KiB
[06/11/2024-16:01:33] [I] Memory Bus Width: 128 bits (ECC disabled)
[06/11/2024-16:01:33] [I] Memory Clock Rate: 1.3 GHz
[06/11/2024-16:01:33] [I]
[06/11/2024-16:01:33] [I] TensorRT version: 8.5.2
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::BatchedNMS_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::BatchTilePlugin_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::Clip_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::CoordConvAC version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::CropAndResizeDynamic version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::CropAndResize version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::DecodeBbox3DPlugin version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::DetectionLayer_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::EfficientNMS_Explicit_TF_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::EfficientNMS_Implicit_TF_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::EfficientNMS_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::FlattenConcat_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::GenerateDetection_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::GridAnchor_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::GridAnchorRect_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::GroupNorm version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 2
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::LayerNorm version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::LReLU_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::MultilevelCropAndResize_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::MultilevelProposeROI_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::MultiscaleDeformableAttnPlugin_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::NMSDynamic_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::NMS_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::Normalize_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::PillarScatterPlugin version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::PriorBox_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::ProposalDynamic version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::ProposalLayer_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::Proposal version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::PyramidROIAlign_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::Region_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::Reorg_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::ResizeNearest_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::ROIAlign_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::RPROI_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::ScatterND version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::SeqLen2Spatial version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::SpecialSlice_TRT version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::SplitGeLU version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::Split version 1
[06/11/2024-16:01:33] [V] [TRT] Registered plugin creator - ::VoxelGeneratorPlugin version 1
[06/11/2024-16:01:33] [I] [TRT] [MemUsageChange] Init CUDA: CPU +220, GPU +0, now: CPU 246, GPU 12021 (MiB)
[06/11/2024-16:01:33] [V] [TRT] Trying to load shared library libnvinfer_builder_resource.so.8.5.2
[06/11/2024-16:01:33] [V] [TRT] Loaded shared library libnvinfer_builder_resource.so.8.5.2
[06/11/2024-16:01:35] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +302, GPU +282, now: CPU 571, GPU 12323 (MiB)
[06/11/2024-16:01:35] [W] [TRT] The implicit batch dimension mode has been deprecated. Please create the network with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag whenever possible.
[06/11/2024-16:01:35] [I] Start parsing network model
[06/11/2024-16:01:35] [V] [TRT] UFFParser: Parsing Input[Op: Input].
[06/11/2024-16:01:35] [V] [TRT] UFFParser: Input → [1,3,300,300]
[06/11/2024-16:01:35] [V] [TRT] UFFParser: Applying order forwarding to: Input
[06/11/2024-16:01:35] [V] [TRT] UFFParser: Parsing FeatureExtractor/MobilenetV2/Conv/weights[Op: Const].
[06/11/2024-16:01:35] [V] [TRT] UFFParser: FeatureExtractor/MobilenetV2/Conv/weights → [3,3,3,32]
[06/11/2024-16:01:35] [V] [TRT] UFFParser: Applying order forwarding to: FeatureExtractor/MobilenetV2/Conv/weights
[06/11/2024-16:01:35] [V] [TRT] UFFParser: Parsing FeatureExtractor/MobilenetV2/Conv/Conv2D[Op: Conv]. Inputs: Input, FeatureExtractor/MobilenetV2/Conv/weights
[06/11/2024-16:01:35] [V] [TRT] UFFParser: Inserting transposes for FeatureExtractor/MobilenetV2/Conv/Conv2D
[06/11/2024-16:01:35] [E] [TRT] UffParser: Parser error: FeatureExtractor/MobilenetV2/Conv/Conv2D: Order size is not matching the number dimensions of TensorRT
[06/11/2024-16:01:35] [E] Failed to parse uff file
[06/11/2024-16:01:35] [I] Finish parsing network model
[06/11/2024-16:01:35] [E] Parsing model failed
[06/11/2024-16:01:35] [E] Failed to create engine from model or file.
[06/11/2024-16:01:35] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8502] # /usr/src/tensorrt/bin/trtexec --uff=ssd_mobilenet.uff --output=NMP --uffInput=input_1,31,300,300 --verbose

Hi,

[E] [TRT] UffParser: Parser error: FeatureExtractor/MobilenetV2/Conv/Conv2D: Order size is not matching the number dimensions of TensorRT

Based on this, it looks like the uff model doesn’t define correctly.
Could you also share the log that converts the TensorFlow model into the uff file?

Thanks.