Using TRT 8.5.1 on RTX 4070 Ti - Unsupported SM Error

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Description

Attempting to build a .engine file from the PeopleNet etlt with tao-converter fails with:
[ERROR] 1: [caskBuilderUtils.cpp::trtSmToCaskCCV::548] Error Code 1: Internal Error (Unsupported SM: 0x809)
[ERROR] Unable to create engine
Segmentation fault (core dumped)

Environment

TensorRT Version:
8.5.1.7

GPU Type:
RTX 4070 Ti

Nvidia Driver Version:
NVIDIA-SMI 525.125.06
Driver Version: 525.125.06
CUDA Version: 12.0

Operating System + Version:
Ubuntu 20.04

Steps To Reproduce

Install tao-converter: v4.0.0_trt8.5.1.7_x86
Download PeopleNet etlt file
Run tao-converter:
tao-converter -k tlt_encode -d 3,544,960 -o output_bbox/BiasAdd,output_cov/Sigmoid -e /data-volume/triton/models/peoplenet/1/resnet34_peoplenet_pruned_int8.engine -b 8 -t fp16 /data-volume/triton/models/peoplenet/1/resnet34_peoplenet_pruned_int8.etlt

2

Hi,

This looks like a TAO Toolkit related issue. We will move this post to the TAO Toolkit forum.

Thanks!

5

May I know the environment you are running tao-converter? Is it in docker, if yes, which docker is it?

6

This is in a Docker container, which is based on: nvcr.io/nvidia/tritonserver:22.07-py3

7

According to Release Notes :: NVIDIA Deep Learning Triton Inference Server Documentation, the TensorRT version is 8.4.1 in nvcr.io/nvidia/tritonserver:22.07-py3 , could you download 8.4 version of tao-converter and retry?

See TAO Converter | NVIDIA NGC
wget --content-disposition 'https://api.ngc.nvidia.com/v2/resources/org/nvidia/team/tao/tao-converter/v3.22.05_trt8.4_x86/files?redirect=true&path=tao-converter' -O tao-converter

8

I get the same error with v3.22.05_trt8.4_x86, and I also tried the very latest, v4.0.0_trt8.5.1.7_x86.

My full steps to repro are:
Docker run nvcr.io/nvidia/tritonserver:22.07-py3
apt-get install libssl-dev -y
export TRT_LIB_PATH=“/usr/lib/x86_64-linux-gnu”
export TRT_INC_PATH=“/usr/include/x86_64-linux-gnu”
TAO_VER=v4.0.0_trt8.5.1.7_x86

wget --content-disposition “https://api.ngc.nvidia.com/v2/resources/nvidia/tao/tao-converter/versions/${TAO_VER}/files?redirect=true&path=tao-converter” -O /app/tao-converter

chmod +x /app/tao-converter

/app/tao-converter -k tlt_encode -d 3,544,960 -o output_bbox/BiasAdd,output_cov/Sigmoid -e /data-volume/triton/models/peoplenet/1/resnet34_peoplenet_pruned_int8.engine -b 8 -t fp16 /data-volume/triton/models/peoplenet/1/resnet34_peoplenet_pruned_int8.etlt

Output:

root@4f86daf5f64b:/app/tao# ./tao-converter -k tlt_encode -d 3,544,960 -o output_bbox/BiasAdd,output_cov/Sigmoid -e /data-volume/triton/models/peoplenet/1/resnet34_peoplenet_pruned_int8.engine -b 8 -t fp16 /data-volume/triton/models/peoplenet/1/resnet34_peoplenet_pruned_int8.etlt
[INFO] [MemUsageChange] Init CUDA: CPU +331, GPU +0, now: CPU 338, GPU 360 (MiB)
[INFO] [MemUsageChange] Init builder kernel library: CPU +0, GPU +0, now: CPU 358, GPU 360 (MiB)
[WARNING] The implicit batch dimension mode has been deprecated. Please create the network with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag whenever possible.
[INFO] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +841, GPU +362, now: CPU 1227, GPU 722 (MiB)
[INFO] [MemUsageChange] Init cuDNN: CPU +129, GPU +58, now: CPU 1356, GPU 780 (MiB)
[INFO] Local timing cache in use. Profiling results in this builder pass will not be stored.
[INFO] Some tactics do not have sufficient workspace memory to run. Increasing workspace size will enable more tactics, please check verbose output for requested sizes.
[ERROR] 1: [caskBuilderUtils.cpp::trtSmToCaskCCV::548] Error Code 1: Internal Error (Unsupported SM: 0x809)
[ERROR] Unable to create engine
Segmentation fault (core dumped)

9

Could you please try to use nvcr.io/nvidia/tritonserver:23.02-py3
and nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf1.15.5 ? Thanks.

10

Upgrading the Docker image from tritonserver:22.07-py3 to tritonserver:23.02-py3 and using the TAO download v4.0.0_trt8.5.1.7_x86 (full download command below) seemed to work. I will run this upgraded image through the rest of the process and open new topics if anything new comes up. Thanks for the help.

wget --content-disposition “https://api.ngc.nvidia.com/v2/resources/org/nvidia/team/tao/tao-converter/v4.0.0_trt8.5.1.7_x86/files?redirect=true&path=tao-converter” -O /app/tao-converter

11

Thanks for the info.

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.