Exclude TacticSources from IBuilderConfig

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Hello!
Since my TensorRT code running on Nano based on: https://github.com/noahmr/yolov5-tensorrt is consuming cca 1 GB RAM I’d like to decrease memory usage by - according your advice - excluding CUDNN from “tactic sources”.
Problem is: I can’t find how exactly to perform this exclusion (neither in TRT doc nor forum)?

I tried it inside /src/yolov5_builder.cpp (which is now part of my app) like this, but doesn’t seem to exclude:

std::unique_ptr<nvinfer1::IBuilderConfig> config(builder->createBuilderConfig()); //LN 218
config->setTacticSources(0 << int(nvinfer1::TacticSource::kCUDNN));                 //MY LINE

Beside CUDNN I’d like to attempt exclusion of CuBLAS/CuBLASLt, so question applies to that as well
Many thanks!

Environment

TensorRT Version : 8.0.1
GPU Type : Jetson Nano (Maxwell)
CUDA Version : 10.2
CUDNN Version : 8.2.1
Operating System + Version : Ubuntu 18.04 (Jetpack 4.6)

2

Hi,

This looks like a Jetson issue. Please refer to the below samples in case useful.

For any further assistance, we will move this post to to Jetson related forum.

Thanks!

4

Sir, not sure if we understood well: inference is implemented and works OK I just want to decrease RAM usage by excluding tactic sources-an approach you suggested in my previous thread. I think I just need (correct) argument(s) for config->setTacticSources() call… And it is C++

5

any opinions on this?

6

Hi @JoleCRO ,
Quoting from TRT developer guide.

TensorRT’s dependencies (NVIDIA cuDNN and NVIDIA cuBLAS) can occupy large amounts of device memory. TensorRT allows you to control whether these libraries are used for inference by using the TacticSources (C++, Python) attribute in the builder configuration. Note that some plugin implementations require these libraries, so that when they are excluded, the network may not be compiled successfully.

Each of the tactic source is explained here along with their default status