TF-TRT Saved Model Optimization with top-k

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

I’m using the TF-TRT optimization flow with a saved model described here:

The model I’m optimizing includes a top-k operation which fails for k > 3000. Is it possible using the flow above to specify an operation blacklist to prevent TensorRT from tying to optimize my top-k operation? I thought this was a supported functionality at some point, but nothing in this updated saved model optimization flow seems obvious for a blacklist setting. As I’ll need to support legacy models that have already been trained, it’s not as easy as just changing the top k setting and re-training.

Environment

TensorRT Version: 7.2.1
GPU Type: GTX 3070
Nvidia Driver Version: 455.45.01
CUDA Version: 11.1.74
CUDNN Version: 8.0.4.30
Operating System + Version: Ubuntu 20.04 lts
Python Version (if applicable): 3.8
TensorFlow Version (if applicable): 2.3.1
PyTorch Version (if applicable): N/A
Baremetal or Container (if container which image + tag): N/A

2

Hi, Request you to share the model and script so that we can try reproducing the issue at our end.

Also we recommend you to check the below samples links, as they might answer your concern

Thanks!

3

Thanks for the quick reply! Thanks for the links, yes as I posted that is the documentation I’m working from. Unfortunately it’s not possible for me to share the model or script as they are proprietary.

That said I don’t think I’d need to share the model or scripts, fundamentally my question is if it’s possible in that linked flow (ie convert → build → save) to blacklist specific layers or operations to prevent TensorRT from considering them for optimization.

Thanks!