Keeping a tensor that only consists of constant values while converting Tensorflow Graph_Def model to TensorRT model

Description

I’m looking for a way to keep a tensor that only consists of constant values while converting Tensorflow Graph_Def model to TensorRT model. I successfully converted the model for optimization but the TensorRT model is missing a tensor that has constant values despite the fact that I explicitly programmed for keeping it. My Python program is as follows:

import tensorflow as tf
from tensorflow.python.compiler.tensorrt import trt_convert as trt

graphdef_path = “/home/…/…/graphdef_model.pb”
tensorrt_path = ‘/home/…/…/tensorrt.pb’

graph = tf.Graph()
with graph.as_default():
with tf.Session() as sess:
# Deserialize the frozen graph:
with tf.gfile.GFile(graphdef_path, ‘rb’) as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())

    # Create a TensorRT graph
    trt_graph = trt.create_inference_graph(
        input_graph_def=graph_def,
        outputs=['transpose_71', 'Softmax', 'concat_100', 'concat_102', 'strided_slice_9'],  # 'strided_slice_9' with the size of [1, 19248, 4] is the tensor that only consists of constant values.
        max_batch_size=1,
        max_workspace_size_bytes=1 << 30,
        precision_mode="FP32")

    with open(tensorrt_path, 'wb') as f:
        f.write(trt_graph.SerializeToString())

How can I keep the ‘strided_slice_9’ tensor in the TensorRT model?

Environment

TensorRT Version: 5.0.2-1
GPU Type: GeForce GTX 1070
Nvidia Driver Version: 440.100
CUDA Version: 10.0
CUDNN Version: 7.4.1
Operating System + Version: Ubuntu 18.04 LTS
Python Version (if applicable): 3.7
TensorFlow Version (if applicable): 1.14.0
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)

Steps To Reproduce

Please include:

  • Exact steps/commands to build your repro
  • Exact steps/commands to run your repro
  • Full traceback of errors encountered

Hi @toshikazu.kuroda,
Can you share the verbose logs and your model, so that we can assist you better.
Thanks!

@AakankshaS

Thanks for the reply. Below are the links to GraphDef and TensorRT models along with the verbose logs.

GraphDef model: https://drive.google.com/file/d/1sKnV1qrjYVHgkrNgQc2UrYxoDHxIBvSp/view?usp=sharing

TensorRT model: https://drive.google.com/file/d/1igVNenQz6Pc1os0-HeMnUuxLKxhFYWgF/view?usp=sharing

/home/…/anaconda3/envs/yolact-env-TF1_14_1/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or ‘1type’ as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / ‘(1,)type’.
_np_qint8 = np.dtype([(“qint8”, np.int8, 1)])
/home/…/anaconda3/envs/yolact-env-TF1_14_1/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or ‘1type’ as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / ‘(1,)type’.
_np_quint8 = np.dtype([(“quint8”, np.uint8, 1)])
/home/…/anaconda3/envs/yolact-env-TF1_14_1/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or ‘1type’ as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / ‘(1,)type’.
_np_qint16 = np.dtype([(“qint16”, np.int16, 1)])
/home/…/anaconda3/envs/yolact-env-TF1_14_1/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or ‘1type’ as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / ‘(1,)type’.
_np_quint16 = np.dtype([(“quint16”, np.uint16, 1)])
/home/…/anaconda3/envs/yolact-env-TF1_14_1/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or ‘1type’ as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / ‘(1,)type’.
_np_qint32 = np.dtype([(“qint32”, np.int32, 1)])
/home/…/anaconda3/envs/yolact-env-TF1_14_1/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or ‘1type’ as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / ‘(1,)type’.
np_resource = np.dtype([(“resource”, np.ubyte, 1)])
/home/…/anaconda3/envs/yolact-env-TF1_14_1/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or ‘1type’ as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / ‘(1,)type’.
_np_qint8 = np.dtype([(“qint8”, np.int8, 1)])
/home/…/anaconda3/envs/yolact-env-TF1_14_1/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or ‘1type’ as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / ‘(1,)type’.
_np_quint8 = np.dtype([(“quint8”, np.uint8, 1)])
/home/…/anaconda3/envs/yolact-env-TF1_14_1/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or ‘1type’ as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / ‘(1,)type’.
_np_qint16 = np.dtype([(“qint16”, np.int16, 1)])
/home/…/anaconda3/envs/yolact-env-TF1_14_1/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or ‘1type’ as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / ‘(1,)type’.
_np_quint16 = np.dtype([(“quint16”, np.uint16, 1)])
/home/…/anaconda3/envs/yolact-env-TF1_14_1/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or ‘1type’ as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / ‘(1,)type’.
_np_qint32 = np.dtype([(“qint32”, np.int32, 1)])
/home/…/anaconda3/envs/yolact-env-TF1_14_1/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or ‘1type’ as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / ‘(1,)type’.
np_resource = np.dtype([(“resource”, np.ubyte, 1)])
WARNING:tensorflow:From /home/…/PycharmProjects/graphdef2tensorrt/Main.py:18: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2020-08-02 16:31:17.121966: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2020-08-02 16:31:17.147812: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-02 16:31:17.148167: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.645
pciBusID: 0000:01:00.0
2020-08-02 16:31:17.149270: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2020-08-02 16:31:17.180354: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2020-08-02 16:31:17.195163: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2020-08-02 16:31:17.200227: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2020-08-02 16:31:17.234958: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2020-08-02 16:31:17.257777: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2020-08-02 16:31:17.317849: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-08-02 16:31:17.317979: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-02 16:31:17.318551: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-02 16:31:17.319002: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-08-02 16:31:17.319474: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-08-02 16:31:17.350317: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2799925000 Hz
2020-08-02 16:31:17.351043: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x56309c232280 executing computations on platform Host. Devices:
2020-08-02 16:31:17.351070: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): ,
2020-08-02 16:31:17.351592: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-02 16:31:17.351991: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.645
pciBusID: 0000:01:00.0
2020-08-02 16:31:17.352026: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2020-08-02 16:31:17.352035: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2020-08-02 16:31:17.352042: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2020-08-02 16:31:17.352049: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2020-08-02 16:31:17.352056: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2020-08-02 16:31:17.352082: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2020-08-02 16:31:17.352090: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-08-02 16:31:17.352138: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-02 16:31:17.352526: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-02 16:31:17.352893: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-08-02 16:31:17.352928: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2020-08-02 16:31:17.490653: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-02 16:31:17.490676: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2020-08-02 16:31:17.490681: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2020-08-02 16:31:17.490806: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-02 16:31:17.491127: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-02 16:31:17.491418: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-02 16:31:17.491824: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7336 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-08-02 16:31:17.493572: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x56309e5e2e80 executing computations on platform CUDA. Devices:
2020-08-02 16:31:17.493586: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): GeForce GTX 1070, Compute Capability 6.1
WARNING:tensorflow:From /home/…/PycharmProjects/graphdef2tensorrt/Main.py:20: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

WARNING:tensorflow:From /home/…/PycharmProjects/graphdef2tensorrt/Main.py:21: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.

2020-08-02 16:31:18.847014: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-02 16:31:18.847341: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2020-08-02 16:31:18.847810: I tensorflow/core/grappler/clusters/single_machine.cc:359] Starting new session
2020-08-02 16:31:18.848429: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-02 16:31:18.848738: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.645
pciBusID: 0000:01:00.0
2020-08-02 16:31:18.848763: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2020-08-02 16:31:18.848772: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2020-08-02 16:31:18.848792: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2020-08-02 16:31:18.848799: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2020-08-02 16:31:18.848805: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2020-08-02 16:31:18.848812: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2020-08-02 16:31:18.848819: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-08-02 16:31:18.848853: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-02 16:31:18.849125: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-02 16:31:18.849373: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-08-02 16:31:18.849390: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-02 16:31:18.849395: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2020-08-02 16:31:18.849399: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2020-08-02 16:31:18.849445: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-02 16:31:18.849719: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-02 16:31:18.850110: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7336 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-08-02 16:31:20.834952: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:716] Optimization results for grappler item: tf_graph
2020-08-02 16:31:20.834978: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 1131 nodes (-1991), 1181 edges (-1822), time = 1150.22205ms.
2020-08-02 16:31:20.834983: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] layout: Graph size after: 1131 nodes (0), 1181 edges (0), time = 119.234ms.
2020-08-02 16:31:20.834986: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 1131 nodes (0), 1181 edges (0), time = 162.241ms.

Process finished with exit code 0

Hi @toshikazu.kuroda,

Kindly allow the access.
Thanks!

I’ve just allowed you the access to both .pb files. I think I did it correctly – this is my first time to share a file via Google Drive. Please let me know if it does not work. I will try again.

Thanks.