I use a model (trt file) converted from onnx, some errors occurs
[E] [TRT] ../rtSafe/safeContext.cpp(105) -Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
[E] [TRT] INVALID_STATE: std::exception
[E] [TRT] INVALID_CONFIG: Deserialize the cuda engine failed.
when I run my program under root privilege with same configuration, the above error disppear.
and my configure is
ubuntu 16.04
Cuda 9.0
Cudnn 7.6.3
Tensorrt 6.0.1.5
Hi,
This is likely due to either (1) mismatched versions of libraries/dependencies, or (2) sometimes this can happen due to Out Of Memory (OOM) errors.
Since you say it works when ran with root privilege, can you check that you didn’t have another process using too much memory at the same time when running as normal user?
You can easily check this with nvidia-smi, and verify that you have plenty of memory available before trying to convert the model.
If this isn’t the case, then it’s probably due to mismatched versions, and maybe when run as user vs. root, your PATH is pointing to different CUDA/CUDNN install locations or something along those lines. You can try looking into that to make sure your environment variables are setup correctly.
Alternatively, you could remove the hassle of host-side dependencies by using our NGC Docker containers instead: TensorRT | NVIDIA NGC
Thanks,
NVIDIA Enterprise Support
I change the permission of “.nv” folder in $HOME directory.
I encountered this error by calling tf and trt in the same function, like
x = tf.constant([1])
x = tf.reshape(x, [-1]) # comment out this line then everything will work
self.engine = self.load_engine(self.trt_engine_path)
self.context = self.engine.create_execution_context()
If I don’t call tf.reshape then it will work
Have you solve this problem?
I meet the same one in tftrt during calibrate step:
2020-02-15 06:31:56.479732: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-02-15 06:31:56.480367: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce GTX 1660 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:01:00.0
2020-02-15 06:31:56.480464: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-02-15 06:31:56.480474: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-02-15 06:31:56.480527: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-02-15 06:31:56.480542: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-02-15 06:31:56.480574: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-02-15 06:31:56.480587: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-02-15 06:31:56.480598: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-02-15 06:31:56.480684: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-02-15 06:31:56.481008: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-02-15 06:31:56.481334: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-02-15 06:31:56.481374: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-02-15 06:31:56.481381: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-02-15 06:31:56.481406: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-02-15 06:31:56.481480: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-02-15 06:31:56.481843: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-02-15 06:31:56.482201: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5261 MB memory) → physical GPU (device: 0, name: GeForce GTX 1660 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
2020-02-15 06:31:57.831949: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7fab100085e0
2020-02-15 06:31:57.832132: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.6
2020-02-15 06:31:57.833452: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.6
2020-02-15 06:32:14.169043: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger …/rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2020-02-15 06:32:14.171850: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger …/rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2020-02-15 06:32:14.172129: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
terminate called without an active exception
Aborted (core dumped)
During the convert step, it cost only 400MB memory but it use the full memory during calibrate?
There is enough memory and I set per_process_gpu_memory_fraction in config.
Anyone could give some ideas about this problem?
Seems I can’t call tensorflow and tensorrt within the same thread
HI
I meet the same one in tftrt during calibrate step, i send only one image to calibrate model, but calibration used all of GPU(12G) and i got error:
Calibration failed: Internal: Failed to build TensorRT engine
I also meet this error during calibrate step:
this is my test script:
import os
2 import sys
3 import time
4 from PIL import Image
5
6 import numpy as np
7 import tensorflow as tf
8 from tensorflow.python.compiler.tensorrt import trt_convert as trt
9
10 sys.path.append(“.”)
11 from preprocessing import vgg_preprocess as vgg_preprocessing
12
13 BATCH_SIZE = 8
14 NUM_CALIB_IMAGES = 32
15
16 # SAVED_MODEL_DIR = “…/models/resnet_v1_fp32/saved_model”
17 SAVED_MODEL_DIR = “…/models/ssd_mobilenet-v2/saved_model”
18 INT8_SAVED_MODEL_DIR = SAVED_MODEL_DIR + “_TFTRT_INT8/2”
19
20 data_directory = “…/imagenet-data/Calibration_data”
21
22 calibration_files = [os.path.join(path, name) for path, _, files in os.walk(data_directory) for name in files]
23 print(‘There are %d calibration files. \n%s\n%s\n…’ % (
24 len(calibration_files), calibration_files[0], calibration_files[-1]))
25 calibration_files = calibration_files[:NUM_CALIB_IMAGES]
26
27
28 def parse_file(filepath):
29 image = tf.io.read_file(filepath)
30 image = tf.image.decode_jpeg(image, channels=3)
31 image = vgg_preprocessing(image, 224, 224)
32 return image
33
34
35 def read_image(image_path):
36 image = Image.open(image_path).convert(‘RGB’)
37 image = image.resize(224, 224)
38 return np.array(image)
39
41 num_calibration_batches = 2
44 dataset = tf.data.Dataset.from_tensor_slices(calibration_files)
45 dataset = dataset.map(map_func=parse_file, num_parallel_calls=20)
46 dataset = dataset.batch(batch_size=BATCH_SIZE)
47 dataset = dataset.repeat(None)
48 calibration_dataset = dataset.take(num_calibration_batches)
49
50
51 def calibration_input_fn():
52 for x in calibration_dataset:
53 ¦ yield (x,)
54
55
56 conversion_params = trt.DEFAULT_TRT_CONVERSION_PARAMS._replace(
57 precision_mode=trt.TrtPrecisionMode.INT8)
58
59 converter = trt.TrtGraphConverterV2(
60 input_saved_model_dir=SAVED_MODEL_DIR,
61 conversion_params=conversion_params)
62
63 converter.convert(calibration_input_fn=calibration_input_fn)
64
65 converter.save(INT8_SAVED_MODEL_DIR)
This is my error message:
There are 64 calibration files.
…/imagenet-data/Calibration_data/ILSVRC2017_test_00000022.JPEG
…/imagenet-data/Calibration_data/ILSVRC2017_test_00000038.JPEG
…
2020-03-19 10:12:50.778300: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-03-19 10:12:50.796129: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-19 10:12:50.796481: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.71
pciBusID: 0000:01:00.0
2020-03-19 10:12:50.796490: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2020-03-19 10:12:50.796520: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-19 10:12:50.796825: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-19 10:12:50.797105: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-03-19 10:12:50.798116: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-19 10:12:50.798390: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.71
pciBusID: 0000:01:00.0
2020-03-19 10:12:50.798399: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2020-03-19 10:12:50.798427: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-19 10:12:50.798701: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-19 10:12:50.798993: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-03-19 10:12:51.132226: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-03-19 10:12:51.132253: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-03-19 10:12:51.132258: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-03-19 10:12:51.132362: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-19 10:12:51.132718: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-19 10:12:51.133007: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-19 10:12:51.133293: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6969 MB memory) → physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5)
2020-03-19 10:12:51.136525: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x16d36b20 executing computations on platform CUDA. Devices:
2020-03-19 10:12:51.136536: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): GeForce RTX 2080, Compute Capability 7.5
2020-03-19 10:12:51.156288: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3696000000 Hz
2020-03-19 10:12:51.156868: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x16e0eff0 executing computations on platform Host. Devices:
2020-03-19 10:12:51.156882: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): Host, Default Version
2020-03-19 10:12:56.139345: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-19 10:12:56.139611: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2020-03-19 10:12:56.139724: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2020-03-19 10:12:56.140537: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-19 10:12:56.140791: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.71
pciBusID: 0000:01:00.0
2020-03-19 10:12:56.140798: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2020-03-19 10:12:56.140838: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-19 10:12:56.141085: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-19 10:12:56.141328: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-03-19 10:12:56.141359: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-03-19 10:12:56.141364: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-03-19 10:12:56.141369: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-03-19 10:12:56.141459: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-19 10:12:56.141716: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-19 10:12:56.141963: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6969 MB memory) → physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5)
2020-03-19 10:12:56.322819: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:716] Optimization results for grappler item: graph_to_optimize
2020-03-19 10:12:56.322850: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] function_optimizer: function_optimizer did nothing. time = 0.002ms.
2020-03-19 10:12:56.322854: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] function_optimizer: function_optimizer did nothing. time = 0ms.
2020-03-19 10:13:00.562811: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-19 10:13:00.563086: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2020-03-19 10:13:00.563566: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2020-03-19 10:13:00.564515: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-19 10:13:00.564753: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.71
pciBusID: 0000:01:00.0
2020-03-19 10:13:00.564762: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2020-03-19 10:13:00.564791: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-19 10:13:00.565027: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-19 10:13:00.565258: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-03-19 10:13:00.565275: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-03-19 10:13:00.565279: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-03-19 10:13:00.565283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-03-19 10:13:00.565361: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-19 10:13:00.565600: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1006] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-19 10:13:00.565828: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6969 MB memory) → physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5)
2020-03-19 10:13:02.015354: I tensorflow/compiler/tf2tensorrt/segment/segment.cc:460] There are 3648 ops of 48 different types in the graph that are not converted to TensorRT: TopKV2, Squeeze, Range, Conv2D, TensorArrayWriteV3, Const, Add, Placeholder, Switch, TensorArrayGatherV3, NoOp, NextIteration, Greater, NonMaxSuppressionV2, StridedSlice, Less, TensorArraySizeV3, ExpandDims, Exit, Cast, Identity, Shape, TensorArrayReadV3, Reshape, Merge, LoopCond, TensorArrayV3, Enter, TensorArrayScatterV3, Mul, Equal, Sub, Minimum, Split, ZerosLike, Pack, Size, Unpack, Assert, DataFormatVecPermute, Transpose, Gather, Tile, Fill, Slice, Where, Maximum, ConcatV2, (For more information see Accelerating Inference In TF-TRT User Guide :: NVIDIA Deep Learning Frameworks Documentation).
2020-03-19 10:13:02.057671: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:633] Number of TensorRT candidate segments: 6
2020-03-19 10:13:02.143827: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:734] TensorRT node TRTEngineOp_0 added for segment 0 consisting of 508 nodes succeeded.
2020-03-19 10:13:02.144771: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:734] TensorRT node BoxPredictor_1/ClassPredictor/TRTEngineOp_1 added for segment 1 consisting of 4 nodes succeeded.
2020-03-19 10:13:02.144819: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:734] TensorRT node TRTEngineOp_2 added for segment 2 consisting of 18 nodes succeeded.
2020-03-19 10:13:02.144914: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:734] TensorRT node TRTEngineOp_3 added for segment 3 consisting of 18 nodes succeeded.
2020-03-19 10:13:02.144968: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:734] TensorRT node TRTEngineOp_4 added for segment 4 consisting of 7 nodes succeeded.
2020-03-19 10:13:02.145006: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:734] TensorRT node TRTEngineOp_5 added for segment 5 consisting of 3 nodes succeeded.
2020-03-19 10:13:02.422876: W tensorflow/compiler/tf2tensorrt/convert/trt_optimization_pass.cc:183] TensorRTOptimizer is probably called on funcdef! This optimizer must NOT be called on function objects.
2020-03-19 10:13:02.587558: W tensorflow/compiler/tf2tensorrt/convert/trt_optimization_pass.cc:183] TensorRTOptimizer is probably called on funcdef! This optimizer must NOT be called on function objects.
2020-03-19 10:13:02.648270: W tensorflow/compiler/tf2tensorrt/convert/trt_optimization_pass.cc:183] TensorRTOptimizer is probably called on funcdef! This optimizer must NOT be called on function objects.
2020-03-19 10:13:02.652283: W tensorflow/compiler/tf2tensorrt/convert/trt_optimization_pass.cc:183] TensorRTOptimizer is probably called on funcdef! This optimizer must NOT be called on function objects.
2020-03-19 10:13:02.655959: W tensorflow/compiler/tf2tensorrt/convert/trt_optimization_pass.cc:183] TensorRTOptimizer is probably called on funcdef! This optimizer must NOT be called on function objects.
2020-03-19 10:13:02.659025: W tensorflow/compiler/tf2tensorrt/convert/trt_optimization_pass.cc:183] TensorRTOptimizer is probably called on funcdef! This optimizer must NOT be called on function objects.
2020-03-19 10:13:02.671226: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:716] Optimization results for grappler item: tf_graph
2020-03-19 10:13:02.671257: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 6095 nodes (-1940), 10082 edges (-2174), time = 452.637ms.
2020-03-19 10:13:02.671261: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] layout: Graph size after: 6297 nodes (202), 10284 edges (202), time = 377.836ms.
2020-03-19 10:13:02.671265: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 6127 nodes (-170), 10114 edges (-170), time = 290.92ms.
2020-03-19 10:13:02.671269: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] TensorRTOptimizer: Graph size after: 5575 nodes (-552), 9543 edges (-571), time = 314.315ms.
2020-03-19 10:13:02.671272: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 5574 nodes (-1), 9543 edges (0), time = 221.983ms.
2020-03-19 10:13:02.671275: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:716] Optimization results for grappler item: TRTEngineOp_5_native_segment
2020-03-19 10:13:02.671278: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 10 nodes (0), 9 edges (0), time = 0.443ms.
2020-03-19 10:13:02.671282: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] layout: Graph size after: 10 nodes (0), 9 edges (0), time = 0.163ms.
2020-03-19 10:13:02.671286: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 10 nodes (0), 9 edges (0), time = 0.343ms.
2020-03-19 10:13:02.671291: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] TensorRTOptimizer: Graph size after: 10 nodes (0), 9 edges (0), time = 0.041ms.
2020-03-19 10:13:02.671294: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 10 nodes (0), 9 edges (0), time = 0.424ms.
2020-03-19 10:13:02.671298: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:716] Optimization results for grappler item: TRTEngineOp_0_native_segment
2020-03-19 10:13:02.671302: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 526 nodes (0), 535 edges (0), time = 36.556ms.
2020-03-19 10:13:02.671305: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] layout: Graph size after: 526 nodes (0), 535 edges (0), time = 49.395ms.
2020-03-19 10:13:02.671309: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 526 nodes (0), 535 edges (0), time = 39.263ms.
2020-03-19 10:13:02.671312: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] TensorRTOptimizer: Graph size after: 526 nodes (0), 535 edges (0), time = 5.479ms.
2020-03-19 10:13:02.671315: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 526 nodes (0), 535 edges (0), time = 37.914ms.
2020-03-19 10:13:02.671319: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:716] Optimization results for grappler item: TRTEngineOp_2_native_segment
2020-03-19 10:13:02.671323: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 24 nodes (0), 27 edges (0), time = 0.676ms.
2020-03-19 10:13:02.671326: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] layout: Graph size after: 24 nodes (0), 27 edges (0), time = 0.305ms.
2020-03-19 10:13:02.671329: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 24 nodes (0), 27 edges (0), time = 0.582ms.
2020-03-19 10:13:02.671333: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] TensorRTOptimizer: Graph size after: 24 nodes (0), 27 edges (0), time = 0.067ms.
2020-03-19 10:13:02.671336: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 24 nodes (0), 27 edges (0), time = 0.53ms.
2020-03-19 10:13:02.671339: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:716] Optimization results for grappler item: TRTEngineOp_4_native_segment
2020-03-19 10:13:02.671343: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 14 nodes (0), 13 edges (0), time = 0.359ms.
2020-03-19 10:13:02.671346: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] layout: Graph size after: 14 nodes (0), 13 edges (0), time = 0.217ms.
2020-03-19 10:13:02.671350: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 14 nodes (0), 13 edges (0), time = 0.393ms.
2020-03-19 10:13:02.671353: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] TensorRTOptimizer: Graph size after: 14 nodes (0), 13 edges (0), time = 0.052ms.
2020-03-19 10:13:02.671357: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 14 nodes (0), 13 edges (0), time = 0.544ms.
2020-03-19 10:13:02.671361: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:716] Optimization results for grappler item: TRTEngineOp_3_native_segment
2020-03-19 10:13:02.671365: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 24 nodes (0), 27 edges (0), time = 0.436ms.
2020-03-19 10:13:02.671368: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] layout: Graph size after: 24 nodes (0), 27 edges (0), time = 0.306ms.
2020-03-19 10:13:02.671372: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 24 nodes (0), 27 edges (0), time = 0.454ms.
2020-03-19 10:13:02.671376: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] TensorRTOptimizer: Graph size after: 24 nodes (0), 27 edges (0), time = 0.043ms.
2020-03-19 10:13:02.671379: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 24 nodes (0), 27 edges (0), time = 0.608ms.
2020-03-19 10:13:02.671383: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:716] Optimization results for grappler item: BoxPredictor_1/ClassPredictor/TRTEngineOp_1_native_segment
2020-03-19 10:13:02.671386: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 6 nodes (0), 5 edges (0), time = 0.261ms.
2020-03-19 10:13:02.671389: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] layout: Graph size after: 6 nodes (0), 5 edges (0), time = 0.148ms.
2020-03-19 10:13:02.671392: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 6 nodes (0), 5 edges (0), time = 0.264ms.
2020-03-19 10:13:02.671395: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] TensorRTOptimizer: Graph size after: 6 nodes (0), 5 edges (0), time = 0.028ms.
2020-03-19 10:13:02.671399: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 6 nodes (0), 5 edges (0), time = 0.262ms.
2020-03-19 10:13:08.026309: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:812] Starting calibration thread on device 0, Calibration Resource @ 0x7f402801dce0
2020-03-19 10:13:08.784457: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger …/rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2020-03-19 10:13:08.784723: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:41] DefaultLogger …/rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
2020-03-19 10:13:08.784995: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:838] Calibration failed: Internal: Failed to build TensorRT engine
2020-03-19 10:13:08.785125: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Internal: Failed to feed calibration data
[[{{node TRTEngineOp_0}}]]
[[Postprocessor/BatchMultiClassNonMaxSuppression/map/while/Identity/_76]]
2020-03-19 10:13:08.785267: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Internal: Failed to feed calibration data
[[{{node TRTEngineOp_0}}]]
Traceback (most recent call last):
File “V2ConvertTFTRTInt8.py”, line 63, in
converter.convert(calibration_input_fn=calibration_input_fn)
File “/home/guo/runprogram/Envs/py36tf20/lib/python3.6/site-packages/tensorflow_core/python/compiler/tensorrt/trt_convert.py”, line 984, in convert
self._converted_func(*map(ops.convert_to_tensor, inp))
File “/home/guo/runprogram/Envs/py36tf20/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py”, line 1081, in call
return self._call_impl(args, kwargs)
File “/home/guo/runprogram/Envs/py36tf20/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py”, line 1121, in _call_impl
return self._call_flat(args, self.captured_inputs, cancellation_manager)
File “/home/guo/runprogram/Envs/py36tf20/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py”, line 1224, in _call_flat
ctx, args, cancellation_manager=cancellation_manager)
File “/home/guo/runprogram/Envs/py36tf20/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py”, line 511, in call
ctx=ctx)
File “/home/guo/runprogram/Envs/py36tf20/lib/python3.6/site-packages/tensorflow_core/python/eager/execute.py”, line 67, in quick_execute
six.raise_from(core._status_to_exception(e.code, message), None)
File “”, line 3, in raise_from
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
(0) Internal: Failed to feed calibration data
[[node TRTEngineOp_0 (defined at /home/guo/runprogram/Envs/py36tf20/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1751) ]]
[[Postprocessor/BatchMultiClassNonMaxSuppression/map/while/Identity/_76]]
(1) Internal: Failed to feed calibration data
[[node TRTEngineOp_0 (defined at /home/guo/runprogram/Envs/py36tf20/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1751) ]]
0 successful operations.
0 derived errors ignored. [Op:__inference_pruned_50623]
Function call stack:
pruned → pruned