Tensorflow computation takes too much time for face recognition and detection

Varun_Rathinam · February 29, 2020, 1:58am

Hi,

I’m completely new to this Nvidia jetson TX2, we were using tensorflow based face recognition and face detection, when we run my own python code on jetson tx2, it takes to start too much time, after face recognition and detection also too much time. I were tried improve the performance of jetson by following commands but i won’t able to achieve any improvement;

sudo ~/jetson_clocks.sh
sudo nvpmodel -m 0

Here i got some log, while run python Test_face_detect.py

Loading model...
2018-12-04 10:26:06.124678: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:864] ARM64 does not support NUMA - returning NUMA node zero
2018-12-04 10:26:06.124833: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1392] Found device 0 with properties: 
name: NVIDIA Tegra X2 major: 6 minor: 2 memoryClockRate(GHz): 1.3005
pciBusID: 0000:00:00.0
totalMemory: 7.66GiB freeMemory: 1.12GiB
2018-12-04 10:26:06.124903: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2018-12-04 10:26:07.977726: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-12-04 10:26:07.977868: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958]      0 
2018-12-04 10:26:07.977943: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0:   N 
2018-12-04 10:26:07.978146: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 455 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)
Model loaded
Loading MTCNN Face detection model
2018-12-04 10:26:26.799989: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2018-12-04 10:26:26.800598: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-12-04 10:26:26.800796: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958]      0 
2018-12-04 10:26:26.801037: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0:   N 
2018-12-04 10:26:26.801456: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 455 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)
WARNING:tensorflow:From /home/nvidia/Face_Recognition_updated/mtcnn_detect.py:329: calling reduce_max (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From /home/nvidia/Face_Recognition_updated/mtcnn_detect.py:331: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
MTCNN Model loaded
[INFO] Starting up...
VIDEOIO ERROR: V4L: device nvcamerasrc sensor-id=0 ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink: Unable to query number of channels

Available Sensor modes : 
2592 x 1944 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
2592 x 1458 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1280 x 720 FR=120.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10

NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 1 WxH = 2592x1458 FrameRate = 30.000000 ...

Please give me any suggestion i don’t know how would i improve;

dusty_nv · February 29, 2020, 1:23am

Hi varunrv50e, you may need to use TensorRT backend for TensorFlow to achieve improved performance. Please refer to these examples:

Varun_Rathinam · February 1, 2020, 7:15am

Hey Hi, Thanks for the reply, i can able to use the sample tf_trt_models. can you give me more elaporate how to use facenet, mtcnn model to we have trained model to convert frozen graph to run on tensor-RT backend engine. I’m completely new to this TensorRT needs some help for our use case. Please give me some suggestions.

naisy · February 1, 2020, 7:16am

I cut out the contours of the face with ssd_mobilenet_v1 and performed facial expression classification with the keras xception model.
Because the expression classification is slow, the speed varies greatly depending on the number of detected faces, but the average is more than 10 FPS on TX2.

The point of speedup is to reduce the input image size. I think that 1280x720 is enough. (Still it is big.)

Accuracy of facial expression classification has changed greatly depending on how to make learning data. It will be better to create data suitable for cutting SSD.

Varun_Rathinam · February 29, 2020, 1:58am

Hey Hi, Thanks for the response. When i try to optimize graph with help of TensorRT on jetson tx2, i got some error at the end of creating inference graph, its going to killed state. I dont know how to resolve this issue, pls give me some suggestion;

f! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:11.504566: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:11.794660: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:12.547723: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:12.828665: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:13.550364: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:13.840069: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:14.453139: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:14.743532: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:15.497577: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:15.778241: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:17.050586: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:17.347970: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:17.968009: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:18.258632: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:18.762323: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:19.052575: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:19.557068: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:19.846993: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:20.459728: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:20.749851: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:21.283185: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:21.564138: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:22.319042: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:22.600504: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:23.213135: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:23.503008: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:24.226531: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:24.516705: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:25.025812: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:25.321041: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:25.889777: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:26.194687: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:27.010231: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:27.309769: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:27.809845: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:28.100460: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:28.806633: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:29.098834: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:29.739406: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:30.020943: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:30.595564: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:30.880356: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:31.573542: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:31.864970: W tensorflow/contrib/tensorrt/convert/trt_optimization_pass.cc:185] TensorRTOptimizer is probably called on funcdef! This optimizer must *NOT* be called on function objects.
2018-12-11 11:26:32.399643: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:404] Optimization results for grappler item: tf_graph
2018-12-11 11:26:32.399875: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   constant folding: Graph size after: 5412 nodes (-491), 9282 edges (-492), time = 1598.47302ms.
2018-12-11 11:26:32.399969: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   layout: Graph size after: 5443 nodes (31), 9293 edges (11), time = 851.81ms.
2018-12-11 11:26:32.399994: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   TensorRTOptimizer: Graph size after: 4967 nodes (-476), 8817 edges (-476), time = 165193.406ms.
2018-12-11 11:26:32.400022: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   constant folding: Graph size after: 4944 nodes (-23), 8817 edges (0), time = 1195.16895ms.
2018-12-11 11:26:32.400043: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   TensorRTOptimizer: Graph size after: 4944 nodes (0), 8817 edges (0), time = 2556.87598ms.
2018-12-11 11:26:32.400066: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:404] Optimization results for grappler item: InceptionResnetV1/Repeat/block35_4/Branch_1/my_trt_op_89_native_segment
2018-12-11 11:26:32.400095: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   constant folding: Graph size after: 4 nodes (0), 3 edges (0), time = 248.235ms.
2018-12-11 11:26:32.400118: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   layout: Graph size after: 4 nodes (0), 3 edges (0), time = 132.236ms.
2018-12-11 11:26:32.400136: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   TensorRTOptimizer: Graph size after: 4 nodes (0), 3 edges (0), time = 33.856ms.
2018-12-11 11:26:32.400155: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   constant folding: Graph size after: 4 nodes (0), 3 edges (0), time = 243.517ms.
2018-12-11 11:26:32.400183: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   TensorRTOptimizer: Graph size after: 4 nodes (0), 3 edges (0), time = 33.689ms.
2018-12-11 11:26:32.400201: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:404] Optimization results for grappler item: InceptionResnetV1/Repeat_1/block17_5/Branch_1/my_trt_op_150_native_segment
2018-12-11 11:26:32.400219: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   constant folding: Graph size after: 4 nodes (0), 3 edges (0), time = 235.774ms.
2018-12-11 11:26:32.400236: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   layout: Graph size after: 4 nodes (0), 3 edges (0), time = 106.326ms.
2018-12-11 11:26:32.400253: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   TensorRTOptimizer: Graph size after: 4 nodes (0), 3 edges (0), time = 33.993ms.
2018-12-11 11:26:32.400270: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   constant folding: Graph size after: 4 nodes (0), 3 edges (0), time = 251.653ms.
2018-12-11 11:26:32.400287: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   TensorRTOptimizer: Graph size after: 4 nodes (0), 3 edges (0), time = 33.934ms.
2018-12-11 11:26:32.400304: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:404] Optimization results for grappler item: InceptionResnetV1/my_trt_op_20_native_segment
2018-12-11 11:26:32.400323: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   constant folding: Graph size after: 5 nodes (0), 4 edges (0), time = 240.373ms.
2018-12-11 11:26:32.400340: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   layout: Graph size after: 5 nodes (0), 4 edges (0), time = 132.394ms.
2018-12-11 11:26:32.400365: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   TensorRTOptimizer: Graph size after: 5 nodes (0), 4 edges (0), time = 33.741ms.
2018-12-11 11:26:32.400383: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   constant folding: Graph size after: 5 nodes (0), 4 edges (0), time = 243.42ms.
2018-12-11 11:26:32.400402: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   TensorRTOptimizer: Graph size after: 5 nodes (0), 4 edges (0), time = 33.685ms.
2018-12-11 11:26:32.400419: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:404] Optimization results for grappler item: InceptionResnetV1/Repeat_1/block17_7/Branch_1/my_trt_op_168_native_segment
2018-12-11 11:26:32.400436: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   constant folding: Graph size after: 4 nodes (0), 3 edges (0), time = 235.648ms.
2018-12-11 11:26:32.400454: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   layout: Graph size after: 4 nodes (0), 3 edges (0), time = 106.342ms.
2018-12-11 11:26:32.400471: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   TensorRTOptimizer: Graph size after: 4 nodes (0), 3 edges (0), time = 34.013ms.
2018-12-11 11:26:32.400490: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   constant folding: Graph size after: 4 nodes (0), 3 edges (0), time = 251.178ms.
2018-12-11 11:26:32.400509: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   TensorRTOptimizer: Graph size after: 4 nodes (0), 3 edges (0), time = 34.093ms.
2018-12-11 11:26:32.400529: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:404] Optimization results for grappler item: InceptionResnetV1/Mixed_7a/Branch_2/my_trt_op_47_native_segment
2018-12-11 11:26:32.400547: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   constant folding: Graph size after: 4 nodes (0), 3 edges (0), time = 240.875ms.
2018-12-11 11:26:32.400564: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   layout: Graph size after: 4 nodes (0), 3 edges (0), time = 108.3ms.
2018-12-11 11:26:32.400581: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   TensorRTOptimizer: Graph size after: 4 nodes (0), 3 edges (0), time = 34.529ms.
2018-12-11 11:26:32.400599: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   constant folding: Graph size after: 4 nodes (0), 3 edges (0), time = 257.974ms.
2018-12-11 11:26:32.400616: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   TensorRTOptimizer: Graph size after: 4 nodes (0), 3 edges (0), time = 34.39ms.
2018-12-11 11:26:32.400632: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:404] Optimization results for grappler item: InceptionResnetV1/my_trt_op_53_native_segment
2018-12-11 11:26:32.400649: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   constant folding: Graph size after: 8 nodes (0), 7 edges (0), time = 241.521ms.
2018-12-11 11:26:32.400666: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   layout: Graph size after: 8 nodes (0), 7 edges (0), time = 132.799ms.
2018-12-11 11:26:32.400682: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   TensorRTOptimizer: Graph size after: 8 nodes (0), 7 edges (0), time = 34.171ms.
2018-12-11 11:26:32.400699: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   constant folding: Graph size after: 8 nodes (0), 7 edges (0), time = 244.142ms.
2018-12-11 11:26:32.400716: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   TensorRTOptimizer: Graph size after: 8 nodes (0), 7 edges (0), time = 33.802ms.
2018-12-11 11:26:32.400733: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:404] Optimization results for grappler item: InceptionResnetV1/Repeat/my_trt_op_84_native_segment
2018-12-11 11:26:32.400750: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   constant folding: Graph size after: 23 nodes (0), 22 edges (0), time = 243.283ms.
2018-12-11 11:26:32.400768: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   layout: Invalid argument: The graph is already optimized by layout optimizer.
2018-12-11 11:26:32.400785: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   TensorRTOptimizer: Graph size after: 23 nodes (0), 22 edges (0), time = 35.72ms.
2018-12-11 11:26:32.400802: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   constant folding: Graph size after: 23 nodes (0), 22 edges (0), time = 246.508ms.
2018-12-11 11:26:32.400819: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   TensorRTOptimizer: Graph size after: 23 nodes (0), 22 edges (0), time = 35.759ms.
2018-12-11 11:26:32.400835: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:404] Optimization results for grappler item: InceptionResnetV1/Repeat_2/my_trt_op_216_native_segment
2018-12-11 11:26:32.400853: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   constant folding: Graph size after: 19 nodes (0), 18 edges (0), time = 252.621ms.
2018-12-11 11:26:32.400870: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   layout: Invalid argument: The graph is already optimized by layout optimizer.
2018-12-11 11:26:32.400888: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   TensorRTOptimizer: Graph size after: 19 nodes (0), 18 edges (0), time = 37.182ms.
2018-12-11 11:26:32.400905: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   constant folding: Graph size after: 19 nodes (0), 18 edges (0), time = 264.772ms.
2018-12-11 11:26:32.400923: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   TensorRTOptimizer: Graph size after: 19 nodes (0), 18 edges (0), time = 37.311ms.
2018-12-11 11:26:32.400940: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:404] Optimization results for grappler item: InceptionResnetV1/Repeat_2/block8_3/Branch_1/my_trt_op_205_native_segment
2018-12-11 11:26:32.400958: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   constant folding: Graph size after: 4 nodes (0), 3 edges (0), time = 235.927ms.
2018-12-11 11:26:32.400978: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   layout: Graph size after: 4 nodes (0), 3 edges (0), time = 106.613ms.
2018-12-11 11:26:32.400997: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:406]   TensorRTOptimizer: Graph size after: 4 nodes (0), 3 edges (0), time = 34.167ms.
Killed