2019-08-09 15:01:26.286608: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-09 15:01:26.501403: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2019-08-09 15:01:27.053089: I tensorflow/core/grappler/clusters/single_machine.cc:359] Starting new session
2019-08-09 15:01:27.122703: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-09 15:01:27.123840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.4175
pciBusID: 0000:01:00.0
2019-08-09 15:01:28.647443: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-08-09 15:01:28.685479: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-08-09 15:01:28.687044: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2019-08-09 15:01:28.688017: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2019-08-09 15:01:28.722024: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2019-08-09 15:01:28.722931: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2019-08-09 15:01:28.815518: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-08-09 15:01:28.815875: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-09 15:01:28.817142: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-09 15:01:28.818382: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-08-09 15:01:28.895605: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-09 15:01:28.895694: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2019-08-09 15:01:28.895716: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2019-08-09 15:01:28.949736: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-09 15:01:28.950898: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-09 15:01:29.017896: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3127 MB memory) → physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2019-08-09 15:01:55.659202: I tensorflow/compiler/tf2tensorrt/segment/segment.cc:460] There are 186 ops of 33 different types in the graph that are not converted to TensorRT: TensorArrayGatherV3, NextIteration, TensorArrayReadV3, Add, Identity, Switch, LoopCond, Maximum, Range, LogicalAnd, Less, Merge, Enter, Shape, Split, ReverseV2, ExpandDims, TensorArraySizeV3, TensorArrayWriteV3, NoOp, ConcatV2, Minimum, StridedSlice, Mul, Exit, Pack, Fill, TensorArrayScatterV3, Reshape, Transpose, Placeholder, DataFormatVecPermute, TensorArrayV3, (For more information see Accelerating Inference In TF-TRT User Guide :: NVIDIA Deep Learning Frameworks Documentation).
2019-08-09 15:01:55.660907: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:733] Number of TensorRT candidate segments: 6
2019-08-09 15:01:58.474504: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2019-08-09 15:01:59.076592: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:835] TensorRT node BiLSTM/TRTEngineOp_0 added for segment 0 consisting of 4 nodes succeeded.
2019-08-09 15:01:59.129976: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:835] TensorRT node TRTEngineOp_1 added for segment 1 consisting of 11 nodes succeeded.
2019-08-09 15:01:59.132114: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:835] TensorRT node TRTEngineOp_2 added for segment 2 consisting of 11 nodes succeeded.
2019-08-09 15:02:00.992569: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:835] TensorRT node TRTEngineOp_3 added for segment 3 consisting of 86 nodes succeeded.
2019-08-09 15:02:02.226428: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:835] TensorRT node bbox_pred/TRTEngineOp_4 added for segment 4 consisting of 4 nodes succeeded.
2019-08-09 15:02:02.226850: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:835] TensorRT node cls_pred/TRTEngineOp_5 added for segment 5 consisting of 4 nodes succeeded.
2019-08-09 15:02:03.819555: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:716] Optimization results for grappler item: tf_graph
2019-08-09 15:02:03.901357: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 442 nodes (-58), 540 edges (-58), time = 19980.2031ms.
2019-08-09 15:02:03.901465: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] layout: Graph size after: 450 nodes (8), 548 edges (8), time = 2196.81ms.
2019-08-09 15:02:03.901495: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 450 nodes (0), 548 edges (0), time = 98.207ms.
2019-08-09 15:02:03.901521: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] TensorRTOptimizer: Graph size after: 336 nodes (-114), 432 edges (-116), time = 8268.5332ms.
<class ‘float’>
2019-08-09 15:02:15.572465: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:502] layout failed: Invalid argument: The graph is already optimized by layout optimizer.
2019-08-09 15:02:25.925693: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:632] Building a new TensorRT engine for import/TRTEngineOp_3 input shapes: [[1,848,608,1], [1,848,608,1], [1,848,608,1]]
2019-08-09 15:02:28.936286: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2019-08-09 15:02:35.731861: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2019-08-09 15:03:29.544825: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:632] Building a new TensorRT engine for import/TRTEngineOp_2 input shapes: [[53,512], [53,128]]
2019-08-09 15:03:29.544875: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:632] Building a new TensorRT engine for import/TRTEngineOp_1 input shapes: [[53,512], [53,128]]
2019-08-09 15:03:29.648328: E tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1543] Node BiLSTM/bidirectional_rnn/fw/fw/while/lstm_cell/concat should have an input named ‘BiLSTM/bidirectional_rnn/fw/fw/while/Identity_4’ but it is not available
2019-08-09 15:03:29.648451: W tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:647] Engine creation for import/TRTEngineOp_2 failed. The native segment will be used instead. Reason: Invalid argument: Node BiLSTM/bidirectional_rnn/fw/fw/while/lstm_cell/concat should have an input named ‘BiLSTM/bidirectional_rnn/fw/fw/while/Identity_4’ but it is not available
2019-08-09 15:03:29.651762: E tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1543] Node BiLSTM/bidirectional_rnn/bw/bw/while/lstm_cell/concat should have an input named ‘BiLSTM/bidirectional_rnn/bw/bw/while/Identity_4’ but it is not available
2019-08-09 15:03:29.651884: W tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:647] Engine creation for import/TRTEngineOp_1 failed. The native segment will be used instead. Reason: Invalid argument: Node BiLSTM/bidirectional_rnn/bw/bw/while/lstm_cell/concat should have an input named ‘BiLSTM/bidirectional_rnn/bw/bw/while/Identity_4’ but it is not available
2019-08-09 15:03:31.561342: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:632] Building a new TensorRT engine for import/BiLSTM/TRTEngineOp_0 input shapes: [[2014,256]]
2019-08-09 15:03:31.659786: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:632] Building a new TensorRT engine for import/cls_pred/TRTEngineOp_5 input shapes: [[2014,512]]
2019-08-09 15:03:31.674099: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:632] Building a new TensorRT engine for import/bbox_pred/TRTEngineOp_4 input shapes: [[2014,512]]
At most times the error happens in building a tensorrt engine for some of the engines, but sometimes it not occurs error,why this phenomeon