[u]I am benchmarking TF-TRT with V100 on inception-resnetv2. As shown below the perf. is reducing while the iterations are increasing. Also the temperature of the gpu raises to 85C. I followed the instructions given here https://github.com/tensorflow/tensorrt/tree/master/tftrt/examples/image-classification using the container nvcr.io/nvidia/tensorflow:19.05-py3
Please suggest the reason for low perf.
[/u]
python -u /workspace/nvidia-examples/tensorrt/tftrt/examples/image-classification/image_classification.py --model inception_resnet_v2 --use_synthetic --model_dir /trt/ckpt/inception_resnet_v2 --use_trt --engine_dir /trt/engine_dir/ --precision fp32 --batch_size 128 --num_warmup_iterations 10 --cache --mode benchmark --num_iterations 1000
Loading cached frozen graph from ‘/workspace/graphs/frozen_graph_inception_resnet_v2_1_fp32_128.pb’
batch_size: 128
cache: True
calib_data_dir: None
data_dir: None
default_models_dir: ./data
display_every: 100
engine_dir: /trt/engine_dir/
frozen_graph: None
max_workspace_size: 4294967296
minimum_segment_size: 2
mode: benchmark
model: inception_resnet_v2
model_dir: /trt/ckpt/inception_resnet_v2
num_calib_inputs: 500
num_iterations: 1000
num_warmup_iterations: 10
precision: fp32
target_duration: None
use_synthetic: True
use_trt: True
use_trt_dynamic_op: False
url: http://download.tensorflow.org/models/inception_resnet_v2_2016_08_30.tar.gz
num_nodes(loaded_frozen_graph): 18
num_nodes(trt_only): 2
graph_size(MB)(loaded_frozen_graph): 437.0
time(s)(loading_frozen_graph): 1.7
running inference…
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING: Logging before flag parsing goes to stderr.
W0628 17:42:37.603240 140165592819456 deprecation.py:323] From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:Input graph does not use tf.data.Dataset or contain a QueueRunner. That means predict yields forever. This is probably a mistake.
W0628 17:42:39.576087 140165592819456 estimator.py:974] Input graph does not use tf.data.Dataset or contain a QueueRunner. That means predict yields forever. This is probably a mistake.
2019-06-28 17:42:42.809517: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2500000000 Hz
2019-06-28 17:42:42.830048: I tensorflow/compiler/xla/service/service.cc:161] XLA service 0x1fe74e40 executing computations on platform Host. Devices:
2019-06-28 17:42:42.830119: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (0): ,
2019-06-28 17:42:43.034954: I tensorflow/compiler/xla/service/service.cc:161] XLA service 0x1a81a540 executing computations on platform CUDA. Devices:
2019-06-28 17:42:43.035074: I tensorflow/compiler/xla/service/service.cc:168] StreamExecutor device (0): Tesla V100-PCIE-16GB, Compute Capability 7.0
2019-06-28 17:42:43.035428: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 0000:25:00.0
totalMemory: 15.75GiB freeMemory: 15.44GiB
2019-06-28 17:42:43.035475: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-06-28 17:42:43.743604: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-06-28 17:42:43.743661: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-06-28 17:42:43.743668: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-06-28 17:42:43.743820: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14859 MB memory) -> physical GPU (device: 0, name: Tesla V100-PCIE-16GB, pci bus id: 0000:25:00.0, compute capability: 7.0)
[b] step 100/1000, iter_time(ms)=270.2160, images/sec=473
step 200/1000, iter_time(ms)=485.6253, images/sec=263
step 300/1000, iter_time(ms)=713.1963, images/sec=179
step 400/1000, iter_time(ms)=696.6727, images/sec=183
step 500/1000, iter_time(ms)=635.2487, images/sec=201
step 600/1000, iter_time(ms)=678.3783, images/sec=188
step 700/1000, iter_time(ms)=1127.5783, images/sec=113
step 800/1000, iter_time(ms)=617.7261, images/sec=207
step 900/1000, iter_time(ms)=666.3108, images/sec=192
step 1000/1000, iter_time(ms)=650.7771, images/sec=196
results of inception_resnet_v2:
images/sec: 229
99th_percentile(ms): 1340.82
total_time(s): 660.9
latency_mean(ms): 667.57
latency_median(ms): 651.52
latency_min(ms): 265.40[/b]