pcp@pcpall:~$ sudo su [sudo] password for pcp: root@pcpall:/home/pcp# source activate tensorflow (tensorflow) root@pcpall:/home/pcp# cd ~/benchmarks/scripts/tf_cnn_benchmarks/ (tensorflow) root@pcpall:~/benchmarks/scripts/tf_cnn_benchmarks# python tf_cnn_benchmarks.py --num_gpus=1 --batch_size=64 --model=resnet50 --variable_update=independent --local_parameter_device=gpu WARNING: Logging before flag parsing goes to stderr. W0807 13:39:39.612030 139651777775360 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/models/mobilenet.py:388: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead. W0807 13:39:39.616169 139651777775360 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/models/experimental/deepspeech.py:131: The name tf.nn.rnn_cell.RNNCell is deprecated. Please use tf.compat.v1.nn.rnn_cell.RNNCell instead. W0807 13:39:39.638874 139651777775360 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:737: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead. W0807 13:39:39.639043 139651777775360 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:3531: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead. 2019-08-07 13:39:39.640208: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA 2019-08-07 13:39:39.655133: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3597710000 Hz 2019-08-07 13:39:39.655750: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5574c952e760 executing computations on platform Host. Devices: 2019-08-07 13:39:39.655967: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): , 2019-08-07 13:39:39.665608: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1 2019-08-07 13:39:39.764563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.755 pciBusID: 0000:06:00.0 2019-08-07 13:39:39.766096: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1 2019-08-07 13:39:39.795572: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10 2019-08-07 13:39:39.812319: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10 2019-08-07 13:39:39.816423: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10 2019-08-07 13:39:39.848185: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10 2019-08-07 13:39:39.853200: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10 2019-08-07 13:39:39.910714: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7 2019-08-07 13:39:39.911876: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0 2019-08-07 13:39:39.912297: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1 2019-08-07 13:39:39.998447: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-08-07 13:39:39.998473: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0 2019-08-07 13:39:39.998480: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N 2019-08-07 13:39:40.000193: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10097 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:06:00.0, compute capability: 7.5) 2019-08-07 13:39:40.002204: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5574ca355080 executing computations on platform CUDA. Devices: 2019-08-07 13:39:40.002219: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5 TensorFlow: 1.14 Model: resnet50 Dataset: imagenet (synthetic) Mode: training SingleSess: False Batch size: 64 global 64 per device Num batches: 100 Num epochs: 0.00 Devices: ['/gpu:0'] NUMA bind: False Data format: NCHW Optimizer: sgd Variables: independent ========== W0807 13:39:40.004287 139651777775360 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2790: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. Generating training model W0807 13:39:40.004661 139651777775360 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2810: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead. W0807 13:39:40.009546 139651777775360 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/variable_mgr.py:183: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead. W0807 13:39:40.009781 139651777775360 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:245: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead. W0807 13:39:40.021963 139651777775360 deprecation.py:323] From /root/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:129: conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.keras.layers.Conv2D` instead. W0807 13:39:40.281114 139651777775360 deprecation.py:323] From /root/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:261: max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.MaxPooling2D instead. W0807 13:39:41.953501 139651777775360 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:334: The name tf.nn.xw_plus_b is deprecated. Please use tf.compat.v1.nn.xw_plus_b instead. W0807 13:39:41.954940 139651777775360 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/models/model.py:319: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead. W0807 13:39:41.963033 139651777775360 deprecation.py:323] From /root/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where W0807 13:39:42.488007 139651777775360 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/models/resnet_model.py:333: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead. W0807 13:39:42.546348 139651777775360 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:1237: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead. W0807 13:39:42.651384 139651777775360 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2118: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead. W0807 13:39:42.654066 139651777775360 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2154: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead. Initializing graph W0807 13:39:42.654218 139651777775360 deprecation_wrapper.py:119] From /root/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2211: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead. W0807 13:39:42.824316 139651777775360 deprecation.py:323] From /root/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2266: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.MonitoredTrainingSession 2019-08-07 13:39:43.123536: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.755 pciBusID: 0000:06:00.0 2019-08-07 13:39:43.123602: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1 2019-08-07 13:39:43.123615: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10 2019-08-07 13:39:43.123625: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10 2019-08-07 13:39:43.123635: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10 2019-08-07 13:39:43.123644: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10 2019-08-07 13:39:43.123654: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10 2019-08-07 13:39:43.123664: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7 2019-08-07 13:39:43.124689: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0 2019-08-07 13:39:43.124715: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-08-07 13:39:43.124722: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0 2019-08-07 13:39:43.124728: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N 2019-08-07 13:39:43.125786: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10097 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:06:00.0, compute capability: 7.5) 2019-08-07 13:39:43.564024: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile. I0807 13:39:44.914202 139651777775360 session_manager.py:500] Running local_init_op. I0807 13:39:44.954630 139651777775360 session_manager.py:502] Done running local_init_op. Running warm up 2019-08-07 13:39:46.150692: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10 2019-08-07 13:39:46.543327: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7 Done warm up Step Img/sec total_loss 1 images/sec: 269.2 +/- 0.0 (jitter = 0.0) 8.220 10 images/sec: 266.4 +/- 1.1 (jitter = 4.6) 7.880 20 images/sec: 266.7 +/- 0.8 (jitter = 5.1) 7.910 30 images/sec: 268.0 +/- 0.6 (jitter = 3.4) 7.821 40 images/sec: 267.5 +/- 0.6 (jitter = 4.3) 8.005 50 images/sec: 267.4 +/- 0.5 (jitter = 4.9) 7.769 60 images/sec: 267.8 +/- 0.5 (jitter = 3.7) 8.114 70 images/sec: 268.2 +/- 0.4 (jitter = 2.8) 7.817 80 images/sec: 268.6 +/- 0.4 (jitter = 1.5) 7.979 90 images/sec: 268.6 +/- 0.4 (jitter = 2.2) 8.101 100 images/sec: 268.6 +/- 0.3 (jitter = 2.1) 8.027 ---------------------------------------------------------------- total images/sec: 268.50 ---------------------------------------------------------------- (tensorflow) root@pcpall:~/benchmarks/scripts/tf_cnn_benchmarks#