2023-05-25 18:37:15.997541: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2023-05-25 18:37:15.997537: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them. WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them. 2023-05-25 18:37:18.919354: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2023-05-25 18:37:18.922121: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 /usr/local/lib/python3.6/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn't match a supported version! RequestsDependencyWarning) Using TensorFlow backend. 2023-05-25 18:37:18,971 [INFO] __main__: Loading experiment spec at /data/e1.cfg. /usr/local/lib/python3.6/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn't match a supported version! RequestsDependencyWarning) Using TensorFlow backend. 2023-05-25 18:37:18,972 [INFO] __main__: Loading experiment spec at /data/e1.cfg. 2023-05-25 18:37:18.994864: I tensorflow/core/platform/profile_utils/cpu_utils.cc:109] CPU Frequency: 2700000000 Hz 2023-05-25 18:37:18.994863: I tensorflow/core/platform/profile_utils/cpu_utils.cc:109] CPU Frequency: 2700000000 Hz 2023-05-25 18:37:18.995036: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x37b2000 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2023-05-25 18:37:18.995048: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5e6a830 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2023-05-25 18:37:18.995060: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2023-05-25 18:37:18.995070: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2023-05-25 18:37:18.998203: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1 2023-05-25 18:37:18.998203: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1 2023-05-25 18:37:20.545987: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x61bf030 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2023-05-25 18:37:20.546027: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla V100-SXM3-32GB, Compute Capability 7.0 2023-05-25 18:37:20.548078: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1669] Found device 0 with properties: name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.597 pciBusID: 0000:36:00.0 2023-05-25 18:37:20.548126: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2023-05-25 18:37:20.552504: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5d99e50 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2023-05-25 18:37:20.552542: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla V100-SXM3-32GB, Compute Capability 7.0 2023-05-25 18:37:20.553974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1669] Found device 0 with properties: name: Tesla V100-SXM3-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.597 pciBusID: 0000:34:00.0 2023-05-25 18:37:20.554016: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2023-05-25 18:37:20.578573: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11 2023-05-25 18:37:20.582703: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2023-05-25 18:37:20.583137: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2023-05-25 18:37:20.584171: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11 2023-05-25 18:37:20.585269: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11 2023-05-25 18:37:20.585487: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2023-05-25 18:37:20.587942: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1797] Adding visible gpu devices: 1 2023-05-25 18:37:20.590306: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11 2023-05-25 18:37:20.594941: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2023-05-25 18:37:20.595436: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2023-05-25 18:37:20.596619: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11 2023-05-25 18:37:20.598016: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11 2023-05-25 18:37:20.598279: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2023-05-25 18:37:20.602831: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1797] Adding visible gpu devices: 0 2023-05-25 18:37:21.088690: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1209] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-05-25 18:37:21.088740: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1215] 1 2023-05-25 18:37:21.088748: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1228] 1: N 2023-05-25 18:37:21.091174: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1354] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30999 MB memory) -> physical GPU (device: 1, name: Tesla V100-SXM3-32GB, pci bus id: 0000:36:00.0, compute capability: 7.0) 2023-05-25 18:37:21,092 [INFO] __main__: Custom image mean [103.9000015258789, 116.80000305175781, 123.69999694824219] will be used. 2023-05-25 18:37:21.198067: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1209] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-05-25 18:37:21.198125: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1215] 0 2023-05-25 18:37:21.198136: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1228] 0: N 2023-05-25 18:37:21.200893: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1354] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 25477 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM3-32GB, pci bus id: 0000:34:00.0, compute capability: 7.0) 2023-05-25 18:37:21,202 [INFO] __main__: Custom image mean [103.9000015258789, 116.80000305175781, 123.69999694824219] will be used. Found 312 images belonging to 12 classes. Found 312 images belonging to 12 classes. 2023-05-25 18:37:21,439 [INFO] __main__: Processing dataset (train): /data/train 2023-05-25 18:37:21,451 [INFO] __main__: Processing dataset (train): /data/train Found 312 images belonging to 12 classes. Found 312 images belonging to 12 classes. 2023-05-25 18:37:21,568 [INFO] __main__: Processing dataset (validation): /data/train WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:74: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. 2023-05-25 18:37:21,568 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:74: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. 2023-05-25 18:37:21,568 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. 2023-05-25 18:37:21,568 [INFO] __main__: Processing dataset (validation): /data/train WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:74: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. 2023-05-25 18:37:21,569 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:74: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. 2023-05-25 18:37:21,569 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. 2023-05-25 18:37:21,570 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. 2023-05-25 18:37:21,571 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. 2023-05-25 18:37:21,586 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. 2023-05-25 18:37:21,589 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead. 2023-05-25 18:37:21,592 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead. 2023-05-25 18:37:21,595 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/third_party/keras/tensorflow_backend.py:183: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead. 2023-05-25 18:37:21,608 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/third_party/keras/tensorflow_backend.py:183: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/third_party/keras/tensorflow_backend.py:183: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead. 2023-05-25 18:37:21,614 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/third_party/keras/tensorflow_backend.py:183: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/third_party/keras/tensorflow_backend.py:187: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead. 2023-05-25 18:37:21,946 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/third_party/keras/tensorflow_backend.py:187: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/third_party/keras/tensorflow_backend.py:187: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead. 2023-05-25 18:37:22,001 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/third_party/keras/tensorflow_backend.py:187: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. 2023-05-25 18:37:22,373 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. 2023-05-25 18:37:22,373 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. 2023-05-25 18:37:22,373 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. 2023-05-25 18:37:22,490 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. 2023-05-25 18:37:22,490 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. 2023-05-25 18:37:22,490 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. 2023-05-25 18:37:22,637 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. 2023-05-25 18:37:22,818 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. __________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to ================================================================================================== input_1 (InputLayer) (None, 3, 224, 224) 0 __________________________________________________________________________________________________ conv1 (Conv2D) (None, 64, 112, 112) 9408 input_1[0][0] __________________________________________________________________________________________________ bn_conv1 (BatchNormalization) (None, 64, 112, 112) 256 conv1[0][0] __________________________________________________________________________________________________ activation_1 (Activation) (None, 64, 112, 112) 0 bn_conv1[0][0] __________________________________________________________________________________________________ max_pooling2d_1 (MaxPooling2D) (None, 64, 56, 56) 0 activation_1[0][0] __________________________________________________________________________________________________ block_1a_conv_1 (Conv2D) (None, 64, 56, 56) 36864 max_pooling2d_1[0][0] __________________________________________________________________________________________________ block_1a_bn_1 (BatchNormalizati (None, 64, 56, 56) 256 block_1a_conv_1[0][0] __________________________________________________________________________________________________ block_1a_relu_1 (Activation) (None, 64, 56, 56) 0 block_1a_bn_1[0][0] __________________________________________________________________________________________________ block_1a_conv_2 (Conv2D) (None, 64, 56, 56) 36864 block_1a_relu_1[0][0] __________________________________________________________________________________________________ block_1a_conv_shortcut (Conv2D) (None, 64, 56, 56) 4096 max_pooling2d_1[0][0] __________________________________________________________________________________________________ block_1a_bn_2 (BatchNormalizati (None, 64, 56, 56) 256 block_1a_conv_2[0][0] __________________________________________________________________________________________________ block_1a_bn_shortcut (BatchNorm (None, 64, 56, 56) 256 block_1a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_1 (Add) (None, 64, 56, 56) 0 block_1a_bn_2[0][0] block_1a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_1a_relu (Activation) (None, 64, 56, 56) 0 add_1[0][0] __________________________________________________________________________________________________ block_2a_conv_1 (Conv2D) (None, 128, 28, 28) 73728 block_1a_relu[0][0] __________________________________________________________________________________________________ block_2a_bn_1 (BatchNormalizati (None, 128, 28, 28) 512 block_2a_conv_1[0][0] __________________________________________________________________________________________________ block_2a_relu_1 (Activation) (None, 128, 28, 28) 0 block_2a_bn_1[0][0] __________________________________________________________________________________________________ block_2a_conv_2 (Conv2D) (None, 128, 28, 28) 147456 block_2a_relu_1[0][0] __________________________________________________________________________________________________ block_2a_conv_shortcut (Conv2D) (None, 128, 28, 28) 8192 block_1a_relu[0][0] __________________________________________________________________________________________________ block_2a_bn_2 (BatchNormalizati (None, 128, 28, 28) 512 block_2a_conv_2[0][0] __________________________________________________________________________________________________ block_2a_bn_shortcut (BatchNorm (None, 128, 28, 28) 512 block_2a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_2 (Add) (None, 128, 28, 28) 0 block_2a_bn_2[0][0] block_2a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_2a_relu (Activation) (None, 128, 28, 28) 0 add_2[0][0] __________________________________________________________________________________________________ block_3a_conv_1 (Conv2D) (None, 256, 14, 14) 294912 block_2a_relu[0][0] __________________________________________________________________________________________________ block_3a_bn_1 (BatchNormalizati (None, 256, 14, 14) 1024 block_3a_conv_1[0][0] __________________________________________________________________________________________________ block_3a_relu_1 (Activation) (None, 256, 14, 14) 0 block_3a_bn_1[0][0] __________________________________________________________________________________________________ block_3a_conv_2 (Conv2D) (None, 256, 14, 14) 589824 block_3a_relu_1[0][0] __________________________________________________________________________________________________ block_3a_conv_shortcut (Conv2D) (None, 256, 14, 14) 32768 block_2a_relu[0][0] __________________________________________________________________________________________________ block_3a_bn_2 (BatchNormalizati (None, 256, 14, 14) 1024 block_3a_conv_2[0][0] __________________________________________________________________________________________________ block_3a_bn_shortcut (BatchNorm (None, 256, 14, 14) 1024 block_3a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_3 (Add) (None, 256, 14, 14) 0 block_3a_bn_2[0][0] block_3a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_3a_relu (Activation) (None, 256, 14, 14) 0 add_3[0][0] __________________________________________________________________________________________________ block_4a_conv_1 (Conv2D) (None, 512, 7, 7) 1179648 block_3a_relu[0][0] __________________________________________________________________________________________________ block_4a_bn_1 (BatchNormalizati (None, 512, 7, 7) 2048 block_4a_conv_1[0][0] __________________________________________________________________________________________________ block_4a_relu_1 (Activation) (None, 512, 7, 7) 0 block_4a_bn_1[0][0] __________________________________________________________________________________________________ block_4a_conv_2 (Conv2D) (None, 512, 7, 7) 2359296 block_4a_relu_1[0][0] __________________________________________________________________________________________________ block_4a_conv_shortcut (Conv2D) (None, 512, 7, 7) 131072 block_3a_relu[0][0] __________________________________________________________________________________________________ block_4a_bn_2 (BatchNormalizati (None, 512, 7, 7) 2048 block_4a_conv_2[0][0] __________________________________________________________________________________________________ block_4a_bn_shortcut (BatchNorm (None, 512, 7, 7) 2048 block_4a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_4 (Add) (None, 512, 7, 7) 0 block_4a_bn_2[0][0] block_4a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_4a_relu (Activation) (None, 512, 7, 7) 0 add_4[0][0] __________________________________________________________________________________________________ average_pooling2d_1 (AveragePoo (None, 512, 1, 1) 0 block_4a_relu[0][0] __________________________________________________________________________________________________ flatten_1 (Flatten) (None, 512) 0 average_pooling2d_1[0][0] __________________________________________________________________________________________________ predictions (Dense) (None, 13) 6669 flatten_1[0][0] ================================================================================================== Total params: 4,922,573 Trainable params: 4,916,685 Non-trainable params: 5,888 __________________________________________________________________________________________________ WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead. 2023-05-25 18:37:27,873 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3295: The name tf.log is deprecated. Please use tf.math.log instead. 2023-05-25 18:37:27,877 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3295: The name tf.log is deprecated. Please use tf.math.log instead. 2023-05-25 18:37:27,892 [INFO] iva.common.logging.logging: Log file already exists at /results/e1/status.json WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead. 2023-05-25 18:37:28,764 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead. 2023-05-25 18:37:28,850 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead. __________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to ================================================================================================== input_1 (InputLayer) (None, 3, 224, 224) 0 __________________________________________________________________________________________________ conv1 (Conv2D) (None, 64, 112, 112) 9408 input_1[0][0] __________________________________________________________________________________________________ bn_conv1 (BatchNormalization) (None, 64, 112, 112) 256 conv1[0][0] __________________________________________________________________________________________________ activation_1 (Activation) (None, 64, 112, 112) 0 bn_conv1[0][0] __________________________________________________________________________________________________ max_pooling2d_1 (MaxPooling2D) (None, 64, 56, 56) 0 activation_1[0][0] __________________________________________________________________________________________________ block_1a_conv_1 (Conv2D) (None, 64, 56, 56) 36864 max_pooling2d_1[0][0] __________________________________________________________________________________________________ block_1a_bn_1 (BatchNormalizati (None, 64, 56, 56) 256 block_1a_conv_1[0][0] __________________________________________________________________________________________________ block_1a_relu_1 (Activation) (None, 64, 56, 56) 0 block_1a_bn_1[0][0] __________________________________________________________________________________________________ block_1a_conv_2 (Conv2D) (None, 64, 56, 56) 36864 block_1a_relu_1[0][0] __________________________________________________________________________________________________ block_1a_conv_shortcut (Conv2D) (None, 64, 56, 56) 4096 max_pooling2d_1[0][0] __________________________________________________________________________________________________ block_1a_bn_2 (BatchNormalizati (None, 64, 56, 56) 256 block_1a_conv_2[0][0] __________________________________________________________________________________________________ block_1a_bn_shortcut (BatchNorm (None, 64, 56, 56) 256 block_1a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_1 (Add) (None, 64, 56, 56) 0 block_1a_bn_2[0][0] block_1a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_1a_relu (Activation) (None, 64, 56, 56) 0 add_1[0][0] __________________________________________________________________________________________________ block_2a_conv_1 (Conv2D) (None, 128, 28, 28) 73728 block_1a_relu[0][0] __________________________________________________________________________________________________ block_2a_bn_1 (BatchNormalizati (None, 128, 28, 28) 512 block_2a_conv_1[0][0] __________________________________________________________________________________________________ block_2a_relu_1 (Activation) (None, 128, 28, 28) 0 block_2a_bn_1[0][0] __________________________________________________________________________________________________ block_2a_conv_2 (Conv2D) (None, 128, 28, 28) 147456 block_2a_relu_1[0][0] __________________________________________________________________________________________________ block_2a_conv_shortcut (Conv2D) (None, 128, 28, 28) 8192 block_1a_relu[0][0] __________________________________________________________________________________________________ block_2a_bn_2 (BatchNormalizati (None, 128, 28, 28) 512 block_2a_conv_2[0][0] __________________________________________________________________________________________________ block_2a_bn_shortcut (BatchNorm (None, 128, 28, 28) 512 block_2a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_2 (Add) (None, 128, 28, 28) 0 block_2a_bn_2[0][0] block_2a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_2a_relu (Activation) (None, 128, 28, 28) 0 add_2[0][0] __________________________________________________________________________________________________ block_3a_conv_1 (Conv2D) (None, 256, 14, 14) 294912 block_2a_relu[0][0] __________________________________________________________________________________________________ block_3a_bn_1 (BatchNormalizati (None, 256, 14, 14) 1024 block_3a_conv_1[0][0] __________________________________________________________________________________________________ block_3a_relu_1 (Activation) (None, 256, 14, 14) 0 block_3a_bn_1[0][0] __________________________________________________________________________________________________ block_3a_conv_2 (Conv2D) (None, 256, 14, 14) 589824 block_3a_relu_1[0][0] __________________________________________________________________________________________________ block_3a_conv_shortcut (Conv2D) (None, 256, 14, 14) 32768 block_2a_relu[0][0] __________________________________________________________________________________________________ block_3a_bn_2 (BatchNormalizati (None, 256, 14, 14) 1024 block_3a_conv_2[0][0] __________________________________________________________________________________________________ block_3a_bn_shortcut (BatchNorm (None, 256, 14, 14) 1024 block_3a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_3 (Add) (None, 256, 14, 14) 0 block_3a_bn_2[0][0] block_3a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_3a_relu (Activation) (None, 256, 14, 14) 0 add_3[0][0] __________________________________________________________________________________________________ block_4a_conv_1 (Conv2D) (None, 512, 7, 7) 1179648 block_3a_relu[0][0] __________________________________________________________________________________________________ block_4a_bn_1 (BatchNormalizati (None, 512, 7, 7) 2048 block_4a_conv_1[0][0] __________________________________________________________________________________________________ block_4a_relu_1 (Activation) (None, 512, 7, 7) 0 block_4a_bn_1[0][0] __________________________________________________________________________________________________ block_4a_conv_2 (Conv2D) (None, 512, 7, 7) 2359296 block_4a_relu_1[0][0] __________________________________________________________________________________________________ block_4a_conv_shortcut (Conv2D) (None, 512, 7, 7) 131072 block_3a_relu[0][0] __________________________________________________________________________________________________ block_4a_bn_2 (BatchNormalizati (None, 512, 7, 7) 2048 block_4a_conv_2[0][0] __________________________________________________________________________________________________ block_4a_bn_shortcut (BatchNorm (None, 512, 7, 7) 2048 block_4a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_4 (Add) (None, 512, 7, 7) 0 block_4a_bn_2[0][0] block_4a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_4a_relu (Activation) (None, 512, 7, 7) 0 add_4[0][0] __________________________________________________________________________________________________ average_pooling2d_1 (AveragePoo (None, 512, 1, 1) 0 block_4a_relu[0][0] __________________________________________________________________________________________________ flatten_1 (Flatten) (None, 512) 0 average_pooling2d_1[0][0] __________________________________________________________________________________________________ predictions (Dense) (None, 13) 6669 flatten_1[0][0] ================================================================================================== Total params: 4,922,573 Trainable params: 4,916,685 Non-trainable params: 5,888 __________________________________________________________________________________________________ WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead. 2023-05-25 18:37:29,629 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3295: The name tf.log is deprecated. Please use tf.math.log instead. 2023-05-25 18:37:29,634 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3295: The name tf.log is deprecated. Please use tf.math.log instead. 2023-05-25 18:37:29,901 [INFO] root: Starting Training Loop. Epoch 1/50 WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead. 2023-05-25 18:37:30,621 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead. 2023-05-25 18:37:30,717 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead. 2023-05-25 18:38:15.928809: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11 2023-05-25 18:38:16.763731: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2023-05-25 18:38:17.461847: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11 2023-05-25 18:38:18.149093: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 f3363b50bf60:218073:218086 [0] NCCL INFO Bootstrap : Using eth0:172.17.0.3<0> f3363b50bf60:218073:218086 [0] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v6 symbol. f3363b50bf60:218073:218086 [0] NCCL INFO NET/Plugin: Loaded net plugin NCCL RDMA Plugin (v5) f3363b50bf60:218073:218086 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol. f3363b50bf60:218073:218086 [0] NCCL INFO NET/Plugin: Loaded coll plugin SHARP (v5) f3363b50bf60:218073:218086 [0] NCCL INFO cudaDriverVersion 11080 NCCL version 2.15.5+cuda11.8 f3363b50bf60:218073:218086 [0] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so f3363b50bf60:218073:218086 [0] NCCL INFO P2P plugin IBext f3363b50bf60:218073:218086 [0] NCCL INFO NET/IB : No device found. f3363b50bf60:218073:218086 [0] NCCL INFO NET/IB : No device found. f3363b50bf60:218073:218086 [0] NCCL INFO NET/Socket : Using [0]eth0:172.17.0.3<0> f3363b50bf60:218073:218086 [0] NCCL INFO Using network Socket f3363b50bf60:218073:218086 [0] NCCL INFO Setting affinity for GPU 0 to 010000,00000001 f3363b50bf60:218073:218086 [0] NCCL INFO Channel 00/12 : 0 1 f3363b50bf60:218073:218086 [0] NCCL INFO Channel 01/12 : 0 1 f3363b50bf60:218073:218086 [0] NCCL INFO Channel 02/12 : 0 1 f3363b50bf60:218073:218086 [0] NCCL INFO Channel 03/12 : 0 1 f3363b50bf60:218073:218086 [0] NCCL INFO Channel 04/12 : 0 1 f3363b50bf60:218073:218086 [0] NCCL INFO Channel 05/12 : 0 1 f3363b50bf60:218073:218086 [0] NCCL INFO Channel 06/12 : 0 1 f3363b50bf60:218073:218086 [0] NCCL INFO Channel 07/12 : 0 1 f3363b50bf60:218073:218086 [0] NCCL INFO Channel 08/12 : 0 1 f3363b50bf60:218073:218086 [0] NCCL INFO Channel 09/12 : 0 1 f3363b50bf60:218073:218086 [0] NCCL INFO Channel 10/12 : 0 1 f3363b50bf60:218073:218086 [0] NCCL INFO Channel 11/12 : 0 1 f3363b50bf60:218073:218086 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 f3363b50bf60:218073:218086 [0] NCCL INFO Channel 00/0 : 0[34000] -> 1[36000] via P2P/IPC f3363b50bf60:218073:218086 [0] NCCL INFO Channel 01/0 : 0[34000] -> 1[36000] via P2P/IPC f3363b50bf60:218073:218086 [0] NCCL INFO Channel 02/0 : 0[34000] -> 1[36000] via P2P/IPC f3363b50bf60:218073:218086 [0] NCCL INFO Channel 03/0 : 0[34000] -> 1[36000] via P2P/IPC f3363b50bf60:218073:218086 [0] NCCL INFO Channel 04/0 : 0[34000] -> 1[36000] via P2P/IPC f3363b50bf60:218073:218086 [0] NCCL INFO Channel 05/0 : 0[34000] -> 1[36000] via P2P/IPC f3363b50bf60:218073:218086 [0] NCCL INFO Channel 06/0 : 0[34000] -> 1[36000] via P2P/IPC f3363b50bf60:218073:218086 [0] NCCL INFO Channel 07/0 : 0[34000] -> 1[36000] via P2P/IPC f3363b50bf60:218073:218086 [0] NCCL INFO Channel 08/0 : 0[34000] -> 1[36000] via P2P/IPC f3363b50bf60:218073:218086 [0] NCCL INFO Channel 09/0 : 0[34000] -> 1[36000] via P2P/IPC f3363b50bf60:218073:218086 [0] NCCL INFO Channel 10/0 : 0[34000] -> 1[36000] via P2P/IPC f3363b50bf60:218073:218086 [0] NCCL INFO Channel 11/0 : 0[34000] -> 1[36000] via P2P/IPC f3363b50bf60:218073:218086 [0] NCCL INFO Connected all rings f3363b50bf60:218073:218086 [0] NCCL INFO Connected all trees f3363b50bf60:218073:218086 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 f3363b50bf60:218073:218086 [0] NCCL INFO 12 coll channels, 16 p2p channels, 16 p2p channels per peer f3363b50bf60:218073:218086 [0] NCCL INFO comm 0x7f9d43932870 rank 0 nranks 2 cudaDev 0 busId 34000 - Init COMPLETE 1/17 [>.............................] - ETA: 14:42 - loss: 2.7433 - acc: 0.0352[f3363b50bf60:218073] Read -1, expected 11865, errno = 1 [f3363b50bf60:218073] Read -1, expected 12889, errno = 1 [f3363b50bf60:218074] Read -1, expected 2359296, errno = 1 [f3363b50bf60:218074] Read -1, expected 589824, errno = 1 [f3363b50bf60:218074] Read -1, expected 32768, errno = 1 [f3363b50bf60:218074] Read -1, expected 147456, errno = 1 [f3363b50bf60:218074] Read -1, expected 20633, errno = 1 [f3363b50bf60:218074] Read -1, expected 147456, errno = 1 [f3363b50bf60:218074] Read -1, expected 294912, errno = 1 [f3363b50bf60:218074] Read -1, expected 37632, errno = 1 [f3363b50bf60:218074] Read -1, expected 16384, errno = 1 [f3363b50bf60:218074] Read -1, expected 2359296, errno = 1 [f3363b50bf60:218074] Read -1, expected 32768, errno = 1 [f3363b50bf60:218074] Read -1, expected 294912, errno = 1 [f3363b50bf60:218074] Read -1, expected 9437184, errno = 1 [f3363b50bf60:218074] Read -1, expected 9437184, errno = 1 [f3363b50bf60:218074] Read -1, expected 16384, errno = 1 [f3363b50bf60:218074] Read -1, expected 4718592, errno = 1 [f3363b50bf60:218074] Read -1, expected 1179648, errno = 1 [f3363b50bf60:218074] Read -1, expected 37632, errno = 1 [f3363b50bf60:218074] Read -1, expected 16384, errno = 1 [f3363b50bf60:218074] Read -1, expected 16384, errno = 1 [f3363b50bf60:218074] Read -1, expected 1179648, errno = 1 [f3363b50bf60:218074] Read -1, expected 37632, errno = 1 [f3363b50bf60:218074] Read -1, expected 1179648, errno = 1 [f3363b50bf60:218074] Read -1, expected 294912, errno = 1 [f3363b50bf60:218074] Read -1, expected 2359296, errno = 1 [f3363b50bf60:218074] Read -1, expected 1179648, errno = 1 [f3363b50bf60:218074] Read -1, expected 131072, errno = 1 [f3363b50bf60:218074] Read -1, expected 147456, errno = 1 [f3363b50bf60:218074] Read -1, expected 9437184, errno = 1 [f3363b50bf60:218074] Read -1, expected 147456, errno = 1 [f3363b50bf60:218074] Read -1, expected 32768, errno = 1 [f3363b50bf60:218074] Read -1, expected 131072, errno = 1 [f3363b50bf60:218074] Read -1, expected 524288, errno = 1 [f3363b50bf60:218074] Read -1, expected 147456, errno = 1 [f3363b50bf60:218074] Read -1, expected 589824, errno = 1 [f3363b50bf60:218074] Read -1, expected 589824, errno = 1 [f3363b50bf60:218074] Read -1, expected 26624, errno = 1 [f3363b50bf60:218074] Read -1, expected 131072, errno = 1 [f3363b50bf60:218074] Read -1, expected 147456, errno = 1 [f3363b50bf60:218074] Read -1, expected 10033, errno = 1 [f3363b50bf60:218074] Read -1, expected 147456, errno = 1 [f3363b50bf60:218074] Read -1, expected 4718592, errno = 1 [f3363b50bf60:218074] Read -1, expected 524288, errno = 1 [f3363b50bf60:218074] Read -1, expected 26624, errno = 1 [f3363b50bf60:218074] Read -1, expected 524288, errno = 1 [f3363b50bf60:218074] Read -1, expected 2359296, errno = 1 [f3363b50bf60:218074] Read -1, expected 360448, errno = 1 [f3363b50bf60:218074] Read -1, expected 4718592, errno = 1 [f3363b50bf60:218074] Read -1, expected 37632, errno = 1 [f3363b50bf60:218074] Read -1, expected 294912, errno = 1 [f3363b50bf60:218074] Read -1, expected 26624, errno = 1 [f3363b50bf60:218074] Read -1, expected 9437184, errno = 1 [f3363b50bf60:218074] Read -1, expected 524288, errno = 1 [f3363b50bf60:218074] Read -1, expected 4718592, errno = 1 [f3363b50bf60:218074] Read -1, expected 131072, errno = 1 [f3363b50bf60:218074] Read -1, expected 147456, errno = 1 [f3363b50bf60:218074] Read -1, expected 589824, errno = 1 [f3363b50bf60:218074] Read -1, expected 32768, errno = 1 17/17 [==============================] - 95s 6s/step - loss: 1.9214 - acc: 0.3771 - val_loss: 1.4459 - val_acc: 0.5417 /usr/local/lib/python3.6/dist-packages/keras/callbacks.py:122: UserWarning: Method on_batch_end() is slow compared to the batch update (0.658656). Check your callbacks. % delta_t_median) 2023-05-25 18:39:24,309 [INFO] root: Training loop in progress Epoch 2/50 17/17 [==============================] - 37s 2s/step - loss: 1.4168 - acc: 0.5594 - val_loss: 1.0241 - val_acc: 0.6807 2023-05-25 18:40:20,998 [INFO] root: Training loop in progress Epoch 3/50 17/17 [==============================] - 36s 2s/step - loss: 1.2612 - acc: 0.6131 - val_loss: 0.9438 - val_acc: 0.7286 2023-05-25 18:41:04,620 [INFO] root: Training loop in progress Epoch 4/50 17/17 [==============================] - 44s 3s/step - loss: 1.1780 - acc: 0.6555 - val_loss: 0.7939 - val_acc: 0.7395 2023-05-25 18:41:55,694 [INFO] root: Training loop in progress Epoch 5/50 17/17 [==============================] - 46s 3s/step - loss: 1.1161 - acc: 0.6659 - val_loss: 0.7370 - val_acc: 0.7733 2023-05-25 18:42:49,296 [INFO] root: Training loop in progress Epoch 6/50 17/17 [==============================] - 49s 3s/step - loss: 1.0519 - acc: 0.7017 - val_loss: 0.6904 - val_acc: 0.7951 2023-05-25 18:43:46,028 [INFO] root: Training loop in progress Epoch 7/50 17/17 [==============================] - 46s 3s/step - loss: 1.0296 - acc: 0.7105 - val_loss: 0.6761 - val_acc: 0.7875 2023-05-25 18:44:39,872 [INFO] root: Training loop in progress Epoch 8/50 17/17 [==============================] - 45s 3s/step - loss: 1.0314 - acc: 0.7084 - val_loss: 0.6622 - val_acc: 0.7913 2023-05-25 18:45:33,385 [INFO] root: Training loop in progress Epoch 9/50 17/17 [==============================] - 49s 3s/step - loss: 0.9882 - acc: 0.7277 - val_loss: 0.6371 - val_acc: 0.7913 2023-05-25 18:46:30,344 [INFO] root: Training loop in progress Epoch 10/50 17/17 [==============================] - 45s 3s/step - loss: 0.9859 - acc: 0.7291 - val_loss: 0.6752 - val_acc: 0.7820 2023-05-25 18:47:24,119 [INFO] root: Training loop in progress Epoch 11/50 17/17 [==============================] - 48s 3s/step - loss: 0.9493 - acc: 0.7424 - val_loss: 0.5688 - val_acc: 0.8207 2023-05-25 18:48:20,412 [INFO] root: Training loop in progress Epoch 12/50 17/17 [==============================] - 44s 3s/step - loss: 0.9420 - acc: 0.7527 - val_loss: 0.6086 - val_acc: 0.8049 2023-05-25 18:49:12,747 [INFO] root: Training loop in progress Epoch 13/50 17/17 [==============================] - 47s 3s/step - loss: 0.9158 - acc: 0.7511 - val_loss: 0.5713 - val_acc: 0.8305 2023-05-25 18:50:08,651 [INFO] root: Training loop in progress Epoch 14/50 17/17 [==============================] - 47s 3s/step - loss: 0.9044 - acc: 0.7604 - val_loss: 0.5503 - val_acc: 0.8305 2023-05-25 18:51:03,087 [INFO] root: Training loop in progress Epoch 15/50 17/17 [==============================] - 47s 3s/step - loss: 0.8985 - acc: 0.7615 - val_loss: 0.5789 - val_acc: 0.8234 2023-05-25 18:51:57,619 [INFO] root: Training loop in progress Epoch 16/50 17/17 [==============================] - 45s 3s/step - loss: 0.8854 - acc: 0.7593 - val_loss: 0.5288 - val_acc: 0.8376 2023-05-25 18:52:51,872 [INFO] root: Training loop in progress Epoch 17/50 17/17 [==============================] - 47s 3s/step - loss: 0.8770 - acc: 0.7700 - val_loss: 0.6018 - val_acc: 0.8120 2023-05-25 18:53:47,423 [INFO] root: Training loop in progress Epoch 18/50 17/17 [==============================] - 46s 3s/step - loss: 0.8810 - acc: 0.7740 - val_loss: 0.5962 - val_acc: 0.8076 2023-05-25 18:54:41,104 [INFO] root: Training loop in progress Epoch 19/50 17/17 [==============================] - 48s 3s/step - loss: 0.8499 - acc: 0.7755 - val_loss: 0.5958 - val_acc: 0.8191 2023-05-25 18:55:36,820 [INFO] root: Training loop in progress Epoch 20/50 17/17 [==============================] - 46s 3s/step - loss: 0.8454 - acc: 0.7847 - val_loss: 0.5189 - val_acc: 0.8518 2023-05-25 18:56:30,706 [INFO] root: Training loop in progress Epoch 21/50 17/17 [==============================] - 48s 3s/step - loss: 0.8365 - acc: 0.7826 - val_loss: 0.5163 - val_acc: 0.8414 2023-05-25 18:57:26,192 [INFO] root: Training loop in progress Epoch 22/50 17/17 [==============================] - 45s 3s/step - loss: 0.8366 - acc: 0.7928 - val_loss: 0.5201 - val_acc: 0.8398 2023-05-25 18:58:20,180 [INFO] root: Training loop in progress Epoch 23/50 17/17 [==============================] - 50s 3s/step - loss: 0.8070 - acc: 0.7987 - val_loss: 0.5197 - val_acc: 0.8458 2023-05-25 18:59:17,740 [INFO] root: Training loop in progress Epoch 24/50 17/17 [==============================] - 44s 3s/step - loss: 0.7974 - acc: 0.8042 - val_loss: 0.4771 - val_acc: 0.8556 2023-05-25 19:00:08,797 [INFO] root: Training loop in progress Epoch 25/50 17/17 [==============================] - 48s 3s/step - loss: 0.8096 - acc: 0.7898 - val_loss: 0.4725 - val_acc: 0.8605 2023-05-25 19:01:06,023 [INFO] root: Training loop in progress Epoch 26/50 17/17 [==============================] - 46s 3s/step - loss: 0.8114 - acc: 0.7969 - val_loss: 0.4840 - val_acc: 0.8583 2023-05-25 19:01:59,781 [INFO] root: Training loop in progress Epoch 27/50 17/17 [==============================] - 48s 3s/step - loss: 0.8081 - acc: 0.8072 - val_loss: 0.5414 - val_acc: 0.8311 2023-05-25 19:02:56,095 [INFO] root: Training loop in progress Epoch 28/50 17/17 [==============================] - 44s 3s/step - loss: 0.7797 - acc: 0.8091 - val_loss: 0.4675 - val_acc: 0.8659 2023-05-25 19:03:48,399 [INFO] root: Training loop in progress Epoch 29/50 17/17 [==============================] - 47s 3s/step - loss: 0.7606 - acc: 0.8201 - val_loss: 0.4652 - val_acc: 0.8627 2023-05-25 19:04:43,121 [INFO] root: Training loop in progress Epoch 30/50 17/17 [==============================] - 49s 3s/step - loss: 0.7799 - acc: 0.8013 - val_loss: 0.4655 - val_acc: 0.8698 2023-05-25 19:05:39,599 [INFO] root: Training loop in progress Epoch 31/50 17/17 [==============================] - 46s 3s/step - loss: 0.7694 - acc: 0.8146 - val_loss: 0.4522 - val_acc: 0.8665 2023-05-25 19:06:34,487 [INFO] root: Training loop in progress Epoch 32/50 17/17 [==============================] - 45s 3s/step - loss: 0.7682 - acc: 0.8196 - val_loss: 0.4411 - val_acc: 0.8681 2023-05-25 19:07:28,245 [INFO] root: Training loop in progress Epoch 33/50 17/17 [==============================] - 46s 3s/step - loss: 0.7659 - acc: 0.8134 - val_loss: 0.4630 - val_acc: 0.8627 2023-05-25 19:08:23,047 [INFO] root: Training loop in progress Epoch 34/50 17/17 [==============================] - 45s 3s/step - loss: 0.7741 - acc: 0.8114 - val_loss: 0.4639 - val_acc: 0.8638 2023-05-25 19:09:16,555 [INFO] root: Training loop in progress Epoch 35/50 17/17 [==============================] - 48s 3s/step - loss: 0.7536 - acc: 0.8173 - val_loss: 0.4365 - val_acc: 0.8796 2023-05-25 19:10:14,035 [INFO] root: Training loop in progress Epoch 36/50 17/17 [==============================] - 45s 3s/step - loss: 0.7424 - acc: 0.8199 - val_loss: 0.4334 - val_acc: 0.8681 2023-05-25 19:11:05,705 [INFO] root: Training loop in progress Epoch 37/50 17/17 [==============================] - 47s 3s/step - loss: 0.7463 - acc: 0.8240 - val_loss: 0.4310 - val_acc: 0.8752 2023-05-25 19:12:01,427 [INFO] root: Training loop in progress Epoch 38/50 17/17 [==============================] - 47s 3s/step - loss: 0.7567 - acc: 0.8304 - val_loss: 0.4336 - val_acc: 0.8687 2023-05-25 19:12:56,135 [INFO] root: Training loop in progress Epoch 39/50 17/17 [==============================] - 46s 3s/step - loss: 0.7289 - acc: 0.8325 - val_loss: 0.4179 - val_acc: 0.8768 2023-05-25 19:13:51,904 [INFO] root: Training loop in progress Epoch 40/50 17/17 [==============================] - 44s 3s/step - loss: 0.7401 - acc: 0.8254 - val_loss: 0.4355 - val_acc: 0.8752 2023-05-25 19:14:44,579 [INFO] root: Training loop in progress Epoch 41/50 17/17 [==============================] - 48s 3s/step - loss: 0.7424 - acc: 0.8290 - val_loss: 0.4272 - val_acc: 0.8708 2023-05-25 19:15:40,444 [INFO] root: Training loop in progress Epoch 42/50 17/17 [==============================] - 45s 3s/step - loss: 0.7520 - acc: 0.8260 - val_loss: 0.4309 - val_acc: 0.8757 2023-05-25 19:16:33,616 [INFO] root: Training loop in progress Epoch 43/50 17/17 [==============================] - 49s 3s/step - loss: 0.7540 - acc: 0.8187 - val_loss: 0.4242 - val_acc: 0.8741 2023-05-25 19:17:30,635 [INFO] root: Training loop in progress Epoch 44/50 17/17 [==============================] - 45s 3s/step - loss: 0.7476 - acc: 0.8276 - val_loss: 0.4223 - val_acc: 0.8747 2023-05-25 19:18:22,839 [INFO] root: Training loop in progress Epoch 45/50 17/17 [==============================] - 47s 3s/step - loss: 0.7333 - acc: 0.8311 - val_loss: 0.4176 - val_acc: 0.8801 2023-05-25 19:19:18,106 [INFO] root: Training loop in progress Epoch 46/50 17/17 [==============================] - 46s 3s/step - loss: 0.7421 - acc: 0.8203 - val_loss: 0.4184 - val_acc: 0.8801 2023-05-25 19:20:12,962 [INFO] root: Training loop in progress Epoch 47/50 17/17 [==============================] - 48s 3s/step - loss: 0.7264 - acc: 0.8279 - val_loss: 0.4172 - val_acc: 0.8790 2023-05-25 19:21:08,567 [INFO] root: Training loop in progress Epoch 48/50 17/17 [==============================] - 45s 3s/step - loss: 0.7319 - acc: 0.8336 - val_loss: 0.4167 - val_acc: 0.8790 2023-05-25 19:22:02,216 [INFO] root: Training loop in progress Epoch 49/50 17/17 [==============================] - 46s 3s/step - loss: 0.7315 - acc: 0.8334 - val_loss: 0.4169 - val_acc: 0.8785 2023-05-25 19:22:56,987 [INFO] root: Training loop in progress Epoch 50/50 17/17 [==============================] - 46s 3s/step - loss: 0.7223 - acc: 0.8361 - val_loss: 0.4195 - val_acc: 0.8774 /usr/local/lib/python3.6/dist-packages/keras/callbacks.py:122: UserWarning: Method on_batch_end() is slow compared to the batch update (0.592189). Check your callbacks. % delta_t_median) 2023-05-25 19:23:44,228 [INFO] root: Final model evaluation in progress. 2023-05-25 19:23:51,870 [INFO] root: Training loop in progress 2023-05-25 19:23:52,092 [INFO] root: Training loop complete. 2023-05-25 19:23:52,093 [INFO] root: Final model evaluation in progress. 2023-05-25 19:24:31,876 [INFO] root: Model evaluation is complete. 2023-05-25 19:24:31,877 [INFO] root: Training finished successfully. 2023-05-25 19:24:31,877 [INFO] root: Model evaluation is complete. 2023-05-25 19:24:31,877 [INFO] __main__: Training finished successfully. 2023-05-25 19:24:31,877 [INFO] __main__: Total Val Loss: 0.4184948205947876 2023-05-25 19:24:31,877 [INFO] __main__: Total Val accuracy: 0.8784741163253784 2023-05-25 19:24:31,877 [INFO] root: Training finished successfully. 2023-05-25 19:24:31,877 [INFO] __main__: Training finished successfully. f3363b50bf60:218073:218086 [0] NCCL INFO comm 0x7f9d43932870 rank 0 nranks 2 cudaDev 0 busId 34000 - Destroy COMPLETE