Using TensorFlow backend. 2021-03-06 13:49:53.957919: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2 WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them. /usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) 2021-03-06 13:50:02.379600: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1 2021-03-06 13:50:02.390091: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1049] ARM64 does not support NUMA - returning NUMA node zero 2021-03-06 13:50:02.390236: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1665] Found device 0 with properties: name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216 pciBusID: 0000:00:00.0 2021-03-06 13:50:02.390319: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2 2021-03-06 13:50:02.402197: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10 2021-03-06 13:50:02.421022: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2021-03-06 13:50:02.463471: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2021-03-06 13:50:02.486886: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10 2021-03-06 13:50:02.505680: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10 2021-03-06 13:50:02.512176: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2021-03-06 13:50:02.512413: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1049] ARM64 does not support NUMA - returning NUMA node zero 2021-03-06 13:50:02.512725: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1049] ARM64 does not support NUMA - returning NUMA node zero 2021-03-06 13:50:02.512838: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1793] Adding visible gpu devices: 0 2021-03-06 13:50:02.536383: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency 2021-03-06 13:50:02.536956: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x13920c20 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2021-03-06 13:50:02.537014: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2021-03-06 13:50:02.706504: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1049] ARM64 does not support NUMA - returning NUMA node zero 2021-03-06 13:50:02.706863: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x14bdf6a0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2021-03-06 13:50:02.706967: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3 2021-03-06 13:50:02.707283: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1049] ARM64 does not support NUMA - returning NUMA node zero 2021-03-06 13:50:02.707414: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1665] Found device 0 with properties: name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216 pciBusID: 0000:00:00.0 2021-03-06 13:50:02.707563: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2 2021-03-06 13:50:02.707678: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10 2021-03-06 13:50:02.707784: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2021-03-06 13:50:02.707901: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2021-03-06 13:50:02.708030: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10 2021-03-06 13:50:02.708160: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10 2021-03-06 13:50:02.708294: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2021-03-06 13:50:02.708459: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1049] ARM64 does not support NUMA - returning NUMA node zero 2021-03-06 13:50:02.708606: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1049] ARM64 does not support NUMA - returning NUMA node zero 2021-03-06 13:50:02.708662: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1793] Adding visible gpu devices: 0 2021-03-06 13:50:02.708750: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2 2021-03-06 13:50:10.682276: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1206] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-03-06 13:50:10.682361: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212] 0 2021-03-06 13:50:10.682416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1225] 0: N 2021-03-06 13:50:10.682845: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1049] ARM64 does not support NUMA - returning NUMA node zero 2021-03-06 13:50:10.683149: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1049] ARM64 does not support NUMA - returning NUMA node zero 2021-03-06 13:50:10.683335: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1351] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 610 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3) WARNING:tensorflow:From /home/q_nerk/.local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4070: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead. 2021-03-06 13:50:37.139881: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:568] remapper failed: Invalid argument: Subshape must have computed start >= end since stride is negative, but is 0 and 2 (computed from start 0 and end 9223372036854775807 over shape with rank 2 and stride-1) 2021-03-06 13:50:37.709370: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:568] layout failed: Invalid argument: Subshape must have computed start >= end since stride is negative, but is 0 and 2 (computed from start 0 and end 9223372036854775807 over shape with rank 2 and stride-1) 2021-03-06 13:50:38.259970: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:568] remapper failed: Invalid argument: Subshape must have computed start >= end since stride is negative, but is 0 and 2 (computed from start 0 and end 9223372036854775807 over shape with rank 2 and stride-1) 2021-03-06 13:50:39.725626: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2021-03-06 13:50:55.625441: W tensorflow/core/common_runtime/bfc_allocator.cc:419] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.78GiB (rounded to 1910708992). If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation. Current allocation summary follows. 2021-03-06 13:50:55.627421: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (256): Total Chunks: 50, Chunks in use: 50. 12.5KiB allocated for chunks. 12.5KiB in use in bin. 2.8KiB client-requested in use in bin. 2021-03-06 13:50:55.627972: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (512): Total Chunks: 10, Chunks in use: 10. 5.0KiB allocated for chunks. 5.0KiB in use in bin. 5.0KiB client-requested in use in bin. 2021-03-06 13:50:55.628551: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (1024): Total Chunks: 20, Chunks in use: 20. 21.8KiB allocated for chunks. 21.8KiB in use in bin. 21.4KiB client-requested in use in bin. 2021-03-06 13:50:55.629055: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (2048): Total Chunks: 10, Chunks in use: 10. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin. 2021-03-06 13:50:55.629528: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (4096): Total Chunks: 6, Chunks in use: 6. 24.0KiB allocated for chunks. 24.0KiB in use in bin. 24.0KiB client-requested in use in bin. 2021-03-06 13:50:55.629973: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:50:55.630628: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (16384): Total Chunks: 1, Chunks in use: 1. 18.0KiB allocated for chunks. 18.0KiB in use in bin. 18.0KiB client-requested in use in bin. 2021-03-06 13:50:55.630756: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:50:55.630921: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (65536): Total Chunks: 1, Chunks in use: 1. 102.0KiB allocated for chunks. 102.0KiB in use in bin. 72.0KiB client-requested in use in bin. 2021-03-06 13:50:55.631090: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (131072): Total Chunks: 2, Chunks in use: 2. 383.0KiB allocated for chunks. 383.0KiB in use in bin. 383.0KiB client-requested in use in bin. 2021-03-06 13:50:55.631311: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (262144): Total Chunks: 2, Chunks in use: 2. 798.0KiB allocated for chunks. 798.0KiB in use in bin. 798.0KiB client-requested in use in bin. 2021-03-06 13:50:55.631495: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (524288): Total Chunks: 1, Chunks in use: 0. 796.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:50:55.631720: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (1048576): Total Chunks: 2, Chunks in use: 2. 2.12MiB allocated for chunks. 2.12MiB in use in bin. 2.12MiB client-requested in use in bin. 2021-03-06 13:50:55.631938: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (2097152): Total Chunks: 2, Chunks in use: 1. 6.98MiB allocated for chunks. 3.38MiB in use in bin. 3.38MiB client-requested in use in bin. 2021-03-06 13:50:55.632195: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (4194304): Total Chunks: 4, Chunks in use: 3. 24.26MiB allocated for chunks. 16.92MiB in use in bin. 16.92MiB client-requested in use in bin. 2021-03-06 13:50:55.632427: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (8388608): Total Chunks: 1, Chunks in use: 1. 12.28MiB allocated for chunks. 12.28MiB in use in bin. 7.92MiB client-requested in use in bin. 2021-03-06 13:50:55.632657: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (16777216): Total Chunks: 1, Chunks in use: 1. 18.00MiB allocated for chunks. 18.00MiB in use in bin. 18.00MiB client-requested in use in bin. 2021-03-06 13:50:55.632856: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (33554432): Total Chunks: 3, Chunks in use: 3. 142.75MiB allocated for chunks. 142.75MiB in use in bin. 142.75MiB client-requested in use in bin. 2021-03-06 13:50:55.633059: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:50:55.633227: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:50:55.633436: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (268435456): Total Chunks: 1, Chunks in use: 0. 401.49MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:50:55.633620: I tensorflow/core/common_runtime/bfc_allocator.cc:888] Bin for 1.78GiB was 256.00MiB, Chunk State: 2021-03-06 13:50:55.633881: I tensorflow/core/common_runtime/bfc_allocator.cc:894] Size: 401.49MiB | Requested Size: 16.36MiB | in_use: 0 | bin_num: 20, prev: Size: 58.25MiB | Requested Size: 58.25MiB | in_use: 1 | bin_num: -1 2021-03-06 13:50:55.634193: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Next region of size 639664128 2021-03-06 13:50:55.634374: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10000 next 1 of size 1280 2021-03-06 13:50:55.634457: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10500 next 2 of size 256 2021-03-06 13:50:55.634528: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10600 next 3 of size 256 2021-03-06 13:50:55.634608: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10700 next 4 of size 256 2021-03-06 13:50:55.634675: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10800 next 5 of size 256 2021-03-06 13:50:55.634741: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10900 next 42 of size 256 2021-03-06 13:50:55.634835: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10a00 next 51 of size 256 2021-03-06 13:50:55.634939: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10b00 next 52 of size 256 2021-03-06 13:50:55.635023: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10c00 next 46 of size 256 2021-03-06 13:50:55.635109: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10d00 next 44 of size 256 2021-03-06 13:50:55.635219: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10e00 next 110 of size 256 2021-03-06 13:50:55.635333: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10f00 next 6 of size 256 2021-03-06 13:50:55.635450: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11000 next 7 of size 256 2021-03-06 13:50:55.635549: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11100 next 8 of size 1024 2021-03-06 13:50:55.635644: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11500 next 9 of size 256 2021-03-06 13:50:55.635755: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11600 next 10 of size 1024 2021-03-06 13:50:55.635874: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11a00 next 11 of size 1024 2021-03-06 13:50:55.635972: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11e00 next 12 of size 512 2021-03-06 13:50:55.636065: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12000 next 13 of size 256 2021-03-06 13:50:55.636176: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12100 next 14 of size 256 2021-03-06 13:50:55.636272: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12200 next 15 of size 2048 2021-03-06 13:50:55.636386: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12a00 next 16 of size 256 2021-03-06 13:50:55.636503: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12b00 next 17 of size 256 2021-03-06 13:50:55.636600: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12c00 next 18 of size 4096 2021-03-06 13:50:55.636697: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b13c00 next 19 of size 2048 2021-03-06 13:50:55.636810: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b14400 next 20 of size 4096 2021-03-06 13:50:55.636933: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15400 next 21 of size 256 2021-03-06 13:50:55.637026: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15500 next 22 of size 256 2021-03-06 13:50:55.637117: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15600 next 23 of size 256 2021-03-06 13:50:55.637229: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15700 next 24 of size 256 2021-03-06 13:50:55.637324: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15800 next 25 of size 512 2021-03-06 13:50:55.637415: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15a00 next 26 of size 256 2021-03-06 13:50:55.637529: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15b00 next 27 of size 256 2021-03-06 13:50:55.637623: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15c00 next 28 of size 256 2021-03-06 13:50:55.637715: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15d00 next 29 of size 256 2021-03-06 13:50:55.637827: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15e00 next 30 of size 256 2021-03-06 13:50:55.637921: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15f00 next 31 of size 256 2021-03-06 13:50:55.638073: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16000 next 32 of size 256 2021-03-06 13:50:55.638212: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16100 next 33 of size 256 2021-03-06 13:50:55.638297: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16200 next 34 of size 256 2021-03-06 13:50:55.638349: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16300 next 35 of size 256 2021-03-06 13:50:55.638397: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16400 next 36 of size 256 2021-03-06 13:50:55.638444: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16500 next 37 of size 256 2021-03-06 13:50:55.638505: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16600 next 38 of size 256 2021-03-06 13:50:55.638557: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16700 next 39 of size 256 2021-03-06 13:50:55.638605: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16800 next 40 of size 256 2021-03-06 13:50:55.638657: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16900 next 41 of size 256 2021-03-06 13:50:55.638732: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16a00 next 113 of size 256 2021-03-06 13:50:55.638808: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16b00 next 114 of size 1792 2021-03-06 13:50:55.638878: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf00b17200 next 43 of size 815104 2021-03-06 13:50:55.638949: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00bde200 next 78 of size 1024 2021-03-06 13:50:55.639038: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00bde600 next 79 of size 261120 2021-03-06 13:50:55.639113: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c1e200 next 80 of size 256 2021-03-06 13:50:55.639189: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c1e300 next 95 of size 294912 2021-03-06 13:50:55.639289: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c66300 next 97 of size 1179648 2021-03-06 13:50:55.639386: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf00d86300 next 45 of size 7700224 2021-03-06 13:50:55.639462: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de200 next 50 of size 256 2021-03-06 13:50:55.639532: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de300 next 103 of size 256 2021-03-06 13:50:55.639622: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de400 next 104 of size 4096 2021-03-06 13:50:55.639712: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014df400 next 105 of size 4096 2021-03-06 13:50:55.639785: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e0400 next 106 of size 256 2021-03-06 13:50:55.639856: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e0500 next 107 of size 4096 2021-03-06 13:50:55.639946: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e1500 next 108 of size 2048 2021-03-06 13:50:55.640022: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e1d00 next 109 of size 1024 2021-03-06 13:50:55.640114: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf014e2100 next 47 of size 3783936 2021-03-06 13:50:55.640198: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187de00 next 82 of size 2048 2021-03-06 13:50:55.640243: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187e600 next 83 of size 512 2021-03-06 13:50:55.640291: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187e800 next 84 of size 1024 2021-03-06 13:50:55.640337: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ec00 next 85 of size 256 2021-03-06 13:50:55.640372: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ed00 next 86 of size 256 2021-03-06 13:50:55.640417: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ee00 next 87 of size 1024 2021-03-06 13:50:55.640463: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187f200 next 88 of size 1024 2021-03-06 13:50:55.640510: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187f600 next 89 of size 1792 2021-03-06 13:50:55.640561: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187fd00 next 90 of size 18432 2021-03-06 13:50:55.640616: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01884500 next 91 of size 256 2021-03-06 13:50:55.640663: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01884600 next 48 of size 104448 2021-03-06 13:50:55.640732: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189de00 next 49 of size 1024 2021-03-06 13:50:55.640793: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e200 next 92 of size 256 2021-03-06 13:50:55.640859: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e300 next 93 of size 256 2021-03-06 13:50:55.640912: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e400 next 94 of size 256 2021-03-06 13:50:55.640964: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e500 next 96 of size 1024 2021-03-06 13:50:55.641031: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e900 next 98 of size 512 2021-03-06 13:50:55.641100: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189eb00 next 99 of size 256 2021-03-06 13:50:55.641170: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189ec00 next 100 of size 2048 2021-03-06 13:50:55.641238: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f400 next 101 of size 512 2021-03-06 13:50:55.641308: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f600 next 102 of size 512 2021-03-06 13:50:55.641381: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f800 next 111 of size 8306688 2021-03-06 13:50:55.641453: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0208b800 next 53 of size 12881408 2021-03-06 13:50:55.641524: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd4600 next 54 of size 4096 2021-03-06 13:50:55.641593: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd5600 next 55 of size 1024 2021-03-06 13:50:55.641661: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd5a00 next 56 of size 2048 2021-03-06 13:50:55.641730: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6200 next 57 of size 2048 2021-03-06 13:50:55.641798: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6a00 next 58 of size 1024 2021-03-06 13:50:55.641866: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6e00 next 59 of size 2048 2021-03-06 13:50:55.641935: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd7600 next 60 of size 512 2021-03-06 13:50:55.642005: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd7800 next 61 of size 3538944 2021-03-06 13:50:55.642128: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037800 next 62 of size 512 2021-03-06 13:50:55.642172: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037a00 next 63 of size 1024 2021-03-06 13:50:55.642210: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037e00 next 64 of size 512 2021-03-06 13:50:55.642244: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038000 next 65 of size 512 2021-03-06 13:50:55.642277: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038200 next 66 of size 256 2021-03-06 13:50:55.642309: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038300 next 67 of size 1024 2021-03-06 13:50:55.642341: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038700 next 68 of size 1024 2021-03-06 13:50:55.642374: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038b00 next 69 of size 522240 2021-03-06 13:50:55.642409: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf030b8300 next 70 of size 1048576 2021-03-06 13:50:55.642444: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf031b8300 next 71 of size 2048 2021-03-06 13:50:55.642478: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf031b8b00 next 72 of size 4718592 2021-03-06 13:50:55.642513: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03638b00 next 73 of size 4718592 2021-03-06 13:50:55.642550: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab8b00 next 74 of size 1024 2021-03-06 13:50:55.642589: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab8f00 next 75 of size 2048 2021-03-06 13:50:55.642629: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab9700 next 76 of size 1024 2021-03-06 13:50:55.642673: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab9b00 next 77 of size 131072 2021-03-06 13:50:55.642719: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ad9b00 next 81 of size 18874368 2021-03-06 13:50:55.642759: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf04cd9b00 next 112 of size 44302336 2021-03-06 13:50:55.642798: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf07719b00 next 115 of size 44302336 2021-03-06 13:50:55.642844: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0a159b00 next 116 of size 61079552 2021-03-06 13:50:55.642893: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf0db99b00 next 18446744073709551615 of size 420996352 2021-03-06 13:50:55.642939: I tensorflow/core/common_runtime/bfc_allocator.cc:917] Summary of in-use Chunks by size: 2021-03-06 13:50:55.642997: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 50 Chunks of size 256 totalling 12.5KiB 2021-03-06 13:50:55.643045: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 10 Chunks of size 512 totalling 5.0KiB 2021-03-06 13:50:55.643101: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 17 Chunks of size 1024 totalling 17.0KiB 2021-03-06 13:50:55.643156: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1280 totalling 1.2KiB 2021-03-06 13:50:55.643203: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 2 Chunks of size 1792 totalling 3.5KiB 2021-03-06 13:50:55.643259: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 10 Chunks of size 2048 totalling 20.0KiB 2021-03-06 13:50:55.643315: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 6 Chunks of size 4096 totalling 24.0KiB 2021-03-06 13:50:55.643371: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 18432 totalling 18.0KiB 2021-03-06 13:50:55.643428: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 104448 totalling 102.0KiB 2021-03-06 13:50:55.643485: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 131072 totalling 128.0KiB 2021-03-06 13:50:55.643542: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 261120 totalling 255.0KiB 2021-03-06 13:50:55.643598: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 294912 totalling 288.0KiB 2021-03-06 13:50:55.643654: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 522240 totalling 510.0KiB 2021-03-06 13:50:55.643710: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1048576 totalling 1.00MiB 2021-03-06 13:50:55.643767: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1179648 totalling 1.12MiB 2021-03-06 13:50:55.643822: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 3538944 totalling 3.38MiB 2021-03-06 13:50:55.643876: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 2 Chunks of size 4718592 totalling 9.00MiB 2021-03-06 13:50:55.643931: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 8306688 totalling 7.92MiB 2021-03-06 13:50:55.643987: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 12881408 totalling 12.28MiB 2021-03-06 13:50:55.644042: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 18874368 totalling 18.00MiB 2021-03-06 13:50:55.644100: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 2 Chunks of size 44302336 totalling 84.50MiB 2021-03-06 13:50:55.644156: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 61079552 totalling 58.25MiB 2021-03-06 13:50:55.644205: I tensorflow/core/common_runtime/bfc_allocator.cc:924] Sum Total of in-use chunks: 196.81MiB 2021-03-06 13:50:55.644253: I tensorflow/core/common_runtime/bfc_allocator.cc:926] total_region_allocated_bytes_: 639664128 memory_limit_: 639664128 available bytes: 0 curr_region_allocation_bytes_: 1279328256 2021-03-06 13:50:55.644322: I tensorflow/core/common_runtime/bfc_allocator.cc:932] Stats: Limit: 639664128 InUse: 206368512 MaxInUse: 223520000 NumAllocs: 128 MaxAllocSize: 61079552 2021-03-06 13:50:55.644431: W tensorflow/core/common_runtime/bfc_allocator.cc:427] ***********************************_________________________________________________________________ 2021-03-06 13:50:55.645268: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10 2021-03-06 13:51:09.083984: W tensorflow/core/common_runtime/bfc_allocator.cc:419] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.13GiB (rounded to 1213014016). If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation. Current allocation summary follows. 2021-03-06 13:51:09.085049: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (256): Total Chunks: 50, Chunks in use: 50. 12.5KiB allocated for chunks. 12.5KiB in use in bin. 2.8KiB client-requested in use in bin. 2021-03-06 13:51:09.085556: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (512): Total Chunks: 10, Chunks in use: 10. 5.0KiB allocated for chunks. 5.0KiB in use in bin. 5.0KiB client-requested in use in bin. 2021-03-06 13:51:09.086281: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (1024): Total Chunks: 19, Chunks in use: 19. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 19.7KiB client-requested in use in bin. 2021-03-06 13:51:09.086820: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (2048): Total Chunks: 10, Chunks in use: 10. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin. 2021-03-06 13:51:09.087223: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (4096): Total Chunks: 6, Chunks in use: 6. 24.0KiB allocated for chunks. 24.0KiB in use in bin. 24.0KiB client-requested in use in bin. 2021-03-06 13:51:09.087643: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:09.088053: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (16384): Total Chunks: 2, Chunks in use: 2. 36.0KiB allocated for chunks. 36.0KiB in use in bin. 36.0KiB client-requested in use in bin. 2021-03-06 13:51:09.088495: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:09.088991: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (65536): Total Chunks: 1, Chunks in use: 1. 102.0KiB allocated for chunks. 102.0KiB in use in bin. 72.0KiB client-requested in use in bin. 2021-03-06 13:51:09.089483: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (131072): Total Chunks: 2, Chunks in use: 2. 383.0KiB allocated for chunks. 383.0KiB in use in bin. 383.0KiB client-requested in use in bin. 2021-03-06 13:51:09.089943: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (262144): Total Chunks: 2, Chunks in use: 2. 798.0KiB allocated for chunks. 798.0KiB in use in bin. 798.0KiB client-requested in use in bin. 2021-03-06 13:51:09.090596: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (524288): Total Chunks: 1, Chunks in use: 0. 779.8KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:09.090946: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (1048576): Total Chunks: 2, Chunks in use: 2. 2.12MiB allocated for chunks. 2.12MiB in use in bin. 2.12MiB client-requested in use in bin. 2021-03-06 13:51:09.091095: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (2097152): Total Chunks: 2, Chunks in use: 1. 6.98MiB allocated for chunks. 3.38MiB in use in bin. 3.38MiB client-requested in use in bin. 2021-03-06 13:51:09.091224: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (4194304): Total Chunks: 3, Chunks in use: 2. 16.34MiB allocated for chunks. 9.00MiB in use in bin. 9.00MiB client-requested in use in bin. 2021-03-06 13:51:09.091411: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (8388608): Total Chunks: 1, Chunks in use: 1. 10.56MiB allocated for chunks. 10.56MiB in use in bin. 10.56MiB client-requested in use in bin. 2021-03-06 13:51:09.091626: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (16777216): Total Chunks: 4, Chunks in use: 4. 80.46MiB allocated for chunks. 80.46MiB in use in bin. 70.81MiB client-requested in use in bin. 2021-03-06 13:51:09.091868: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (33554432): Total Chunks: 1, Chunks in use: 1. 37.12MiB allocated for chunks. 37.12MiB in use in bin. 37.12MiB client-requested in use in bin. 2021-03-06 13:51:09.092075: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:09.092246: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:09.092435: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (268435456): Total Chunks: 1, Chunks in use: 0. 454.31MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:09.092683: I tensorflow/core/common_runtime/bfc_allocator.cc:888] Bin for 1.13GiB was 256.00MiB, Chunk State: 2021-03-06 13:51:09.092966: I tensorflow/core/common_runtime/bfc_allocator.cc:894] Size: 454.31MiB | Requested Size: 18.52MiB | in_use: 0 | bin_num: 20, prev: Size: 37.12MiB | Requested Size: 37.12MiB | in_use: 1 | bin_num: -1 2021-03-06 13:51:09.093168: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Next region of size 639664128 2021-03-06 13:51:09.093371: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10000 next 1 of size 1280 2021-03-06 13:51:09.093563: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10500 next 2 of size 256 2021-03-06 13:51:09.093752: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10600 next 3 of size 256 2021-03-06 13:51:09.093897: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10700 next 4 of size 256 2021-03-06 13:51:09.094301: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10800 next 5 of size 256 2021-03-06 13:51:09.094386: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10900 next 42 of size 256 2021-03-06 13:51:09.094493: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10a00 next 51 of size 256 2021-03-06 13:51:09.094572: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10b00 next 52 of size 256 2021-03-06 13:51:09.094638: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10c00 next 46 of size 256 2021-03-06 13:51:09.094709: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10d00 next 44 of size 256 2021-03-06 13:51:09.094804: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10e00 next 110 of size 256 2021-03-06 13:51:09.094909: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10f00 next 6 of size 256 2021-03-06 13:51:09.094999: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11000 next 7 of size 256 2021-03-06 13:51:09.095099: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11100 next 8 of size 1024 2021-03-06 13:51:09.095216: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11500 next 9 of size 256 2021-03-06 13:51:09.095312: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11600 next 10 of size 1024 2021-03-06 13:51:09.095410: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11a00 next 11 of size 1024 2021-03-06 13:51:09.095530: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11e00 next 12 of size 512 2021-03-06 13:51:09.095627: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12000 next 13 of size 256 2021-03-06 13:51:09.095720: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12100 next 14 of size 256 2021-03-06 13:51:09.095841: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12200 next 15 of size 2048 2021-03-06 13:51:09.095937: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12a00 next 16 of size 256 2021-03-06 13:51:09.096032: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12b00 next 17 of size 256 2021-03-06 13:51:09.096151: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12c00 next 18 of size 4096 2021-03-06 13:51:09.096280: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b13c00 next 19 of size 2048 2021-03-06 13:51:09.096379: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b14400 next 20 of size 4096 2021-03-06 13:51:09.096474: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15400 next 21 of size 256 2021-03-06 13:51:09.096589: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15500 next 22 of size 256 2021-03-06 13:51:09.096707: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15600 next 23 of size 256 2021-03-06 13:51:09.096805: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15700 next 24 of size 256 2021-03-06 13:51:09.096898: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15800 next 25 of size 512 2021-03-06 13:51:09.097016: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15a00 next 26 of size 256 2021-03-06 13:51:09.097112: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15b00 next 27 of size 256 2021-03-06 13:51:09.097203: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15c00 next 28 of size 256 2021-03-06 13:51:09.097316: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15d00 next 29 of size 256 2021-03-06 13:51:09.097439: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15e00 next 30 of size 256 2021-03-06 13:51:09.097537: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15f00 next 31 of size 256 2021-03-06 13:51:09.097628: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16000 next 32 of size 256 2021-03-06 13:51:09.097743: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16100 next 33 of size 256 2021-03-06 13:51:09.097865: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16200 next 34 of size 256 2021-03-06 13:51:09.097964: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16300 next 35 of size 256 2021-03-06 13:51:09.098173: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16400 next 36 of size 256 2021-03-06 13:51:09.098228: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16500 next 37 of size 256 2021-03-06 13:51:09.098268: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16600 next 38 of size 256 2021-03-06 13:51:09.098307: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16700 next 39 of size 256 2021-03-06 13:51:09.098355: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16800 next 40 of size 256 2021-03-06 13:51:09.098393: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16900 next 41 of size 256 2021-03-06 13:51:09.098429: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16a00 next 113 of size 256 2021-03-06 13:51:09.098464: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16b00 next 114 of size 18432 2021-03-06 13:51:09.098504: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf00b1b300 next 43 of size 798464 2021-03-06 13:51:09.098543: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00bde200 next 78 of size 1024 2021-03-06 13:51:09.098595: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00bde600 next 79 of size 261120 2021-03-06 13:51:09.098663: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c1e200 next 80 of size 256 2021-03-06 13:51:09.098722: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c1e300 next 95 of size 294912 2021-03-06 13:51:09.098779: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c66300 next 97 of size 1179648 2021-03-06 13:51:09.098832: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf00d86300 next 45 of size 7700224 2021-03-06 13:51:09.098888: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de200 next 50 of size 256 2021-03-06 13:51:09.098955: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de300 next 103 of size 256 2021-03-06 13:51:09.099023: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de400 next 104 of size 4096 2021-03-06 13:51:09.099094: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014df400 next 105 of size 4096 2021-03-06 13:51:09.099166: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e0400 next 106 of size 256 2021-03-06 13:51:09.099226: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e0500 next 107 of size 4096 2021-03-06 13:51:09.099283: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e1500 next 108 of size 2048 2021-03-06 13:51:09.099340: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e1d00 next 109 of size 1024 2021-03-06 13:51:09.099396: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf014e2100 next 47 of size 3783936 2021-03-06 13:51:09.099450: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187de00 next 82 of size 2048 2021-03-06 13:51:09.099505: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187e600 next 83 of size 512 2021-03-06 13:51:09.099560: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187e800 next 84 of size 1024 2021-03-06 13:51:09.099615: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ec00 next 85 of size 256 2021-03-06 13:51:09.099670: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ed00 next 86 of size 256 2021-03-06 13:51:09.099726: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ee00 next 87 of size 1024 2021-03-06 13:51:09.099781: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187f200 next 88 of size 1024 2021-03-06 13:51:09.099837: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187f600 next 89 of size 1792 2021-03-06 13:51:09.099895: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187fd00 next 90 of size 18432 2021-03-06 13:51:09.099963: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01884500 next 91 of size 256 2021-03-06 13:51:09.100033: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01884600 next 48 of size 104448 2021-03-06 13:51:09.100104: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189de00 next 49 of size 1024 2021-03-06 13:51:09.100172: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e200 next 92 of size 256 2021-03-06 13:51:09.100244: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e300 next 93 of size 256 2021-03-06 13:51:09.100313: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e400 next 94 of size 256 2021-03-06 13:51:09.100380: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e500 next 96 of size 1024 2021-03-06 13:51:09.100448: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e900 next 98 of size 512 2021-03-06 13:51:09.100516: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189eb00 next 99 of size 256 2021-03-06 13:51:09.100583: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189ec00 next 100 of size 2048 2021-03-06 13:51:09.100651: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f400 next 101 of size 512 2021-03-06 13:51:09.100718: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f600 next 102 of size 512 2021-03-06 13:51:09.100788: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f800 next 53 of size 21188096 2021-03-06 13:51:09.100859: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd4600 next 54 of size 4096 2021-03-06 13:51:09.100928: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd5600 next 55 of size 1024 2021-03-06 13:51:09.100997: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd5a00 next 56 of size 2048 2021-03-06 13:51:09.101066: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6200 next 57 of size 2048 2021-03-06 13:51:09.101135: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6a00 next 58 of size 1024 2021-03-06 13:51:09.101203: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6e00 next 59 of size 2048 2021-03-06 13:51:09.101270: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd7600 next 60 of size 512 2021-03-06 13:51:09.101340: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd7800 next 61 of size 3538944 2021-03-06 13:51:09.101410: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037800 next 62 of size 512 2021-03-06 13:51:09.101478: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037a00 next 63 of size 1024 2021-03-06 13:51:09.101546: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037e00 next 64 of size 512 2021-03-06 13:51:09.101615: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038000 next 65 of size 512 2021-03-06 13:51:09.101682: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038200 next 66 of size 256 2021-03-06 13:51:09.101750: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038300 next 67 of size 1024 2021-03-06 13:51:09.101818: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038700 next 68 of size 1024 2021-03-06 13:51:09.101887: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038b00 next 69 of size 522240 2021-03-06 13:51:09.101960: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf030b8300 next 70 of size 1048576 2021-03-06 13:51:09.102078: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf031b8300 next 71 of size 2048 2021-03-06 13:51:09.102138: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf031b8b00 next 72 of size 4718592 2021-03-06 13:51:09.102172: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03638b00 next 73 of size 4718592 2021-03-06 13:51:09.102203: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab8b00 next 74 of size 1024 2021-03-06 13:51:09.102233: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab8f00 next 75 of size 2048 2021-03-06 13:51:09.102261: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab9700 next 76 of size 1024 2021-03-06 13:51:09.102290: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab9b00 next 77 of size 131072 2021-03-06 13:51:09.102319: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ad9b00 next 81 of size 18874368 2021-03-06 13:51:09.102349: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf04cd9b00 next 112 of size 22151168 2021-03-06 13:51:09.102379: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf061f9b00 next 111 of size 11075584 2021-03-06 13:51:09.102408: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf06c89b00 next 118 of size 22151168 2021-03-06 13:51:09.102436: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf081a9b00 next 115 of size 38928384 2021-03-06 13:51:09.102466: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf0a6c9b00 next 18446744073709551615 of size 476374272 2021-03-06 13:51:09.102495: I tensorflow/core/common_runtime/bfc_allocator.cc:917] Summary of in-use Chunks by size: 2021-03-06 13:51:09.102534: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 50 Chunks of size 256 totalling 12.5KiB 2021-03-06 13:51:09.102572: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 10 Chunks of size 512 totalling 5.0KiB 2021-03-06 13:51:09.102613: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 17 Chunks of size 1024 totalling 17.0KiB 2021-03-06 13:51:09.102654: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1280 totalling 1.2KiB 2021-03-06 13:51:09.102695: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1792 totalling 1.8KiB 2021-03-06 13:51:09.102738: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 10 Chunks of size 2048 totalling 20.0KiB 2021-03-06 13:51:09.102782: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 6 Chunks of size 4096 totalling 24.0KiB 2021-03-06 13:51:09.102828: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 2 Chunks of size 18432 totalling 36.0KiB 2021-03-06 13:51:09.102874: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 104448 totalling 102.0KiB 2021-03-06 13:51:09.102920: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 131072 totalling 128.0KiB 2021-03-06 13:51:09.102968: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 261120 totalling 255.0KiB 2021-03-06 13:51:09.103017: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 294912 totalling 288.0KiB 2021-03-06 13:51:09.103065: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 522240 totalling 510.0KiB 2021-03-06 13:51:09.103114: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1048576 totalling 1.00MiB 2021-03-06 13:51:09.103154: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1179648 totalling 1.12MiB 2021-03-06 13:51:09.103193: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 3538944 totalling 3.38MiB 2021-03-06 13:51:09.103239: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 2 Chunks of size 4718592 totalling 9.00MiB 2021-03-06 13:51:09.103288: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 11075584 totalling 10.56MiB 2021-03-06 13:51:09.103336: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 18874368 totalling 18.00MiB 2021-03-06 13:51:09.103385: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 21188096 totalling 20.21MiB 2021-03-06 13:51:09.103433: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 2 Chunks of size 22151168 totalling 42.25MiB 2021-03-06 13:51:09.103481: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 38928384 totalling 37.12MiB 2021-03-06 13:51:09.103529: I tensorflow/core/common_runtime/bfc_allocator.cc:924] Sum Total of in-use chunks: 144.01MiB 2021-03-06 13:51:09.103573: I tensorflow/core/common_runtime/bfc_allocator.cc:926] total_region_allocated_bytes_: 639664128 memory_limit_: 639664128 available bytes: 0 curr_region_allocation_bytes_: 1279328256 2021-03-06 13:51:09.103627: I tensorflow/core/common_runtime/bfc_allocator.cc:932] Stats: Limit: 639664128 InUse: 151007232 MaxInUse: 223520000 NumAllocs: 150 MaxAllocSize: 61079552 2021-03-06 13:51:09.103699: W tensorflow/core/common_runtime/bfc_allocator.cc:427] ****x*********************__________________________________________________________________________ 2021-03-06 13:51:19.415345: W tensorflow/core/common_runtime/bfc_allocator.cc:419] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.04GiB (rounded to 1115889664). If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation. Current allocation summary follows. 2021-03-06 13:51:19.416259: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (256): Total Chunks: 50, Chunks in use: 50. 12.5KiB allocated for chunks. 12.5KiB in use in bin. 2.8KiB client-requested in use in bin. 2021-03-06 13:51:19.416876: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (512): Total Chunks: 10, Chunks in use: 10. 5.0KiB allocated for chunks. 5.0KiB in use in bin. 5.0KiB client-requested in use in bin. 2021-03-06 13:51:19.417378: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (1024): Total Chunks: 19, Chunks in use: 19. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 19.7KiB client-requested in use in bin. 2021-03-06 13:51:19.417867: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (2048): Total Chunks: 10, Chunks in use: 10. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin. 2021-03-06 13:51:19.419188: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (4096): Total Chunks: 6, Chunks in use: 6. 24.0KiB allocated for chunks. 24.0KiB in use in bin. 24.0KiB client-requested in use in bin. 2021-03-06 13:51:19.419332: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:19.419883: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (16384): Total Chunks: 1, Chunks in use: 1. 18.0KiB allocated for chunks. 18.0KiB in use in bin. 18.0KiB client-requested in use in bin. 2021-03-06 13:51:19.420260: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:19.420774: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (65536): Total Chunks: 2, Chunks in use: 2. 174.0KiB allocated for chunks. 174.0KiB in use in bin. 144.0KiB client-requested in use in bin. 2021-03-06 13:51:19.421290: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (131072): Total Chunks: 2, Chunks in use: 2. 383.0KiB allocated for chunks. 383.0KiB in use in bin. 383.0KiB client-requested in use in bin. 2021-03-06 13:51:19.421769: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (262144): Total Chunks: 2, Chunks in use: 2. 798.0KiB allocated for chunks. 798.0KiB in use in bin. 798.0KiB client-requested in use in bin. 2021-03-06 13:51:19.422643: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (524288): Total Chunks: 1, Chunks in use: 0. 725.8KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:19.422804: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (1048576): Total Chunks: 2, Chunks in use: 2. 2.12MiB allocated for chunks. 2.12MiB in use in bin. 2.12MiB client-requested in use in bin. 2021-03-06 13:51:19.422944: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (2097152): Total Chunks: 2, Chunks in use: 1. 6.98MiB allocated for chunks. 3.38MiB in use in bin. 3.38MiB client-requested in use in bin. 2021-03-06 13:51:19.423139: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (4194304): Total Chunks: 4, Chunks in use: 4. 21.62MiB allocated for chunks. 21.62MiB in use in bin. 19.56MiB client-requested in use in bin. 2021-03-06 13:51:19.423386: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (8388608): Total Chunks: 1, Chunks in use: 1. 10.56MiB allocated for chunks. 10.56MiB in use in bin. 10.56MiB client-requested in use in bin. 2021-03-06 13:51:19.423585: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (16777216): Total Chunks: 3, Chunks in use: 3. 64.77MiB allocated for chunks. 64.77MiB in use in bin. 55.12MiB client-requested in use in bin. 2021-03-06 13:51:19.423790: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:19.423957: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:19.424153: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:19.424371: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (268435456): Total Chunks: 1, Chunks in use: 0. 501.84MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:19.424553: I tensorflow/core/common_runtime/bfc_allocator.cc:888] Bin for 1.04GiB was 256.00MiB, Chunk State: 2021-03-06 13:51:19.424808: I tensorflow/core/common_runtime/bfc_allocator.cc:894] Size: 501.84MiB | Requested Size: 63.53MiB | in_use: 0 | bin_num: 20, prev: Size: 26.56MiB | Requested Size: 26.56MiB | in_use: 1 | bin_num: -1 2021-03-06 13:51:19.424962: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Next region of size 639664128 2021-03-06 13:51:19.425159: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10000 next 1 of size 1280 2021-03-06 13:51:19.425311: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10500 next 2 of size 256 2021-03-06 13:51:19.425483: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10600 next 3 of size 256 2021-03-06 13:51:19.425660: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10700 next 4 of size 256 2021-03-06 13:51:19.425803: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10800 next 5 of size 256 2021-03-06 13:51:19.425988: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10900 next 42 of size 256 2021-03-06 13:51:19.426459: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10a00 next 51 of size 256 2021-03-06 13:51:19.426525: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10b00 next 52 of size 256 2021-03-06 13:51:19.426588: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10c00 next 46 of size 256 2021-03-06 13:51:19.426674: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10d00 next 44 of size 256 2021-03-06 13:51:19.426770: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10e00 next 110 of size 256 2021-03-06 13:51:19.426854: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10f00 next 6 of size 256 2021-03-06 13:51:19.426939: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11000 next 7 of size 256 2021-03-06 13:51:19.427060: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11100 next 8 of size 1024 2021-03-06 13:51:19.427179: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11500 next 9 of size 256 2021-03-06 13:51:19.427274: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11600 next 10 of size 1024 2021-03-06 13:51:19.427366: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11a00 next 11 of size 1024 2021-03-06 13:51:19.427481: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11e00 next 12 of size 512 2021-03-06 13:51:19.427578: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12000 next 13 of size 256 2021-03-06 13:51:19.427669: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12100 next 14 of size 256 2021-03-06 13:51:19.427784: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12200 next 15 of size 2048 2021-03-06 13:51:19.427878: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12a00 next 16 of size 256 2021-03-06 13:51:19.427970: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12b00 next 17 of size 256 2021-03-06 13:51:19.428098: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12c00 next 18 of size 4096 2021-03-06 13:51:19.428221: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b13c00 next 19 of size 2048 2021-03-06 13:51:19.428334: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b14400 next 20 of size 4096 2021-03-06 13:51:19.428448: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15400 next 21 of size 256 2021-03-06 13:51:19.428559: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15500 next 22 of size 256 2021-03-06 13:51:19.428671: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15600 next 23 of size 256 2021-03-06 13:51:19.428784: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15700 next 24 of size 256 2021-03-06 13:51:19.428896: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15800 next 25 of size 512 2021-03-06 13:51:19.429008: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15a00 next 26 of size 256 2021-03-06 13:51:19.429122: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15b00 next 27 of size 256 2021-03-06 13:51:19.429234: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15c00 next 28 of size 256 2021-03-06 13:51:19.429347: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15d00 next 29 of size 256 2021-03-06 13:51:19.429459: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15e00 next 30 of size 256 2021-03-06 13:51:19.429571: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15f00 next 31 of size 256 2021-03-06 13:51:19.429682: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16000 next 32 of size 256 2021-03-06 13:51:19.429796: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16100 next 33 of size 256 2021-03-06 13:51:19.429908: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16200 next 34 of size 256 2021-03-06 13:51:19.430304: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16300 next 35 of size 256 2021-03-06 13:51:19.430530: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16400 next 36 of size 256 2021-03-06 13:51:19.431573: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16500 next 37 of size 256 2021-03-06 13:51:19.431671: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16600 next 38 of size 256 2021-03-06 13:51:19.431759: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16700 next 39 of size 256 2021-03-06 13:51:19.431847: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16800 next 40 of size 256 2021-03-06 13:51:19.431935: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16900 next 41 of size 256 2021-03-06 13:51:19.432021: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16a00 next 113 of size 256 2021-03-06 13:51:19.432110: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16b00 next 115 of size 73728 2021-03-06 13:51:19.432198: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf00b28b00 next 43 of size 743168 2021-03-06 13:51:19.432286: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00bde200 next 78 of size 1024 2021-03-06 13:51:19.432380: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00bde600 next 79 of size 261120 2021-03-06 13:51:19.432468: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c1e200 next 80 of size 256 2021-03-06 13:51:19.432556: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c1e300 next 95 of size 294912 2021-03-06 13:51:19.432645: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c66300 next 97 of size 1179648 2021-03-06 13:51:19.432735: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00d86300 next 45 of size 7700224 2021-03-06 13:51:19.432822: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de200 next 50 of size 256 2021-03-06 13:51:19.432909: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de300 next 103 of size 256 2021-03-06 13:51:19.432998: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de400 next 104 of size 4096 2021-03-06 13:51:19.433085: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014df400 next 105 of size 4096 2021-03-06 13:51:19.433173: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e0400 next 106 of size 256 2021-03-06 13:51:19.433261: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e0500 next 107 of size 4096 2021-03-06 13:51:19.433349: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e1500 next 108 of size 2048 2021-03-06 13:51:19.433440: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e1d00 next 109 of size 1024 2021-03-06 13:51:19.433527: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf014e2100 next 47 of size 3783936 2021-03-06 13:51:19.433614: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187de00 next 82 of size 2048 2021-03-06 13:51:19.433702: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187e600 next 83 of size 512 2021-03-06 13:51:19.433790: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187e800 next 84 of size 1024 2021-03-06 13:51:19.433878: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ec00 next 85 of size 256 2021-03-06 13:51:19.433966: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ed00 next 86 of size 256 2021-03-06 13:51:19.434217: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ee00 next 87 of size 1024 2021-03-06 13:51:19.434259: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187f200 next 88 of size 1024 2021-03-06 13:51:19.434297: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187f600 next 89 of size 1792 2021-03-06 13:51:19.434340: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187fd00 next 90 of size 18432 2021-03-06 13:51:19.434380: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01884500 next 91 of size 256 2021-03-06 13:51:19.434419: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01884600 next 48 of size 104448 2021-03-06 13:51:19.434459: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189de00 next 49 of size 1024 2021-03-06 13:51:19.434497: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e200 next 92 of size 256 2021-03-06 13:51:19.434537: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e300 next 93 of size 256 2021-03-06 13:51:19.434579: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e400 next 94 of size 256 2021-03-06 13:51:19.434623: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e500 next 96 of size 1024 2021-03-06 13:51:19.434669: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e900 next 98 of size 512 2021-03-06 13:51:19.434717: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189eb00 next 99 of size 256 2021-03-06 13:51:19.434768: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189ec00 next 100 of size 2048 2021-03-06 13:51:19.434889: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f400 next 101 of size 512 2021-03-06 13:51:19.434935: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f600 next 102 of size 512 2021-03-06 13:51:19.434980: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f800 next 53 of size 21188096 2021-03-06 13:51:19.435024: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd4600 next 54 of size 4096 2021-03-06 13:51:19.435068: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd5600 next 55 of size 1024 2021-03-06 13:51:19.435112: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd5a00 next 56 of size 2048 2021-03-06 13:51:19.435159: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6200 next 57 of size 2048 2021-03-06 13:51:19.435207: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6a00 next 58 of size 1024 2021-03-06 13:51:19.435259: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6e00 next 59 of size 2048 2021-03-06 13:51:19.435314: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd7600 next 60 of size 512 2021-03-06 13:51:19.435373: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd7800 next 61 of size 3538944 2021-03-06 13:51:19.435435: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037800 next 62 of size 512 2021-03-06 13:51:19.435498: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037a00 next 63 of size 1024 2021-03-06 13:51:19.435564: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037e00 next 64 of size 512 2021-03-06 13:51:19.435629: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038000 next 65 of size 512 2021-03-06 13:51:19.435696: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038200 next 66 of size 256 2021-03-06 13:51:19.435763: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038300 next 67 of size 1024 2021-03-06 13:51:19.435830: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038700 next 68 of size 1024 2021-03-06 13:51:19.435899: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038b00 next 69 of size 522240 2021-03-06 13:51:19.435971: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf030b8300 next 70 of size 1048576 2021-03-06 13:51:19.436041: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf031b8300 next 71 of size 2048 2021-03-06 13:51:19.436109: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf031b8b00 next 72 of size 4718592 2021-03-06 13:51:19.436180: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03638b00 next 73 of size 4718592 2021-03-06 13:51:19.436248: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab8b00 next 74 of size 1024 2021-03-06 13:51:19.436315: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab8f00 next 75 of size 2048 2021-03-06 13:51:19.436382: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab9700 next 76 of size 1024 2021-03-06 13:51:19.436451: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab9b00 next 77 of size 131072 2021-03-06 13:51:19.436626: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ad9b00 next 81 of size 18874368 2021-03-06 13:51:19.436661: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf04cd9b00 next 111 of size 5537792 2021-03-06 13:51:19.436696: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf05221b00 next 112 of size 11075584 2021-03-06 13:51:19.436729: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf05cb1b00 next 116 of size 27852800 2021-03-06 13:51:19.436764: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf07741b00 next 18446744073709551615 of size 526214400 2021-03-06 13:51:19.436796: I tensorflow/core/common_runtime/bfc_allocator.cc:917] Summary of in-use Chunks by size: 2021-03-06 13:51:19.436837: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 50 Chunks of size 256 totalling 12.5KiB 2021-03-06 13:51:19.436877: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 10 Chunks of size 512 totalling 5.0KiB 2021-03-06 13:51:19.436918: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 17 Chunks of size 1024 totalling 17.0KiB 2021-03-06 13:51:19.436962: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1280 totalling 1.2KiB 2021-03-06 13:51:19.437007: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1792 totalling 1.8KiB 2021-03-06 13:51:19.437054: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 10 Chunks of size 2048 totalling 20.0KiB 2021-03-06 13:51:19.437103: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 6 Chunks of size 4096 totalling 24.0KiB 2021-03-06 13:51:19.437154: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 18432 totalling 18.0KiB 2021-03-06 13:51:19.437207: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 73728 totalling 72.0KiB 2021-03-06 13:51:19.437261: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 104448 totalling 102.0KiB 2021-03-06 13:51:19.437315: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 131072 totalling 128.0KiB 2021-03-06 13:51:19.437371: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 261120 totalling 255.0KiB 2021-03-06 13:51:19.437428: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 294912 totalling 288.0KiB 2021-03-06 13:51:19.437485: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 522240 totalling 510.0KiB 2021-03-06 13:51:19.437539: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1048576 totalling 1.00MiB 2021-03-06 13:51:19.437594: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1179648 totalling 1.12MiB 2021-03-06 13:51:19.437650: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 3538944 totalling 3.38MiB 2021-03-06 13:51:19.437705: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 2 Chunks of size 4718592 totalling 9.00MiB 2021-03-06 13:51:19.437759: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 5537792 totalling 5.28MiB 2021-03-06 13:51:19.437813: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 7700224 totalling 7.34MiB 2021-03-06 13:51:19.437869: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 11075584 totalling 10.56MiB 2021-03-06 13:51:19.437925: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 18874368 totalling 18.00MiB 2021-03-06 13:51:19.437981: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 21188096 totalling 20.21MiB 2021-03-06 13:51:19.438112: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 27852800 totalling 26.56MiB 2021-03-06 13:51:19.438146: I tensorflow/core/common_runtime/bfc_allocator.cc:924] Sum Total of in-use chunks: 103.88MiB 2021-03-06 13:51:19.438173: I tensorflow/core/common_runtime/bfc_allocator.cc:926] total_region_allocated_bytes_: 639664128 memory_limit_: 639664128 available bytes: 0 curr_region_allocation_bytes_: 1279328256 2021-03-06 13:51:19.438207: I tensorflow/core/common_runtime/bfc_allocator.cc:932] Stats: Limit: 639664128 InUse: 108922624 MaxInUse: 223520000 NumAllocs: 172 MaxAllocSize: 66617344 2021-03-06 13:51:19.438257: W tensorflow/core/common_runtime/bfc_allocator.cc:427] ****x*************__________________________________________________________________________________ 2021-03-06 13:51:29.907405: W tensorflow/core/common_runtime/bfc_allocator.cc:419] Allocator (GPU_0_bfc) ran out of memory trying to allocate 586.66MiB (rounded to 615153664). If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation. Current allocation summary follows. 2021-03-06 13:51:29.908317: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (256): Total Chunks: 50, Chunks in use: 50. 12.5KiB allocated for chunks. 12.5KiB in use in bin. 2.8KiB client-requested in use in bin. 2021-03-06 13:51:29.909360: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (512): Total Chunks: 10, Chunks in use: 10. 5.0KiB allocated for chunks. 5.0KiB in use in bin. 5.0KiB client-requested in use in bin. 2021-03-06 13:51:29.911816: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (1024): Total Chunks: 19, Chunks in use: 19. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 19.7KiB client-requested in use in bin. 2021-03-06 13:51:29.912871: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (2048): Total Chunks: 10, Chunks in use: 10. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin. 2021-03-06 13:51:29.913172: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (4096): Total Chunks: 6, Chunks in use: 6. 24.0KiB allocated for chunks. 24.0KiB in use in bin. 24.0KiB client-requested in use in bin. 2021-03-06 13:51:29.913496: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:29.913887: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (16384): Total Chunks: 1, Chunks in use: 1. 18.0KiB allocated for chunks. 18.0KiB in use in bin. 18.0KiB client-requested in use in bin. 2021-03-06 13:51:29.914364: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:29.914780: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (65536): Total Chunks: 2, Chunks in use: 2. 174.0KiB allocated for chunks. 174.0KiB in use in bin. 144.0KiB client-requested in use in bin. 2021-03-06 13:51:29.915231: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (131072): Total Chunks: 2, Chunks in use: 2. 383.0KiB allocated for chunks. 383.0KiB in use in bin. 383.0KiB client-requested in use in bin. 2021-03-06 13:51:29.915479: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (262144): Total Chunks: 2, Chunks in use: 2. 798.0KiB allocated for chunks. 798.0KiB in use in bin. 798.0KiB client-requested in use in bin. 2021-03-06 13:51:29.915706: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (524288): Total Chunks: 1, Chunks in use: 0. 725.8KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:29.915900: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (1048576): Total Chunks: 2, Chunks in use: 2. 2.12MiB allocated for chunks. 2.12MiB in use in bin. 2.12MiB client-requested in use in bin. 2021-03-06 13:51:29.916114: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (2097152): Total Chunks: 2, Chunks in use: 1. 6.98MiB allocated for chunks. 3.38MiB in use in bin. 3.38MiB client-requested in use in bin. 2021-03-06 13:51:29.916310: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (4194304): Total Chunks: 4, Chunks in use: 4. 21.62MiB allocated for chunks. 21.62MiB in use in bin. 19.56MiB client-requested in use in bin. 2021-03-06 13:51:29.916542: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (8388608): Total Chunks: 1, Chunks in use: 1. 10.56MiB allocated for chunks. 10.56MiB in use in bin. 10.56MiB client-requested in use in bin. 2021-03-06 13:51:29.916742: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (16777216): Total Chunks: 3, Chunks in use: 3. 64.77MiB allocated for chunks. 64.77MiB in use in bin. 55.12MiB client-requested in use in bin. 2021-03-06 13:51:29.916948: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:29.917117: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:29.917312: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:29.917496: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (268435456): Total Chunks: 1, Chunks in use: 0. 501.84MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:29.917713: I tensorflow/core/common_runtime/bfc_allocator.cc:888] Bin for 586.66MiB was 256.00MiB, Chunk State: 2021-03-06 13:51:29.917970: I tensorflow/core/common_runtime/bfc_allocator.cc:894] Size: 501.84MiB | Requested Size: 25.30MiB | in_use: 0 | bin_num: 20, prev: Size: 26.56MiB | Requested Size: 26.56MiB | in_use: 1 | bin_num: -1 2021-03-06 13:51:29.918265: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Next region of size 639664128 2021-03-06 13:51:29.918356: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10000 next 1 of size 1280 2021-03-06 13:51:29.918439: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10500 next 2 of size 256 2021-03-06 13:51:29.918520: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10600 next 3 of size 256 2021-03-06 13:51:29.918585: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10700 next 4 of size 256 2021-03-06 13:51:29.918655: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10800 next 5 of size 256 2021-03-06 13:51:29.918753: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10900 next 42 of size 256 2021-03-06 13:51:29.918858: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10a00 next 51 of size 256 2021-03-06 13:51:29.918943: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10b00 next 52 of size 256 2021-03-06 13:51:29.919031: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10c00 next 46 of size 256 2021-03-06 13:51:29.919143: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10d00 next 44 of size 256 2021-03-06 13:51:29.919261: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10e00 next 110 of size 256 2021-03-06 13:51:29.919355: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10f00 next 6 of size 256 2021-03-06 13:51:29.919445: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11000 next 7 of size 256 2021-03-06 13:51:29.919564: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11100 next 8 of size 1024 2021-03-06 13:51:29.919659: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11500 next 9 of size 256 2021-03-06 13:51:29.919751: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11600 next 10 of size 1024 2021-03-06 13:51:29.919864: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11a00 next 11 of size 1024 2021-03-06 13:51:29.919984: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11e00 next 12 of size 512 2021-03-06 13:51:29.920080: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12000 next 13 of size 256 2021-03-06 13:51:29.920174: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12100 next 14 of size 256 2021-03-06 13:51:29.920272: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12200 next 15 of size 2048 2021-03-06 13:51:29.920389: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12a00 next 16 of size 256 2021-03-06 13:51:29.920610: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12b00 next 17 of size 256 2021-03-06 13:51:29.920724: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12c00 next 18 of size 4096 2021-03-06 13:51:29.920802: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b13c00 next 19 of size 2048 2021-03-06 13:51:29.920850: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b14400 next 20 of size 4096 2021-03-06 13:51:29.920896: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15400 next 21 of size 256 2021-03-06 13:51:29.920943: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15500 next 22 of size 256 2021-03-06 13:51:29.921027: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15600 next 23 of size 256 2021-03-06 13:51:29.921099: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15700 next 24 of size 256 2021-03-06 13:51:29.921159: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15800 next 25 of size 512 2021-03-06 13:51:29.921223: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15a00 next 26 of size 256 2021-03-06 13:51:29.921327: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15b00 next 27 of size 256 2021-03-06 13:51:29.921411: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15c00 next 28 of size 256 2021-03-06 13:51:29.921484: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15d00 next 29 of size 256 2021-03-06 13:51:29.921555: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15e00 next 30 of size 256 2021-03-06 13:51:29.921652: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15f00 next 31 of size 256 2021-03-06 13:51:29.921745: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16000 next 32 of size 256 2021-03-06 13:51:29.921820: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16100 next 33 of size 256 2021-03-06 13:51:29.921889: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16200 next 34 of size 256 2021-03-06 13:51:29.922075: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16300 next 35 of size 256 2021-03-06 13:51:29.922221: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16400 next 36 of size 256 2021-03-06 13:51:29.922363: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16500 next 37 of size 256 2021-03-06 13:51:29.922504: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16600 next 38 of size 256 2021-03-06 13:51:29.922615: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16700 next 39 of size 256 2021-03-06 13:51:29.922689: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16800 next 40 of size 256 2021-03-06 13:51:29.922756: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16900 next 41 of size 256 2021-03-06 13:51:29.922826: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16a00 next 113 of size 256 2021-03-06 13:51:29.922898: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16b00 next 115 of size 73728 2021-03-06 13:51:29.922974: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf00b28b00 next 43 of size 743168 2021-03-06 13:51:29.923111: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00bde200 next 78 of size 1024 2021-03-06 13:51:29.923176: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00bde600 next 79 of size 261120 2021-03-06 13:51:29.923244: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c1e200 next 80 of size 256 2021-03-06 13:51:29.923299: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c1e300 next 95 of size 294912 2021-03-06 13:51:29.923376: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c66300 next 97 of size 1179648 2021-03-06 13:51:29.923452: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00d86300 next 45 of size 7700224 2021-03-06 13:51:29.923521: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de200 next 50 of size 256 2021-03-06 13:51:29.923589: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de300 next 103 of size 256 2021-03-06 13:51:29.923657: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de400 next 104 of size 4096 2021-03-06 13:51:29.923725: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014df400 next 105 of size 4096 2021-03-06 13:51:29.923792: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e0400 next 106 of size 256 2021-03-06 13:51:29.923859: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e0500 next 107 of size 4096 2021-03-06 13:51:29.923926: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e1500 next 108 of size 2048 2021-03-06 13:51:29.923994: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e1d00 next 109 of size 1024 2021-03-06 13:51:29.924062: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf014e2100 next 47 of size 3783936 2021-03-06 13:51:29.924129: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187de00 next 82 of size 2048 2021-03-06 13:51:29.924196: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187e600 next 83 of size 512 2021-03-06 13:51:29.924263: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187e800 next 84 of size 1024 2021-03-06 13:51:29.924330: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ec00 next 85 of size 256 2021-03-06 13:51:29.924396: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ed00 next 86 of size 256 2021-03-06 13:51:29.924463: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ee00 next 87 of size 1024 2021-03-06 13:51:29.924530: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187f200 next 88 of size 1024 2021-03-06 13:51:29.924601: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187f600 next 89 of size 1792 2021-03-06 13:51:29.924670: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187fd00 next 90 of size 18432 2021-03-06 13:51:29.924737: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01884500 next 91 of size 256 2021-03-06 13:51:29.924805: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01884600 next 48 of size 104448 2021-03-06 13:51:29.924884: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189de00 next 49 of size 1024 2021-03-06 13:51:29.924947: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e200 next 92 of size 256 2021-03-06 13:51:29.924988: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e300 next 93 of size 256 2021-03-06 13:51:29.925023: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e400 next 94 of size 256 2021-03-06 13:51:29.925059: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e500 next 96 of size 1024 2021-03-06 13:51:29.925094: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e900 next 98 of size 512 2021-03-06 13:51:29.925129: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189eb00 next 99 of size 256 2021-03-06 13:51:29.925164: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189ec00 next 100 of size 2048 2021-03-06 13:51:29.925197: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f400 next 101 of size 512 2021-03-06 13:51:29.925231: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f600 next 102 of size 512 2021-03-06 13:51:29.925266: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f800 next 53 of size 21188096 2021-03-06 13:51:29.925301: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd4600 next 54 of size 4096 2021-03-06 13:51:29.925338: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd5600 next 55 of size 1024 2021-03-06 13:51:29.925375: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd5a00 next 56 of size 2048 2021-03-06 13:51:29.925415: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6200 next 57 of size 2048 2021-03-06 13:51:29.925456: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6a00 next 58 of size 1024 2021-03-06 13:51:29.925499: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6e00 next 59 of size 2048 2021-03-06 13:51:29.925543: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd7600 next 60 of size 512 2021-03-06 13:51:29.925588: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd7800 next 61 of size 3538944 2021-03-06 13:51:29.925637: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037800 next 62 of size 512 2021-03-06 13:51:29.925685: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037a00 next 63 of size 1024 2021-03-06 13:51:29.925734: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037e00 next 64 of size 512 2021-03-06 13:51:29.925783: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038000 next 65 of size 512 2021-03-06 13:51:29.925825: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038200 next 66 of size 256 2021-03-06 13:51:29.925897: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038300 next 67 of size 1024 2021-03-06 13:51:29.925940: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038700 next 68 of size 1024 2021-03-06 13:51:29.925984: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038b00 next 69 of size 522240 2021-03-06 13:51:29.926054: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf030b8300 next 70 of size 1048576 2021-03-06 13:51:29.926100: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf031b8300 next 71 of size 2048 2021-03-06 13:51:29.926144: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf031b8b00 next 72 of size 4718592 2021-03-06 13:51:29.926188: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03638b00 next 73 of size 4718592 2021-03-06 13:51:29.926233: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab8b00 next 74 of size 1024 2021-03-06 13:51:29.926279: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab8f00 next 75 of size 2048 2021-03-06 13:51:29.926329: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab9700 next 76 of size 1024 2021-03-06 13:51:29.926390: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab9b00 next 77 of size 131072 2021-03-06 13:51:29.926460: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ad9b00 next 81 of size 18874368 2021-03-06 13:51:29.926533: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf04cd9b00 next 111 of size 5537792 2021-03-06 13:51:29.926600: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf05221b00 next 112 of size 11075584 2021-03-06 13:51:29.926668: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf05cb1b00 next 116 of size 27852800 2021-03-06 13:51:29.926740: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf07741b00 next 18446744073709551615 of size 526214400 2021-03-06 13:51:29.926805: I tensorflow/core/common_runtime/bfc_allocator.cc:917] Summary of in-use Chunks by size: 2021-03-06 13:51:29.926885: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 50 Chunks of size 256 totalling 12.5KiB 2021-03-06 13:51:29.926953: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 10 Chunks of size 512 totalling 5.0KiB 2021-03-06 13:51:29.927018: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 17 Chunks of size 1024 totalling 17.0KiB 2021-03-06 13:51:29.927085: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1280 totalling 1.2KiB 2021-03-06 13:51:29.927138: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1792 totalling 1.8KiB 2021-03-06 13:51:29.927185: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 10 Chunks of size 2048 totalling 20.0KiB 2021-03-06 13:51:29.927235: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 6 Chunks of size 4096 totalling 24.0KiB 2021-03-06 13:51:29.927288: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 18432 totalling 18.0KiB 2021-03-06 13:51:29.927339: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 73728 totalling 72.0KiB 2021-03-06 13:51:29.927396: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 104448 totalling 102.0KiB 2021-03-06 13:51:29.927491: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 131072 totalling 128.0KiB 2021-03-06 13:51:29.927547: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 261120 totalling 255.0KiB 2021-03-06 13:51:29.927608: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 294912 totalling 288.0KiB 2021-03-06 13:51:29.927673: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 522240 totalling 510.0KiB 2021-03-06 13:51:29.927739: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1048576 totalling 1.00MiB 2021-03-06 13:51:29.927807: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1179648 totalling 1.12MiB 2021-03-06 13:51:29.927875: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 3538944 totalling 3.38MiB 2021-03-06 13:51:29.927946: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 2 Chunks of size 4718592 totalling 9.00MiB 2021-03-06 13:51:29.928015: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 5537792 totalling 5.28MiB 2021-03-06 13:51:29.928084: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 7700224 totalling 7.34MiB 2021-03-06 13:51:29.928174: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 11075584 totalling 10.56MiB 2021-03-06 13:51:29.928229: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 18874368 totalling 18.00MiB 2021-03-06 13:51:29.928281: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 21188096 totalling 20.21MiB 2021-03-06 13:51:29.928340: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 27852800 totalling 26.56MiB 2021-03-06 13:51:29.928396: I tensorflow/core/common_runtime/bfc_allocator.cc:924] Sum Total of in-use chunks: 103.88MiB 2021-03-06 13:51:29.928443: I tensorflow/core/common_runtime/bfc_allocator.cc:926] total_region_allocated_bytes_: 639664128 memory_limit_: 639664128 available bytes: 0 curr_region_allocation_bytes_: 1279328256 2021-03-06 13:51:29.928502: I tensorflow/core/common_runtime/bfc_allocator.cc:932] Stats: Limit: 639664128 InUse: 108922624 MaxInUse: 223520000 NumAllocs: 174 MaxAllocSize: 66617344 2021-03-06 13:51:29.928643: W tensorflow/core/common_runtime/bfc_allocator.cc:427] ****x*************__________________________________________________________________________________ 2021-03-06 13:51:40.696345: W tensorflow/core/common_runtime/bfc_allocator.cc:419] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.04GiB (rounded to 1116110848). If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation. Current allocation summary follows. 2021-03-06 13:51:40.697266: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (256): Total Chunks: 50, Chunks in use: 50. 12.5KiB allocated for chunks. 12.5KiB in use in bin. 2.8KiB client-requested in use in bin. 2021-03-06 13:51:40.697800: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (512): Total Chunks: 10, Chunks in use: 10. 5.0KiB allocated for chunks. 5.0KiB in use in bin. 5.0KiB client-requested in use in bin. 2021-03-06 13:51:40.698729: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (1024): Total Chunks: 19, Chunks in use: 19. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 19.7KiB client-requested in use in bin. 2021-03-06 13:51:40.699178: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (2048): Total Chunks: 10, Chunks in use: 10. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin. 2021-03-06 13:51:40.699817: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (4096): Total Chunks: 6, Chunks in use: 6. 24.0KiB allocated for chunks. 24.0KiB in use in bin. 24.0KiB client-requested in use in bin. 2021-03-06 13:51:40.700827: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:40.701981: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (16384): Total Chunks: 1, Chunks in use: 1. 18.0KiB allocated for chunks. 18.0KiB in use in bin. 18.0KiB client-requested in use in bin. 2021-03-06 13:51:40.702869: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:40.703227: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (65536): Total Chunks: 1, Chunks in use: 1. 102.0KiB allocated for chunks. 102.0KiB in use in bin. 72.0KiB client-requested in use in bin. 2021-03-06 13:51:40.703578: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (131072): Total Chunks: 2, Chunks in use: 2. 383.0KiB allocated for chunks. 383.0KiB in use in bin. 383.0KiB client-requested in use in bin. 2021-03-06 13:51:40.703964: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (262144): Total Chunks: 4, Chunks in use: 3. 1.56MiB allocated for chunks. 1.06MiB in use in bin. 1.06MiB client-requested in use in bin. 2021-03-06 13:51:40.704287: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (524288): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:40.704758: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (1048576): Total Chunks: 2, Chunks in use: 2. 2.12MiB allocated for chunks. 2.12MiB in use in bin. 2.12MiB client-requested in use in bin. 2021-03-06 13:51:40.705215: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (2097152): Total Chunks: 3, Chunks in use: 3. 9.62MiB allocated for chunks. 9.62MiB in use in bin. 8.66MiB client-requested in use in bin. 2021-03-06 13:51:40.705835: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (4194304): Total Chunks: 4, Chunks in use: 4. 21.62MiB allocated for chunks. 21.62MiB in use in bin. 19.56MiB client-requested in use in bin. 2021-03-06 13:51:40.706602: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (8388608): Total Chunks: 1, Chunks in use: 0. 12.28MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:40.706954: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (16777216): Total Chunks: 2, Chunks in use: 2. 39.28MiB allocated for chunks. 39.28MiB in use in bin. 39.28MiB client-requested in use in bin. 2021-03-06 13:51:40.707478: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:40.707724: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:40.707840: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:40.707990: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (268435456): Total Chunks: 1, Chunks in use: 0. 522.96MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:40.708171: I tensorflow/core/common_runtime/bfc_allocator.cc:888] Bin for 1.04GiB was 256.00MiB, Chunk State: 2021-03-06 13:51:40.708423: I tensorflow/core/common_runtime/bfc_allocator.cc:894] Size: 522.96MiB | Requested Size: 39.77MiB | in_use: 0 | bin_num: 20, prev: Size: 21.28MiB | Requested Size: 21.28MiB | in_use: 1 | bin_num: -1 2021-03-06 13:51:40.708554: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Next region of size 639664128 2021-03-06 13:51:40.708677: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10000 next 1 of size 1280 2021-03-06 13:51:40.708792: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10500 next 2 of size 256 2021-03-06 13:51:40.708889: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10600 next 3 of size 256 2021-03-06 13:51:40.709083: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10700 next 4 of size 256 2021-03-06 13:51:40.709417: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10800 next 5 of size 256 2021-03-06 13:51:40.709578: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10900 next 42 of size 256 2021-03-06 13:51:40.709792: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10a00 next 51 of size 256 2021-03-06 13:51:40.709982: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10b00 next 52 of size 256 2021-03-06 13:51:40.710244: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10c00 next 46 of size 256 2021-03-06 13:51:40.710318: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10d00 next 44 of size 256 2021-03-06 13:51:40.710417: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10e00 next 110 of size 256 2021-03-06 13:51:40.710546: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10f00 next 6 of size 256 2021-03-06 13:51:40.710664: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11000 next 7 of size 256 2021-03-06 13:51:40.710752: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11100 next 8 of size 1024 2021-03-06 13:51:40.710890: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11500 next 9 of size 256 2021-03-06 13:51:40.711016: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11600 next 10 of size 1024 2021-03-06 13:51:40.711120: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11a00 next 11 of size 1024 2021-03-06 13:51:40.711243: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11e00 next 12 of size 512 2021-03-06 13:51:40.711377: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12000 next 13 of size 256 2021-03-06 13:51:40.711455: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12100 next 14 of size 256 2021-03-06 13:51:40.711596: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12200 next 15 of size 2048 2021-03-06 13:51:40.711719: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12a00 next 16 of size 256 2021-03-06 13:51:40.711821: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12b00 next 17 of size 256 2021-03-06 13:51:40.711941: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12c00 next 18 of size 4096 2021-03-06 13:51:40.712102: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b13c00 next 19 of size 2048 2021-03-06 13:51:40.712217: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b14400 next 20 of size 4096 2021-03-06 13:51:40.712313: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15400 next 21 of size 256 2021-03-06 13:51:40.712452: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15500 next 22 of size 256 2021-03-06 13:51:40.712588: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15600 next 23 of size 256 2021-03-06 13:51:40.712722: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15700 next 24 of size 256 2021-03-06 13:51:40.712859: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15800 next 25 of size 512 2021-03-06 13:51:40.712977: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15a00 next 26 of size 256 2021-03-06 13:51:40.713093: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15b00 next 27 of size 256 2021-03-06 13:51:40.713207: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15c00 next 28 of size 256 2021-03-06 13:51:40.713344: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15d00 next 29 of size 256 2021-03-06 13:51:40.713459: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15e00 next 30 of size 256 2021-03-06 13:51:40.713584: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15f00 next 31 of size 256 2021-03-06 13:51:40.713696: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16000 next 32 of size 256 2021-03-06 13:51:40.713806: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16100 next 33 of size 256 2021-03-06 13:51:40.713914: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16200 next 34 of size 256 2021-03-06 13:51:40.714170: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16300 next 35 of size 256 2021-03-06 13:51:40.714275: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16400 next 36 of size 256 2021-03-06 13:51:40.714326: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16500 next 37 of size 256 2021-03-06 13:51:40.714411: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16600 next 38 of size 256 2021-03-06 13:51:40.714494: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16700 next 39 of size 256 2021-03-06 13:51:40.714542: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16800 next 40 of size 256 2021-03-06 13:51:40.714613: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16900 next 41 of size 256 2021-03-06 13:51:40.714713: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16a00 next 113 of size 256 2021-03-06 13:51:40.714807: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16b00 next 116 of size 294912 2021-03-06 13:51:40.714893: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf00b5eb00 next 43 of size 521984 2021-03-06 13:51:40.714982: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00bde200 next 78 of size 1024 2021-03-06 13:51:40.715055: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00bde600 next 79 of size 261120 2021-03-06 13:51:40.715142: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c1e200 next 80 of size 256 2021-03-06 13:51:40.715228: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c1e300 next 95 of size 294912 2021-03-06 13:51:40.715306: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c66300 next 97 of size 1179648 2021-03-06 13:51:40.715399: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00d86300 next 45 of size 7700224 2021-03-06 13:51:40.715459: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de200 next 50 of size 256 2021-03-06 13:51:40.715552: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de300 next 103 of size 256 2021-03-06 13:51:40.715645: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de400 next 104 of size 4096 2021-03-06 13:51:40.715726: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014df400 next 105 of size 4096 2021-03-06 13:51:40.715812: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e0400 next 106 of size 256 2021-03-06 13:51:40.715869: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e0500 next 107 of size 4096 2021-03-06 13:51:40.715960: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e1500 next 108 of size 2048 2021-03-06 13:51:40.716048: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e1d00 next 109 of size 1024 2021-03-06 13:51:40.716153: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e2100 next 47 of size 3783936 2021-03-06 13:51:40.716240: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187de00 next 82 of size 2048 2021-03-06 13:51:40.716332: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187e600 next 83 of size 512 2021-03-06 13:51:40.716418: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187e800 next 84 of size 1024 2021-03-06 13:51:40.716506: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ec00 next 85 of size 256 2021-03-06 13:51:40.716576: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ed00 next 86 of size 256 2021-03-06 13:51:40.716661: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ee00 next 87 of size 1024 2021-03-06 13:51:40.716810: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187f200 next 88 of size 1024 2021-03-06 13:51:40.716932: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187f600 next 89 of size 1792 2021-03-06 13:51:40.717067: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187fd00 next 90 of size 18432 2021-03-06 13:51:40.717159: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01884500 next 91 of size 256 2021-03-06 13:51:40.717247: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01884600 next 48 of size 104448 2021-03-06 13:51:40.717339: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189de00 next 49 of size 1024 2021-03-06 13:51:40.717430: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e200 next 92 of size 256 2021-03-06 13:51:40.717512: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e300 next 93 of size 256 2021-03-06 13:51:40.717599: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e400 next 94 of size 256 2021-03-06 13:51:40.717659: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e500 next 96 of size 1024 2021-03-06 13:51:40.717746: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e900 next 98 of size 512 2021-03-06 13:51:40.717836: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189eb00 next 99 of size 256 2021-03-06 13:51:40.717932: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189ec00 next 100 of size 2048 2021-03-06 13:51:40.718060: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f400 next 101 of size 512 2021-03-06 13:51:40.718136: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f600 next 102 of size 512 2021-03-06 13:51:40.718171: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f800 next 111 of size 2768896 2021-03-06 13:51:40.718222: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01b43800 next 114 of size 5537792 2021-03-06 13:51:40.718259: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf0208b800 next 53 of size 12881408 2021-03-06 13:51:40.718302: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd4600 next 54 of size 4096 2021-03-06 13:51:40.718334: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd5600 next 55 of size 1024 2021-03-06 13:51:40.718366: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd5a00 next 56 of size 2048 2021-03-06 13:51:40.718423: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6200 next 57 of size 2048 2021-03-06 13:51:40.718458: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6a00 next 58 of size 1024 2021-03-06 13:51:40.718547: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6e00 next 59 of size 2048 2021-03-06 13:51:40.718584: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd7600 next 60 of size 512 2021-03-06 13:51:40.718657: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd7800 next 61 of size 3538944 2021-03-06 13:51:40.718712: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037800 next 62 of size 512 2021-03-06 13:51:40.718759: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037a00 next 63 of size 1024 2021-03-06 13:51:40.718845: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037e00 next 64 of size 512 2021-03-06 13:51:40.718888: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038000 next 65 of size 512 2021-03-06 13:51:40.718952: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038200 next 66 of size 256 2021-03-06 13:51:40.718995: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038300 next 67 of size 1024 2021-03-06 13:51:40.719063: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038700 next 68 of size 1024 2021-03-06 13:51:40.719133: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038b00 next 69 of size 522240 2021-03-06 13:51:40.719186: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf030b8300 next 70 of size 1048576 2021-03-06 13:51:40.719255: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf031b8300 next 71 of size 2048 2021-03-06 13:51:40.719339: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf031b8b00 next 72 of size 4718592 2021-03-06 13:51:40.719386: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03638b00 next 73 of size 4718592 2021-03-06 13:51:40.719453: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab8b00 next 74 of size 1024 2021-03-06 13:51:40.719520: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab8f00 next 75 of size 2048 2021-03-06 13:51:40.719572: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab9700 next 76 of size 1024 2021-03-06 13:51:40.719641: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab9b00 next 77 of size 131072 2021-03-06 13:51:40.719730: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ad9b00 next 81 of size 18874368 2021-03-06 13:51:40.719789: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf04cd9b00 next 118 of size 22315008 2021-03-06 13:51:40.719864: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf06221b00 next 18446744073709551615 of size 548365568 2021-03-06 13:51:40.719929: I tensorflow/core/common_runtime/bfc_allocator.cc:917] Summary of in-use Chunks by size: 2021-03-06 13:51:40.720010: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 50 Chunks of size 256 totalling 12.5KiB 2021-03-06 13:51:40.720071: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 10 Chunks of size 512 totalling 5.0KiB 2021-03-06 13:51:40.720135: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 17 Chunks of size 1024 totalling 17.0KiB 2021-03-06 13:51:40.720170: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1280 totalling 1.2KiB 2021-03-06 13:51:40.720217: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1792 totalling 1.8KiB 2021-03-06 13:51:40.720250: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 10 Chunks of size 2048 totalling 20.0KiB 2021-03-06 13:51:40.720294: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 6 Chunks of size 4096 totalling 24.0KiB 2021-03-06 13:51:40.720329: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 18432 totalling 18.0KiB 2021-03-06 13:51:40.720361: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 104448 totalling 102.0KiB 2021-03-06 13:51:40.720405: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 131072 totalling 128.0KiB 2021-03-06 13:51:40.720438: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 261120 totalling 255.0KiB 2021-03-06 13:51:40.720483: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 2 Chunks of size 294912 totalling 576.0KiB 2021-03-06 13:51:40.720521: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 522240 totalling 510.0KiB 2021-03-06 13:51:40.720570: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1048576 totalling 1.00MiB 2021-03-06 13:51:40.720611: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1179648 totalling 1.12MiB 2021-03-06 13:51:40.720662: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 2768896 totalling 2.64MiB 2021-03-06 13:51:40.720705: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 3538944 totalling 3.38MiB 2021-03-06 13:51:40.720761: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 3783936 totalling 3.61MiB 2021-03-06 13:51:40.720809: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 2 Chunks of size 4718592 totalling 9.00MiB 2021-03-06 13:51:40.720869: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 5537792 totalling 5.28MiB 2021-03-06 13:51:40.720914: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 7700224 totalling 7.34MiB 2021-03-06 13:51:40.720974: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 18874368 totalling 18.00MiB 2021-03-06 13:51:40.721023: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 22315008 totalling 21.28MiB 2021-03-06 13:51:40.721116: I tensorflow/core/common_runtime/bfc_allocator.cc:924] Sum Total of in-use chunks: 74.29MiB 2021-03-06 13:51:40.721212: I tensorflow/core/common_runtime/bfc_allocator.cc:926] total_region_allocated_bytes_: 639664128 memory_limit_: 639664128 available bytes: 0 curr_region_allocation_bytes_: 1279328256 2021-03-06 13:51:40.721269: I tensorflow/core/common_runtime/bfc_allocator.cc:932] Stats: Limit: 639664128 InUse: 77895168 MaxInUse: 223520000 NumAllocs: 196 MaxAllocSize: 66617344 2021-03-06 13:51:40.721479: W tensorflow/core/common_runtime/bfc_allocator.cc:427] ****_**********_____________________________________________________________________________________ 2021-03-06 13:51:51.252832: W tensorflow/core/common_runtime/bfc_allocator.cc:419] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.05GiB (rounded to 1129578496). If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation. Current allocation summary follows. 2021-03-06 13:51:51.253319: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (256): Total Chunks: 50, Chunks in use: 50. 12.5KiB allocated for chunks. 12.5KiB in use in bin. 2.8KiB client-requested in use in bin. 2021-03-06 13:51:51.253523: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (512): Total Chunks: 10, Chunks in use: 10. 5.0KiB allocated for chunks. 5.0KiB in use in bin. 5.0KiB client-requested in use in bin. 2021-03-06 13:51:51.253712: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (1024): Total Chunks: 19, Chunks in use: 19. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 19.7KiB client-requested in use in bin. 2021-03-06 13:51:51.253898: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (2048): Total Chunks: 10, Chunks in use: 10. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin. 2021-03-06 13:51:51.254229: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (4096): Total Chunks: 6, Chunks in use: 6. 24.0KiB allocated for chunks. 24.0KiB in use in bin. 24.0KiB client-requested in use in bin. 2021-03-06 13:51:51.254424: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:51.254605: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (16384): Total Chunks: 1, Chunks in use: 1. 18.0KiB allocated for chunks. 18.0KiB in use in bin. 18.0KiB client-requested in use in bin. 2021-03-06 13:51:51.254772: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:51.254960: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (65536): Total Chunks: 1, Chunks in use: 1. 102.0KiB allocated for chunks. 102.0KiB in use in bin. 72.0KiB client-requested in use in bin. 2021-03-06 13:51:51.255150: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (131072): Total Chunks: 2, Chunks in use: 2. 383.0KiB allocated for chunks. 383.0KiB in use in bin. 383.0KiB client-requested in use in bin. 2021-03-06 13:51:51.255339: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (262144): Total Chunks: 2, Chunks in use: 2. 798.0KiB allocated for chunks. 798.0KiB in use in bin. 798.0KiB client-requested in use in bin. 2021-03-06 13:51:51.255517: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (524288): Total Chunks: 1, Chunks in use: 0. 797.8KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:51.255693: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (1048576): Total Chunks: 4, Chunks in use: 4. 4.57MiB allocated for chunks. 4.57MiB in use in bin. 4.57MiB client-requested in use in bin. 2021-03-06 13:51:51.255879: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (2097152): Total Chunks: 4, Chunks in use: 4. 11.88MiB allocated for chunks. 11.88MiB in use in bin. 9.98MiB client-requested in use in bin. 2021-03-06 13:51:51.256056: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (4194304): Total Chunks: 2, Chunks in use: 2. 9.00MiB allocated for chunks. 9.00MiB in use in bin. 9.00MiB client-requested in use in bin. 2021-03-06 13:51:51.256219: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:51.256400: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (16777216): Total Chunks: 2, Chunks in use: 2. 38.21MiB allocated for chunks. 38.21MiB in use in bin. 36.64MiB client-requested in use in bin. 2021-03-06 13:51:51.256566: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:51.256724: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:51.256883: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:51.257055: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (268435456): Total Chunks: 1, Chunks in use: 0. 544.24MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:51:51.257229: I tensorflow/core/common_runtime/bfc_allocator.cc:888] Bin for 1.05GiB was 256.00MiB, Chunk State: 2021-03-06 13:51:51.257442: I tensorflow/core/common_runtime/bfc_allocator.cc:894] Size: 544.24MiB | Requested Size: 27.88MiB | in_use: 0 | bin_num: 20, prev: Size: 18.00MiB | Requested Size: 18.00MiB | in_use: 1 | bin_num: -1 2021-03-06 13:51:51.257593: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Next region of size 639664128 2021-03-06 13:51:51.257749: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10000 next 1 of size 1280 2021-03-06 13:51:51.257893: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10500 next 2 of size 256 2021-03-06 13:51:51.258154: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10600 next 3 of size 256 2021-03-06 13:51:51.258310: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10700 next 4 of size 256 2021-03-06 13:51:51.258439: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10800 next 5 of size 256 2021-03-06 13:51:51.258508: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10900 next 42 of size 256 2021-03-06 13:51:51.258573: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10a00 next 51 of size 256 2021-03-06 13:51:51.258634: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10b00 next 52 of size 256 2021-03-06 13:51:51.260498: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10c00 next 46 of size 256 2021-03-06 13:51:51.260757: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10d00 next 44 of size 256 2021-03-06 13:51:51.260820: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10e00 next 110 of size 256 2021-03-06 13:51:51.260870: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10f00 next 6 of size 256 2021-03-06 13:51:51.260982: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11000 next 7 of size 256 2021-03-06 13:51:51.261097: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11100 next 8 of size 1024 2021-03-06 13:51:51.261207: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11500 next 9 of size 256 2021-03-06 13:51:51.261329: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11600 next 10 of size 1024 2021-03-06 13:51:51.261459: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11a00 next 11 of size 1024 2021-03-06 13:51:51.261602: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11e00 next 12 of size 512 2021-03-06 13:51:51.261743: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12000 next 13 of size 256 2021-03-06 13:51:51.261887: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12100 next 14 of size 256 2021-03-06 13:51:51.262116: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12200 next 15 of size 2048 2021-03-06 13:51:51.262269: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12a00 next 16 of size 256 2021-03-06 13:51:51.262412: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12b00 next 17 of size 256 2021-03-06 13:51:51.262548: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12c00 next 18 of size 4096 2021-03-06 13:51:51.262765: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b13c00 next 19 of size 2048 2021-03-06 13:51:51.262855: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b14400 next 20 of size 4096 2021-03-06 13:51:51.262928: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15400 next 21 of size 256 2021-03-06 13:51:51.262994: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15500 next 22 of size 256 2021-03-06 13:51:51.263061: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15600 next 23 of size 256 2021-03-06 13:51:51.263131: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15700 next 24 of size 256 2021-03-06 13:51:51.263208: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15800 next 25 of size 512 2021-03-06 13:51:51.263294: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15a00 next 26 of size 256 2021-03-06 13:51:51.263384: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15b00 next 27 of size 256 2021-03-06 13:51:51.263479: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15c00 next 28 of size 256 2021-03-06 13:51:51.263579: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15d00 next 29 of size 256 2021-03-06 13:51:51.263682: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15e00 next 30 of size 256 2021-03-06 13:51:51.263789: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15f00 next 31 of size 256 2021-03-06 13:51:51.263938: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16000 next 32 of size 256 2021-03-06 13:51:51.264044: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16100 next 33 of size 256 2021-03-06 13:51:51.264149: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16200 next 34 of size 256 2021-03-06 13:51:51.264253: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16300 next 35 of size 256 2021-03-06 13:51:51.264355: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16400 next 36 of size 256 2021-03-06 13:51:51.264460: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16500 next 37 of size 256 2021-03-06 13:51:51.264564: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16600 next 38 of size 256 2021-03-06 13:51:51.264667: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16700 next 39 of size 256 2021-03-06 13:51:51.264791: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16800 next 40 of size 256 2021-03-06 13:51:51.264890: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16900 next 41 of size 256 2021-03-06 13:51:51.264955: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16a00 next 113 of size 256 2021-03-06 13:51:51.265015: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf00b16b00 next 43 of size 816896 2021-03-06 13:51:51.265072: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00bde200 next 78 of size 1024 2021-03-06 13:51:51.265130: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00bde600 next 79 of size 261120 2021-03-06 13:51:51.265226: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c1e200 next 80 of size 256 2021-03-06 13:51:51.265283: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c1e300 next 95 of size 294912 2021-03-06 13:51:51.265331: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c66300 next 97 of size 1179648 2021-03-06 13:51:51.265371: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00d86300 next 111 of size 2768896 2021-03-06 13:51:51.265444: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0102a300 next 118 of size 1179648 2021-03-06 13:51:51.265526: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0114a300 next 45 of size 3751680 2021-03-06 13:51:51.265611: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de200 next 50 of size 256 2021-03-06 13:51:51.265695: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de300 next 103 of size 256 2021-03-06 13:51:51.265784: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de400 next 104 of size 4096 2021-03-06 13:51:51.265902: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014df400 next 105 of size 4096 2021-03-06 13:51:51.265960: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e0400 next 106 of size 256 2021-03-06 13:51:51.266030: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e0500 next 107 of size 4096 2021-03-06 13:51:51.266085: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e1500 next 108 of size 2048 2021-03-06 13:51:51.266136: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e1d00 next 109 of size 1024 2021-03-06 13:51:51.266202: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e2100 next 112 of size 1384448 2021-03-06 13:51:51.266259: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01634100 next 47 of size 2399488 2021-03-06 13:51:51.266319: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187de00 next 82 of size 2048 2021-03-06 13:51:51.266397: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187e600 next 83 of size 512 2021-03-06 13:51:51.266475: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187e800 next 84 of size 1024 2021-03-06 13:51:51.266523: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ec00 next 85 of size 256 2021-03-06 13:51:51.266562: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ed00 next 86 of size 256 2021-03-06 13:51:51.266609: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ee00 next 87 of size 1024 2021-03-06 13:51:51.266658: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187f200 next 88 of size 1024 2021-03-06 13:51:51.266698: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187f600 next 89 of size 1792 2021-03-06 13:51:51.266775: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187fd00 next 90 of size 18432 2021-03-06 13:51:51.266853: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01884500 next 91 of size 256 2021-03-06 13:51:51.266944: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01884600 next 48 of size 104448 2021-03-06 13:51:51.267020: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189de00 next 49 of size 1024 2021-03-06 13:51:51.267073: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e200 next 92 of size 256 2021-03-06 13:51:51.267125: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e300 next 93 of size 256 2021-03-06 13:51:51.267168: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e400 next 94 of size 256 2021-03-06 13:51:51.267210: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e500 next 96 of size 1024 2021-03-06 13:51:51.267252: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e900 next 98 of size 512 2021-03-06 13:51:51.267306: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189eb00 next 99 of size 256 2021-03-06 13:51:51.267351: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189ec00 next 100 of size 2048 2021-03-06 13:51:51.267383: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f400 next 101 of size 512 2021-03-06 13:51:51.267408: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f600 next 102 of size 512 2021-03-06 13:51:51.267434: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f800 next 53 of size 21188096 2021-03-06 13:51:51.267468: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd4600 next 54 of size 4096 2021-03-06 13:51:51.267497: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd5600 next 55 of size 1024 2021-03-06 13:51:51.267525: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd5a00 next 56 of size 2048 2021-03-06 13:51:51.267560: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6200 next 57 of size 2048 2021-03-06 13:51:51.267599: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6a00 next 58 of size 1024 2021-03-06 13:51:51.267630: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6e00 next 59 of size 2048 2021-03-06 13:51:51.267661: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd7600 next 60 of size 512 2021-03-06 13:51:51.267701: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd7800 next 61 of size 3538944 2021-03-06 13:51:51.267743: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037800 next 62 of size 512 2021-03-06 13:51:51.267785: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037a00 next 63 of size 1024 2021-03-06 13:51:51.267818: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037e00 next 64 of size 512 2021-03-06 13:51:51.267852: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038000 next 65 of size 512 2021-03-06 13:51:51.267893: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038200 next 66 of size 256 2021-03-06 13:51:51.267935: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038300 next 67 of size 1024 2021-03-06 13:51:51.267978: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038700 next 68 of size 1024 2021-03-06 13:51:51.268021: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038b00 next 69 of size 522240 2021-03-06 13:51:51.268073: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf030b8300 next 70 of size 1048576 2021-03-06 13:51:51.268118: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf031b8300 next 71 of size 2048 2021-03-06 13:51:51.268217: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf031b8b00 next 72 of size 4718592 2021-03-06 13:51:51.268271: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03638b00 next 73 of size 4718592 2021-03-06 13:51:51.268313: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab8b00 next 74 of size 1024 2021-03-06 13:51:51.268353: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab8f00 next 75 of size 2048 2021-03-06 13:51:51.268391: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab9700 next 76 of size 1024 2021-03-06 13:51:51.268434: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab9b00 next 77 of size 131072 2021-03-06 13:51:51.268476: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ad9b00 next 81 of size 18874368 2021-03-06 13:51:51.268520: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf04cd9b00 next 18446744073709551615 of size 570680576 2021-03-06 13:51:51.268564: I tensorflow/core/common_runtime/bfc_allocator.cc:917] Summary of in-use Chunks by size: 2021-03-06 13:51:51.268620: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 50 Chunks of size 256 totalling 12.5KiB 2021-03-06 13:51:51.268674: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 10 Chunks of size 512 totalling 5.0KiB 2021-03-06 13:51:51.268732: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 17 Chunks of size 1024 totalling 17.0KiB 2021-03-06 13:51:51.268786: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1280 totalling 1.2KiB 2021-03-06 13:51:51.268858: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1792 totalling 1.8KiB 2021-03-06 13:51:51.268919: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 10 Chunks of size 2048 totalling 20.0KiB 2021-03-06 13:51:51.268979: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 6 Chunks of size 4096 totalling 24.0KiB 2021-03-06 13:51:51.269046: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 18432 totalling 18.0KiB 2021-03-06 13:51:51.269125: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 104448 totalling 102.0KiB 2021-03-06 13:51:51.269173: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 131072 totalling 128.0KiB 2021-03-06 13:51:51.269217: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 261120 totalling 255.0KiB 2021-03-06 13:51:51.269256: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 294912 totalling 288.0KiB 2021-03-06 13:51:51.269293: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 522240 totalling 510.0KiB 2021-03-06 13:51:51.269333: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1048576 totalling 1.00MiB 2021-03-06 13:51:51.269370: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 2 Chunks of size 1179648 totalling 2.25MiB 2021-03-06 13:51:51.269417: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1384448 totalling 1.32MiB 2021-03-06 13:51:51.269455: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 2399488 totalling 2.29MiB 2021-03-06 13:51:51.269494: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 2768896 totalling 2.64MiB 2021-03-06 13:51:51.269538: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 3538944 totalling 3.38MiB 2021-03-06 13:51:51.269584: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 3751680 totalling 3.58MiB 2021-03-06 13:51:51.269631: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 2 Chunks of size 4718592 totalling 9.00MiB 2021-03-06 13:51:51.269681: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 18874368 totalling 18.00MiB 2021-03-06 13:51:51.269733: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 21188096 totalling 20.21MiB 2021-03-06 13:51:51.269784: I tensorflow/core/common_runtime/bfc_allocator.cc:924] Sum Total of in-use chunks: 65.01MiB 2021-03-06 13:51:51.269836: I tensorflow/core/common_runtime/bfc_allocator.cc:926] total_region_allocated_bytes_: 639664128 memory_limit_: 639664128 available bytes: 0 curr_region_allocation_bytes_: 1279328256 2021-03-06 13:51:51.269895: I tensorflow/core/common_runtime/bfc_allocator.cc:932] Stats: Limit: 639664128 InUse: 68166656 MaxInUse: 394892800 NumAllocs: 221 MaxAllocSize: 316997632 2021-03-06 13:51:51.270087: W tensorflow/core/common_runtime/bfc_allocator.cc:427] ***********_________________________________________________________________________________________ 2021-03-06 13:52:01.948374: W tensorflow/core/common_runtime/bfc_allocator.cc:419] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.08GiB (rounded to 1164574720). If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation. Current allocation summary follows. 2021-03-06 13:52:01.949259: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (256): Total Chunks: 50, Chunks in use: 50. 12.5KiB allocated for chunks. 12.5KiB in use in bin. 2.8KiB client-requested in use in bin. 2021-03-06 13:52:01.949791: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (512): Total Chunks: 10, Chunks in use: 10. 5.0KiB allocated for chunks. 5.0KiB in use in bin. 5.0KiB client-requested in use in bin. 2021-03-06 13:52:01.950645: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (1024): Total Chunks: 19, Chunks in use: 19. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 19.7KiB client-requested in use in bin. 2021-03-06 13:52:01.951142: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (2048): Total Chunks: 10, Chunks in use: 10. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin. 2021-03-06 13:52:01.951630: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (4096): Total Chunks: 6, Chunks in use: 6. 24.0KiB allocated for chunks. 24.0KiB in use in bin. 24.0KiB client-requested in use in bin. 2021-03-06 13:52:01.952061: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:01.952563: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (16384): Total Chunks: 1, Chunks in use: 1. 18.0KiB allocated for chunks. 18.0KiB in use in bin. 18.0KiB client-requested in use in bin. 2021-03-06 13:52:01.953000: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:01.953490: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (65536): Total Chunks: 1, Chunks in use: 1. 102.0KiB allocated for chunks. 102.0KiB in use in bin. 72.0KiB client-requested in use in bin. 2021-03-06 13:52:01.953920: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (131072): Total Chunks: 2, Chunks in use: 2. 383.0KiB allocated for chunks. 383.0KiB in use in bin. 383.0KiB client-requested in use in bin. 2021-03-06 13:52:01.954605: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (262144): Total Chunks: 2, Chunks in use: 2. 798.0KiB allocated for chunks. 798.0KiB in use in bin. 798.0KiB client-requested in use in bin. 2021-03-06 13:52:01.954760: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (524288): Total Chunks: 3, Chunks in use: 2. 2.10MiB allocated for chunks. 1.44MiB in use in bin. 1.32MiB client-requested in use in bin. 2021-03-06 13:52:01.954910: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (1048576): Total Chunks: 4, Chunks in use: 4. 4.77MiB allocated for chunks. 4.77MiB in use in bin. 4.77MiB client-requested in use in bin. 2021-03-06 13:52:01.955058: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (2097152): Total Chunks: 2, Chunks in use: 1. 5.66MiB allocated for chunks. 3.38MiB in use in bin. 3.38MiB client-requested in use in bin. 2021-03-06 13:52:01.955281: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (4194304): Total Chunks: 4, Chunks in use: 4. 18.20MiB allocated for chunks. 18.20MiB in use in bin. 16.14MiB client-requested in use in bin. 2021-03-06 13:52:01.955502: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (8388608): Total Chunks: 1, Chunks in use: 0. 15.71MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:01.955730: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (16777216): Total Chunks: 2, Chunks in use: 2. 35.32MiB allocated for chunks. 35.32MiB in use in bin. 35.32MiB client-requested in use in bin. 2021-03-06 13:52:01.955902: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:01.956070: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:01.956277: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:01.956494: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (268435456): Total Chunks: 1, Chunks in use: 0. 526.92MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:01.956719: I tensorflow/core/common_runtime/bfc_allocator.cc:888] Bin for 1.08GiB was 256.00MiB, Chunk State: 2021-03-06 13:52:01.956953: I tensorflow/core/common_runtime/bfc_allocator.cc:894] Size: 526.92MiB | Requested Size: 21.94MiB | in_use: 0 | bin_num: 20, prev: Size: 17.32MiB | Requested Size: 17.32MiB | in_use: 1 | bin_num: -1 2021-03-06 13:52:01.957143: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Next region of size 639664128 2021-03-06 13:52:01.957339: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10000 next 1 of size 1280 2021-03-06 13:52:01.957529: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10500 next 2 of size 256 2021-03-06 13:52:01.957713: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10600 next 3 of size 256 2021-03-06 13:52:01.957859: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10700 next 4 of size 256 2021-03-06 13:52:01.958314: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10800 next 5 of size 256 2021-03-06 13:52:01.958453: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10900 next 42 of size 256 2021-03-06 13:52:01.958523: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10a00 next 51 of size 256 2021-03-06 13:52:01.958607: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10b00 next 52 of size 256 2021-03-06 13:52:01.958688: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10c00 next 46 of size 256 2021-03-06 13:52:01.958753: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10d00 next 44 of size 256 2021-03-06 13:52:01.958821: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10e00 next 110 of size 256 2021-03-06 13:52:01.958920: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10f00 next 6 of size 256 2021-03-06 13:52:01.959024: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11000 next 7 of size 256 2021-03-06 13:52:01.959122: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11100 next 8 of size 1024 2021-03-06 13:52:01.959217: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11500 next 9 of size 256 2021-03-06 13:52:01.959330: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11600 next 10 of size 1024 2021-03-06 13:52:01.959424: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11a00 next 11 of size 1024 2021-03-06 13:52:01.959518: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11e00 next 12 of size 512 2021-03-06 13:52:01.959634: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12000 next 13 of size 256 2021-03-06 13:52:01.959729: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12100 next 14 of size 256 2021-03-06 13:52:01.959822: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12200 next 15 of size 2048 2021-03-06 13:52:01.959938: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12a00 next 16 of size 256 2021-03-06 13:52:01.960035: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12b00 next 17 of size 256 2021-03-06 13:52:01.960128: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12c00 next 18 of size 4096 2021-03-06 13:52:01.960243: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b13c00 next 19 of size 2048 2021-03-06 13:52:01.960361: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b14400 next 20 of size 4096 2021-03-06 13:52:01.960456: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15400 next 21 of size 256 2021-03-06 13:52:01.960547: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15500 next 22 of size 256 2021-03-06 13:52:01.960659: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15600 next 23 of size 256 2021-03-06 13:52:01.960775: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15700 next 24 of size 256 2021-03-06 13:52:01.960870: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15800 next 25 of size 512 2021-03-06 13:52:01.960961: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15a00 next 26 of size 256 2021-03-06 13:52:01.961074: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15b00 next 27 of size 256 2021-03-06 13:52:01.961167: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15c00 next 28 of size 256 2021-03-06 13:52:01.961259: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15d00 next 29 of size 256 2021-03-06 13:52:01.961370: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15e00 next 30 of size 256 2021-03-06 13:52:01.961487: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15f00 next 31 of size 256 2021-03-06 13:52:01.961581: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16000 next 32 of size 256 2021-03-06 13:52:01.961673: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16100 next 33 of size 256 2021-03-06 13:52:01.961784: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16200 next 34 of size 256 2021-03-06 13:52:01.961896: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16300 next 35 of size 256 2021-03-06 13:52:01.962081: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16400 next 36 of size 256 2021-03-06 13:52:01.962191: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16500 next 37 of size 256 2021-03-06 13:52:01.962271: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16600 next 38 of size 256 2021-03-06 13:52:01.962314: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16700 next 39 of size 256 2021-03-06 13:52:01.962353: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16800 next 40 of size 256 2021-03-06 13:52:01.962393: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16900 next 41 of size 256 2021-03-06 13:52:01.962436: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16a00 next 113 of size 256 2021-03-06 13:52:01.962480: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16b00 next 43 of size 816896 2021-03-06 13:52:01.962525: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00bde200 next 78 of size 1024 2021-03-06 13:52:01.962564: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00bde600 next 79 of size 261120 2021-03-06 13:52:01.962607: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c1e200 next 80 of size 256 2021-03-06 13:52:01.962648: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c1e300 next 95 of size 294912 2021-03-06 13:52:01.962690: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c66300 next 97 of size 1179648 2021-03-06 13:52:01.962735: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00d86300 next 112 of size 1384448 2021-03-06 13:52:01.962794: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00ed8300 next 114 of size 692224 2021-03-06 13:52:01.962843: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf00f81300 next 111 of size 692224 2021-03-06 13:52:01.962893: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0102a300 next 45 of size 4931328 2021-03-06 13:52:01.962945: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de200 next 50 of size 256 2021-03-06 13:52:01.962997: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de300 next 103 of size 256 2021-03-06 13:52:01.963063: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de400 next 104 of size 4096 2021-03-06 13:52:01.963134: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014df400 next 105 of size 4096 2021-03-06 13:52:01.963211: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e0400 next 106 of size 256 2021-03-06 13:52:01.963269: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e0500 next 107 of size 4096 2021-03-06 13:52:01.963325: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e1500 next 108 of size 2048 2021-03-06 13:52:01.963421: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e1d00 next 109 of size 1024 2021-03-06 13:52:01.963492: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e2100 next 116 of size 1384448 2021-03-06 13:52:01.963548: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf01634100 next 47 of size 2399488 2021-03-06 13:52:01.963604: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187de00 next 82 of size 2048 2021-03-06 13:52:01.963661: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187e600 next 83 of size 512 2021-03-06 13:52:01.963736: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187e800 next 84 of size 1024 2021-03-06 13:52:01.963807: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ec00 next 85 of size 256 2021-03-06 13:52:01.963865: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ed00 next 86 of size 256 2021-03-06 13:52:01.963954: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ee00 next 87 of size 1024 2021-03-06 13:52:01.964028: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187f200 next 88 of size 1024 2021-03-06 13:52:01.964085: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187f600 next 89 of size 1792 2021-03-06 13:52:01.964165: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187fd00 next 90 of size 18432 2021-03-06 13:52:01.964226: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01884500 next 91 of size 256 2021-03-06 13:52:01.964299: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01884600 next 48 of size 104448 2021-03-06 13:52:01.964372: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189de00 next 49 of size 1024 2021-03-06 13:52:01.964428: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e200 next 92 of size 256 2021-03-06 13:52:01.964500: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e300 next 93 of size 256 2021-03-06 13:52:01.964569: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e400 next 94 of size 256 2021-03-06 13:52:01.964625: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e500 next 96 of size 1024 2021-03-06 13:52:01.964682: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e900 next 98 of size 512 2021-03-06 13:52:01.964752: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189eb00 next 99 of size 256 2021-03-06 13:52:01.964822: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189ec00 next 100 of size 2048 2021-03-06 13:52:01.964879: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f400 next 101 of size 512 2021-03-06 13:52:01.964945: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f600 next 102 of size 512 2021-03-06 13:52:01.965016: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f800 next 118 of size 4718592 2021-03-06 13:52:01.965076: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf01d1f800 next 53 of size 16469504 2021-03-06 13:52:01.965147: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd4600 next 54 of size 4096 2021-03-06 13:52:01.965216: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd5600 next 55 of size 1024 2021-03-06 13:52:01.965273: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd5a00 next 56 of size 2048 2021-03-06 13:52:01.965339: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6200 next 57 of size 2048 2021-03-06 13:52:01.965406: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6a00 next 58 of size 1024 2021-03-06 13:52:01.965475: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6e00 next 59 of size 2048 2021-03-06 13:52:01.965532: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd7600 next 60 of size 512 2021-03-06 13:52:01.965601: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd7800 next 61 of size 3538944 2021-03-06 13:52:01.965670: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037800 next 62 of size 512 2021-03-06 13:52:01.965737: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037a00 next 63 of size 1024 2021-03-06 13:52:01.965806: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037e00 next 64 of size 512 2021-03-06 13:52:01.965874: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038000 next 65 of size 512 2021-03-06 13:52:01.965941: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038200 next 66 of size 256 2021-03-06 13:52:01.966008: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038300 next 67 of size 1024 2021-03-06 13:52:01.966132: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038700 next 68 of size 1024 2021-03-06 13:52:01.966178: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038b00 next 69 of size 522240 2021-03-06 13:52:01.966212: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf030b8300 next 70 of size 1048576 2021-03-06 13:52:01.966244: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf031b8300 next 71 of size 2048 2021-03-06 13:52:01.966274: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf031b8b00 next 72 of size 4718592 2021-03-06 13:52:01.966302: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03638b00 next 73 of size 4718592 2021-03-06 13:52:01.966330: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab8b00 next 74 of size 1024 2021-03-06 13:52:01.966357: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab8f00 next 75 of size 2048 2021-03-06 13:52:01.966385: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab9700 next 76 of size 1024 2021-03-06 13:52:01.966414: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab9b00 next 77 of size 131072 2021-03-06 13:52:01.966444: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ad9b00 next 81 of size 18874368 2021-03-06 13:52:01.966474: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf04cd9b00 next 119 of size 18161664 2021-03-06 13:52:01.966505: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf05e2bb00 next 18446744073709551615 of size 552518912 2021-03-06 13:52:01.966534: I tensorflow/core/common_runtime/bfc_allocator.cc:917] Summary of in-use Chunks by size: 2021-03-06 13:52:01.966573: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 50 Chunks of size 256 totalling 12.5KiB 2021-03-06 13:52:01.966612: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 10 Chunks of size 512 totalling 5.0KiB 2021-03-06 13:52:01.966652: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 17 Chunks of size 1024 totalling 17.0KiB 2021-03-06 13:52:01.966694: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1280 totalling 1.2KiB 2021-03-06 13:52:01.966735: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1792 totalling 1.8KiB 2021-03-06 13:52:01.966778: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 10 Chunks of size 2048 totalling 20.0KiB 2021-03-06 13:52:01.966823: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 6 Chunks of size 4096 totalling 24.0KiB 2021-03-06 13:52:01.966869: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 18432 totalling 18.0KiB 2021-03-06 13:52:01.966915: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 104448 totalling 102.0KiB 2021-03-06 13:52:01.966962: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 131072 totalling 128.0KiB 2021-03-06 13:52:01.967011: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 261120 totalling 255.0KiB 2021-03-06 13:52:01.967059: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 294912 totalling 288.0KiB 2021-03-06 13:52:01.967107: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 522240 totalling 510.0KiB 2021-03-06 13:52:01.967156: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 692224 totalling 676.0KiB 2021-03-06 13:52:01.967205: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 816896 totalling 797.8KiB 2021-03-06 13:52:01.967253: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1048576 totalling 1.00MiB 2021-03-06 13:52:01.967300: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1179648 totalling 1.12MiB 2021-03-06 13:52:01.967347: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 2 Chunks of size 1384448 totalling 2.64MiB 2021-03-06 13:52:01.967394: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 3538944 totalling 3.38MiB 2021-03-06 13:52:01.967442: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 3 Chunks of size 4718592 totalling 13.50MiB 2021-03-06 13:52:01.967489: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 4931328 totalling 4.70MiB 2021-03-06 13:52:01.967537: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 18161664 totalling 17.32MiB 2021-03-06 13:52:01.967584: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 18874368 totalling 18.00MiB 2021-03-06 13:52:01.967632: I tensorflow/core/common_runtime/bfc_allocator.cc:924] Sum Total of in-use chunks: 64.45MiB 2021-03-06 13:52:01.967675: I tensorflow/core/common_runtime/bfc_allocator.cc:926] total_region_allocated_bytes_: 639664128 memory_limit_: 639664128 available bytes: 0 curr_region_allocation_bytes_: 1279328256 2021-03-06 13:52:01.967730: I tensorflow/core/common_runtime/bfc_allocator.cc:932] Stats: Limit: 639664128 InUse: 67584000 MaxInUse: 394892800 NumAllocs: 246 MaxAllocSize: 316997632 2021-03-06 13:52:01.967818: W tensorflow/core/common_runtime/bfc_allocator.cc:427] ***__*********______________________________________________________________________________________ 2021-03-06 13:52:12.054533: W tensorflow/core/common_runtime/bfc_allocator.cc:419] Allocator (GPU_0_bfc) ran out of memory trying to allocate 563.19MiB (rounded to 590544896). If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation. Current allocation summary follows. 2021-03-06 13:52:12.055427: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (256): Total Chunks: 50, Chunks in use: 50. 12.5KiB allocated for chunks. 12.5KiB in use in bin. 2.8KiB client-requested in use in bin. 2021-03-06 13:52:12.055972: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (512): Total Chunks: 10, Chunks in use: 10. 5.0KiB allocated for chunks. 5.0KiB in use in bin. 5.0KiB client-requested in use in bin. 2021-03-06 13:52:12.056447: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (1024): Total Chunks: 19, Chunks in use: 19. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 19.7KiB client-requested in use in bin. 2021-03-06 13:52:12.056963: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (2048): Total Chunks: 10, Chunks in use: 10. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin. 2021-03-06 13:52:12.057475: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (4096): Total Chunks: 6, Chunks in use: 6. 24.0KiB allocated for chunks. 24.0KiB in use in bin. 24.0KiB client-requested in use in bin. 2021-03-06 13:52:12.057952: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:12.058680: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (16384): Total Chunks: 1, Chunks in use: 1. 18.0KiB allocated for chunks. 18.0KiB in use in bin. 18.0KiB client-requested in use in bin. 2021-03-06 13:52:12.058902: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:12.059043: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (65536): Total Chunks: 1, Chunks in use: 1. 102.0KiB allocated for chunks. 102.0KiB in use in bin. 72.0KiB client-requested in use in bin. 2021-03-06 13:52:12.059221: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (131072): Total Chunks: 2, Chunks in use: 2. 383.0KiB allocated for chunks. 383.0KiB in use in bin. 383.0KiB client-requested in use in bin. 2021-03-06 13:52:12.059431: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (262144): Total Chunks: 2, Chunks in use: 2. 798.0KiB allocated for chunks. 798.0KiB in use in bin. 798.0KiB client-requested in use in bin. 2021-03-06 13:52:12.059610: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (524288): Total Chunks: 3, Chunks in use: 2. 2.10MiB allocated for chunks. 1.44MiB in use in bin. 1.32MiB client-requested in use in bin. 2021-03-06 13:52:12.059836: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (1048576): Total Chunks: 4, Chunks in use: 4. 4.77MiB allocated for chunks. 4.77MiB in use in bin. 4.77MiB client-requested in use in bin. 2021-03-06 13:52:12.060023: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (2097152): Total Chunks: 2, Chunks in use: 1. 5.66MiB allocated for chunks. 3.38MiB in use in bin. 3.38MiB client-requested in use in bin. 2021-03-06 13:52:12.060248: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (4194304): Total Chunks: 4, Chunks in use: 4. 18.20MiB allocated for chunks. 18.20MiB in use in bin. 16.14MiB client-requested in use in bin. 2021-03-06 13:52:12.060445: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (8388608): Total Chunks: 1, Chunks in use: 0. 15.71MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:12.060674: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (16777216): Total Chunks: 2, Chunks in use: 2. 35.32MiB allocated for chunks. 35.32MiB in use in bin. 35.32MiB client-requested in use in bin. 2021-03-06 13:52:12.060849: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:12.061048: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:12.061254: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:12.061438: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (268435456): Total Chunks: 1, Chunks in use: 0. 526.92MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:12.061665: I tensorflow/core/common_runtime/bfc_allocator.cc:888] Bin for 563.19MiB was 256.00MiB, Chunk State: 2021-03-06 13:52:12.061901: I tensorflow/core/common_runtime/bfc_allocator.cc:894] Size: 526.92MiB | Requested Size: 28.50MiB | in_use: 0 | bin_num: 20, prev: Size: 17.32MiB | Requested Size: 17.32MiB | in_use: 1 | bin_num: -1 2021-03-06 13:52:12.062213: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Next region of size 639664128 2021-03-06 13:52:12.062412: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10000 next 1 of size 1280 2021-03-06 13:52:12.062492: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10500 next 2 of size 256 2021-03-06 13:52:12.062554: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10600 next 3 of size 256 2021-03-06 13:52:12.062631: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10700 next 4 of size 256 2021-03-06 13:52:12.062697: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10800 next 5 of size 256 2021-03-06 13:52:12.062762: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10900 next 42 of size 256 2021-03-06 13:52:12.062852: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10a00 next 51 of size 256 2021-03-06 13:52:12.062953: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10b00 next 52 of size 256 2021-03-06 13:52:12.063039: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10c00 next 46 of size 256 2021-03-06 13:52:12.063125: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10d00 next 44 of size 256 2021-03-06 13:52:12.063235: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10e00 next 110 of size 256 2021-03-06 13:52:12.063330: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10f00 next 6 of size 256 2021-03-06 13:52:12.063422: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11000 next 7 of size 256 2021-03-06 13:52:12.063545: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11100 next 8 of size 1024 2021-03-06 13:52:12.063666: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11500 next 9 of size 256 2021-03-06 13:52:12.063781: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11600 next 10 of size 1024 2021-03-06 13:52:12.063903: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11a00 next 11 of size 1024 2021-03-06 13:52:12.064019: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11e00 next 12 of size 512 2021-03-06 13:52:12.064137: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12000 next 13 of size 256 2021-03-06 13:52:12.064232: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12100 next 14 of size 256 2021-03-06 13:52:12.064326: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12200 next 15 of size 2048 2021-03-06 13:52:12.064441: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12a00 next 16 of size 256 2021-03-06 13:52:12.064536: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12b00 next 17 of size 256 2021-03-06 13:52:12.064630: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12c00 next 18 of size 4096 2021-03-06 13:52:12.064749: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b13c00 next 19 of size 2048 2021-03-06 13:52:12.064866: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b14400 next 20 of size 4096 2021-03-06 13:52:12.064962: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15400 next 21 of size 256 2021-03-06 13:52:12.065054: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15500 next 22 of size 256 2021-03-06 13:52:12.065167: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15600 next 23 of size 256 2021-03-06 13:52:12.065287: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15700 next 24 of size 256 2021-03-06 13:52:12.065385: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15800 next 25 of size 512 2021-03-06 13:52:12.065479: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15a00 next 26 of size 256 2021-03-06 13:52:12.065590: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15b00 next 27 of size 256 2021-03-06 13:52:12.065708: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15c00 next 28 of size 256 2021-03-06 13:52:12.065802: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15d00 next 29 of size 256 2021-03-06 13:52:12.065894: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15e00 next 30 of size 256 2021-03-06 13:52:12.066007: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15f00 next 31 of size 256 2021-03-06 13:52:12.066210: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16000 next 32 of size 256 2021-03-06 13:52:12.066283: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16100 next 33 of size 256 2021-03-06 13:52:12.066323: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16200 next 34 of size 256 2021-03-06 13:52:12.066371: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16300 next 35 of size 256 2021-03-06 13:52:12.066412: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16400 next 36 of size 256 2021-03-06 13:52:12.066447: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16500 next 37 of size 256 2021-03-06 13:52:12.066481: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16600 next 38 of size 256 2021-03-06 13:52:12.066517: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16700 next 39 of size 256 2021-03-06 13:52:12.066562: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16800 next 40 of size 256 2021-03-06 13:52:12.066600: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16900 next 41 of size 256 2021-03-06 13:52:12.066638: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16a00 next 113 of size 256 2021-03-06 13:52:12.066679: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16b00 next 43 of size 816896 2021-03-06 13:52:12.066743: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00bde200 next 78 of size 1024 2021-03-06 13:52:12.066805: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00bde600 next 79 of size 261120 2021-03-06 13:52:12.066857: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c1e200 next 80 of size 256 2021-03-06 13:52:12.066909: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c1e300 next 95 of size 294912 2021-03-06 13:52:12.066981: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c66300 next 97 of size 1179648 2021-03-06 13:52:12.067051: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00d86300 next 112 of size 1384448 2021-03-06 13:52:12.067110: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00ed8300 next 114 of size 692224 2021-03-06 13:52:12.067164: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf00f81300 next 111 of size 692224 2021-03-06 13:52:12.067233: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0102a300 next 45 of size 4931328 2021-03-06 13:52:12.067301: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de200 next 50 of size 256 2021-03-06 13:52:12.067369: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de300 next 103 of size 256 2021-03-06 13:52:12.067426: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de400 next 104 of size 4096 2021-03-06 13:52:12.067481: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014df400 next 105 of size 4096 2021-03-06 13:52:12.067548: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e0400 next 106 of size 256 2021-03-06 13:52:12.067620: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e0500 next 107 of size 4096 2021-03-06 13:52:12.067678: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e1500 next 108 of size 2048 2021-03-06 13:52:12.067746: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e1d00 next 109 of size 1024 2021-03-06 13:52:12.067817: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e2100 next 116 of size 1384448 2021-03-06 13:52:12.067874: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf01634100 next 47 of size 2399488 2021-03-06 13:52:12.067946: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187de00 next 82 of size 2048 2021-03-06 13:52:12.068016: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187e600 next 83 of size 512 2021-03-06 13:52:12.068073: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187e800 next 84 of size 1024 2021-03-06 13:52:12.068143: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ec00 next 85 of size 256 2021-03-06 13:52:12.068213: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ed00 next 86 of size 256 2021-03-06 13:52:12.068270: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ee00 next 87 of size 1024 2021-03-06 13:52:12.068342: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187f200 next 88 of size 1024 2021-03-06 13:52:12.068413: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187f600 next 89 of size 1792 2021-03-06 13:52:12.068473: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187fd00 next 90 of size 18432 2021-03-06 13:52:12.068543: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01884500 next 91 of size 256 2021-03-06 13:52:12.068614: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01884600 next 48 of size 104448 2021-03-06 13:52:12.068682: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189de00 next 49 of size 1024 2021-03-06 13:52:12.068751: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e200 next 92 of size 256 2021-03-06 13:52:12.068819: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e300 next 93 of size 256 2021-03-06 13:52:12.068886: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e400 next 94 of size 256 2021-03-06 13:52:12.068954: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e500 next 96 of size 1024 2021-03-06 13:52:12.069030: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e900 next 98 of size 512 2021-03-06 13:52:12.069087: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189eb00 next 99 of size 256 2021-03-06 13:52:12.069184: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189ec00 next 100 of size 2048 2021-03-06 13:52:12.069255: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f400 next 101 of size 512 2021-03-06 13:52:12.069312: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f600 next 102 of size 512 2021-03-06 13:52:12.069381: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f800 next 118 of size 4718592 2021-03-06 13:52:12.069452: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf01d1f800 next 53 of size 16469504 2021-03-06 13:52:12.069508: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd4600 next 54 of size 4096 2021-03-06 13:52:12.069580: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd5600 next 55 of size 1024 2021-03-06 13:52:12.069651: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd5a00 next 56 of size 2048 2021-03-06 13:52:12.069707: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6200 next 57 of size 2048 2021-03-06 13:52:12.069777: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6a00 next 58 of size 1024 2021-03-06 13:52:12.069847: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6e00 next 59 of size 2048 2021-03-06 13:52:12.069904: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd7600 next 60 of size 512 2021-03-06 13:52:12.069972: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd7800 next 61 of size 3538944 2021-03-06 13:52:12.070094: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037800 next 62 of size 512 2021-03-06 13:52:12.070145: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037a00 next 63 of size 1024 2021-03-06 13:52:12.070178: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037e00 next 64 of size 512 2021-03-06 13:52:12.070204: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038000 next 65 of size 512 2021-03-06 13:52:12.070229: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038200 next 66 of size 256 2021-03-06 13:52:12.070256: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038300 next 67 of size 1024 2021-03-06 13:52:12.070284: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038700 next 68 of size 1024 2021-03-06 13:52:12.070313: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038b00 next 69 of size 522240 2021-03-06 13:52:12.070342: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf030b8300 next 70 of size 1048576 2021-03-06 13:52:12.070370: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf031b8300 next 71 of size 2048 2021-03-06 13:52:12.070399: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf031b8b00 next 72 of size 4718592 2021-03-06 13:52:12.070427: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03638b00 next 73 of size 4718592 2021-03-06 13:52:12.070456: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab8b00 next 74 of size 1024 2021-03-06 13:52:12.070487: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab8f00 next 75 of size 2048 2021-03-06 13:52:12.070519: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab9700 next 76 of size 1024 2021-03-06 13:52:12.070553: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab9b00 next 77 of size 131072 2021-03-06 13:52:12.070590: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ad9b00 next 81 of size 18874368 2021-03-06 13:52:12.070628: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf04cd9b00 next 119 of size 18161664 2021-03-06 13:52:12.070667: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf05e2bb00 next 18446744073709551615 of size 552518912 2021-03-06 13:52:12.070704: I tensorflow/core/common_runtime/bfc_allocator.cc:917] Summary of in-use Chunks by size: 2021-03-06 13:52:12.070750: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 50 Chunks of size 256 totalling 12.5KiB 2021-03-06 13:52:12.070796: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 10 Chunks of size 512 totalling 5.0KiB 2021-03-06 13:52:12.070841: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 17 Chunks of size 1024 totalling 17.0KiB 2021-03-06 13:52:12.070886: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1280 totalling 1.2KiB 2021-03-06 13:52:12.070931: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1792 totalling 1.8KiB 2021-03-06 13:52:12.070977: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 10 Chunks of size 2048 totalling 20.0KiB 2021-03-06 13:52:12.071025: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 6 Chunks of size 4096 totalling 24.0KiB 2021-03-06 13:52:12.071073: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 18432 totalling 18.0KiB 2021-03-06 13:52:12.071121: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 104448 totalling 102.0KiB 2021-03-06 13:52:12.071170: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 131072 totalling 128.0KiB 2021-03-06 13:52:12.071219: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 261120 totalling 255.0KiB 2021-03-06 13:52:12.071267: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 294912 totalling 288.0KiB 2021-03-06 13:52:12.071315: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 522240 totalling 510.0KiB 2021-03-06 13:52:12.071363: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 692224 totalling 676.0KiB 2021-03-06 13:52:12.071413: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 816896 totalling 797.8KiB 2021-03-06 13:52:12.071461: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1048576 totalling 1.00MiB 2021-03-06 13:52:12.071508: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1179648 totalling 1.12MiB 2021-03-06 13:52:12.071555: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 2 Chunks of size 1384448 totalling 2.64MiB 2021-03-06 13:52:12.071601: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 3538944 totalling 3.38MiB 2021-03-06 13:52:12.071649: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 3 Chunks of size 4718592 totalling 13.50MiB 2021-03-06 13:52:12.071696: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 4931328 totalling 4.70MiB 2021-03-06 13:52:12.071744: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 18161664 totalling 17.32MiB 2021-03-06 13:52:12.071792: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 18874368 totalling 18.00MiB 2021-03-06 13:52:12.071840: I tensorflow/core/common_runtime/bfc_allocator.cc:924] Sum Total of in-use chunks: 64.45MiB 2021-03-06 13:52:12.071883: I tensorflow/core/common_runtime/bfc_allocator.cc:926] total_region_allocated_bytes_: 639664128 memory_limit_: 639664128 available bytes: 0 curr_region_allocation_bytes_: 1279328256 2021-03-06 13:52:12.071936: I tensorflow/core/common_runtime/bfc_allocator.cc:932] Stats: Limit: 639664128 InUse: 67584000 MaxInUse: 394892800 NumAllocs: 247 MaxAllocSize: 316997632 2021-03-06 13:52:12.072014: W tensorflow/core/common_runtime/bfc_allocator.cc:427] ***__*********______________________________________________________________________________________ 2021-03-06 13:52:23.151652: W tensorflow/core/common_runtime/bfc_allocator.cc:419] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.15GiB (rounded to 2305163264). If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation. Current allocation summary follows. 2021-03-06 13:52:23.152548: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (256): Total Chunks: 50, Chunks in use: 50. 12.5KiB allocated for chunks. 12.5KiB in use in bin. 2.8KiB client-requested in use in bin. 2021-03-06 13:52:23.153155: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (512): Total Chunks: 10, Chunks in use: 10. 5.0KiB allocated for chunks. 5.0KiB in use in bin. 5.0KiB client-requested in use in bin. 2021-03-06 13:52:23.153658: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (1024): Total Chunks: 19, Chunks in use: 19. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 19.7KiB client-requested in use in bin. 2021-03-06 13:52:23.154524: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (2048): Total Chunks: 10, Chunks in use: 10. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin. 2021-03-06 13:52:23.155051: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (4096): Total Chunks: 6, Chunks in use: 6. 24.0KiB allocated for chunks. 24.0KiB in use in bin. 24.0KiB client-requested in use in bin. 2021-03-06 13:52:23.155420: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:23.155799: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (16384): Total Chunks: 1, Chunks in use: 1. 18.0KiB allocated for chunks. 18.0KiB in use in bin. 18.0KiB client-requested in use in bin. 2021-03-06 13:52:23.156445: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:23.156907: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (65536): Total Chunks: 1, Chunks in use: 1. 102.0KiB allocated for chunks. 102.0KiB in use in bin. 72.0KiB client-requested in use in bin. 2021-03-06 13:52:23.157377: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (131072): Total Chunks: 2, Chunks in use: 2. 383.0KiB allocated for chunks. 383.0KiB in use in bin. 383.0KiB client-requested in use in bin. 2021-03-06 13:52:23.157860: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (262144): Total Chunks: 2, Chunks in use: 2. 798.0KiB allocated for chunks. 798.0KiB in use in bin. 798.0KiB client-requested in use in bin. 2021-03-06 13:52:23.158630: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (524288): Total Chunks: 1, Chunks in use: 0. 797.8KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:23.158860: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (1048576): Total Chunks: 3, Chunks in use: 3. 3.45MiB allocated for chunks. 3.45MiB in use in bin. 3.45MiB client-requested in use in bin. 2021-03-06 13:52:23.159020: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (2097152): Total Chunks: 4, Chunks in use: 4. 10.94MiB allocated for chunks. 10.94MiB in use in bin. 9.98MiB client-requested in use in bin. 2021-03-06 13:52:23.159214: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (4194304): Total Chunks: 3, Chunks in use: 3. 13.70MiB allocated for chunks. 13.70MiB in use in bin. 11.64MiB client-requested in use in bin. 2021-03-06 13:52:23.159371: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:23.159590: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (16777216): Total Chunks: 3, Chunks in use: 3. 56.85MiB allocated for chunks. 56.85MiB in use in bin. 54.64MiB client-requested in use in bin. 2021-03-06 13:52:23.159800: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:23.160006: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:23.160204: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:23.160419: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (268435456): Total Chunks: 1, Chunks in use: 0. 522.96MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:23.160643: I tensorflow/core/common_runtime/bfc_allocator.cc:888] Bin for 2.15GiB was 256.00MiB, Chunk State: 2021-03-06 13:52:23.160925: I tensorflow/core/common_runtime/bfc_allocator.cc:894] Size: 522.96MiB | Requested Size: 66.00MiB | in_use: 0 | bin_num: 20, prev: Size: 18.64MiB | Requested Size: 18.64MiB | in_use: 1 | bin_num: -1 2021-03-06 13:52:23.161138: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Next region of size 639664128 2021-03-06 13:52:23.161370: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10000 next 1 of size 1280 2021-03-06 13:52:23.161537: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10500 next 2 of size 256 2021-03-06 13:52:23.161918: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10600 next 3 of size 256 2021-03-06 13:52:23.162330: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10700 next 4 of size 256 2021-03-06 13:52:23.162568: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10800 next 5 of size 256 2021-03-06 13:52:23.162763: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10900 next 42 of size 256 2021-03-06 13:52:23.162967: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10a00 next 51 of size 256 2021-03-06 13:52:23.163153: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10b00 next 52 of size 256 2021-03-06 13:52:23.163357: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10c00 next 46 of size 256 2021-03-06 13:52:23.163585: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10d00 next 44 of size 256 2021-03-06 13:52:23.163769: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10e00 next 110 of size 256 2021-03-06 13:52:23.163886: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10f00 next 6 of size 256 2021-03-06 13:52:23.163983: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11000 next 7 of size 256 2021-03-06 13:52:23.164079: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11100 next 8 of size 1024 2021-03-06 13:52:23.164177: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11500 next 9 of size 256 2021-03-06 13:52:23.164285: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11600 next 10 of size 1024 2021-03-06 13:52:23.164425: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11a00 next 11 of size 1024 2021-03-06 13:52:23.164565: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11e00 next 12 of size 512 2021-03-06 13:52:23.164709: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12000 next 13 of size 256 2021-03-06 13:52:23.164865: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12100 next 14 of size 256 2021-03-06 13:52:23.165009: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12200 next 15 of size 2048 2021-03-06 13:52:23.165161: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12a00 next 16 of size 256 2021-03-06 13:52:23.165318: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12b00 next 17 of size 256 2021-03-06 13:52:23.165462: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12c00 next 18 of size 4096 2021-03-06 13:52:23.165624: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b13c00 next 19 of size 2048 2021-03-06 13:52:23.165762: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b14400 next 20 of size 4096 2021-03-06 13:52:23.165839: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15400 next 21 of size 256 2021-03-06 13:52:23.165886: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15500 next 22 of size 256 2021-03-06 13:52:23.165922: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15600 next 23 of size 256 2021-03-06 13:52:23.165960: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15700 next 24 of size 256 2021-03-06 13:52:23.166004: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15800 next 25 of size 512 2021-03-06 13:52:23.166070: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15a00 next 26 of size 256 2021-03-06 13:52:23.166123: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15b00 next 27 of size 256 2021-03-06 13:52:23.166179: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15c00 next 28 of size 256 2021-03-06 13:52:23.166226: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15d00 next 29 of size 256 2021-03-06 13:52:23.166275: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15e00 next 30 of size 256 2021-03-06 13:52:23.166335: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15f00 next 31 of size 256 2021-03-06 13:52:23.166388: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16000 next 32 of size 256 2021-03-06 13:52:23.166488: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16100 next 33 of size 256 2021-03-06 13:52:23.166594: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16200 next 34 of size 256 2021-03-06 13:52:23.166703: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16300 next 35 of size 256 2021-03-06 13:52:23.166810: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16400 next 36 of size 256 2021-03-06 13:52:23.166917: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16500 next 37 of size 256 2021-03-06 13:52:23.167041: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16600 next 38 of size 256 2021-03-06 13:52:23.167126: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16700 next 39 of size 256 2021-03-06 13:52:23.167188: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16800 next 40 of size 256 2021-03-06 13:52:23.167263: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16900 next 41 of size 256 2021-03-06 13:52:23.167323: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16a00 next 113 of size 256 2021-03-06 13:52:23.167387: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf00b16b00 next 43 of size 816896 2021-03-06 13:52:23.167459: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00bde200 next 78 of size 1024 2021-03-06 13:52:23.167546: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00bde600 next 79 of size 261120 2021-03-06 13:52:23.167603: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c1e200 next 80 of size 256 2021-03-06 13:52:23.167656: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c1e300 next 95 of size 294912 2021-03-06 13:52:23.167711: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c66300 next 97 of size 1179648 2021-03-06 13:52:23.167769: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00d86300 next 111 of size 2768896 2021-03-06 13:52:23.167872: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0102a300 next 45 of size 4931328 2021-03-06 13:52:23.167943: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de200 next 50 of size 256 2021-03-06 13:52:23.167999: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de300 next 103 of size 256 2021-03-06 13:52:23.168057: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de400 next 104 of size 4096 2021-03-06 13:52:23.168107: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014df400 next 105 of size 4096 2021-03-06 13:52:23.168156: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e0400 next 106 of size 256 2021-03-06 13:52:23.168213: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e0500 next 107 of size 4096 2021-03-06 13:52:23.168281: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e1500 next 108 of size 2048 2021-03-06 13:52:23.168334: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e1d00 next 109 of size 1024 2021-03-06 13:52:23.168378: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e2100 next 116 of size 1384448 2021-03-06 13:52:23.168415: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01634100 next 47 of size 2399488 2021-03-06 13:52:23.168482: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187de00 next 82 of size 2048 2021-03-06 13:52:23.168554: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187e600 next 83 of size 512 2021-03-06 13:52:23.168627: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187e800 next 84 of size 1024 2021-03-06 13:52:23.168725: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ec00 next 85 of size 256 2021-03-06 13:52:23.168778: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ed00 next 86 of size 256 2021-03-06 13:52:23.168830: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ee00 next 87 of size 1024 2021-03-06 13:52:23.168872: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187f200 next 88 of size 1024 2021-03-06 13:52:23.168959: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187f600 next 89 of size 1792 2021-03-06 13:52:23.169014: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187fd00 next 90 of size 18432 2021-03-06 13:52:23.169062: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01884500 next 91 of size 256 2021-03-06 13:52:23.169107: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01884600 next 48 of size 104448 2021-03-06 13:52:23.169155: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189de00 next 49 of size 1024 2021-03-06 13:52:23.169211: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e200 next 92 of size 256 2021-03-06 13:52:23.169264: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e300 next 93 of size 256 2021-03-06 13:52:23.169313: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e400 next 94 of size 256 2021-03-06 13:52:23.169366: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e500 next 96 of size 1024 2021-03-06 13:52:23.169423: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e900 next 98 of size 512 2021-03-06 13:52:23.169484: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189eb00 next 99 of size 256 2021-03-06 13:52:23.169550: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189ec00 next 100 of size 2048 2021-03-06 13:52:23.169615: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f400 next 101 of size 512 2021-03-06 13:52:23.169685: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f600 next 102 of size 512 2021-03-06 13:52:23.169756: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f800 next 53 of size 21188096 2021-03-06 13:52:23.169829: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd4600 next 54 of size 4096 2021-03-06 13:52:23.169912: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd5600 next 55 of size 1024 2021-03-06 13:52:23.169969: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd5a00 next 56 of size 2048 2021-03-06 13:52:23.170029: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6200 next 57 of size 2048 2021-03-06 13:52:23.170073: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6a00 next 58 of size 1024 2021-03-06 13:52:23.170115: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6e00 next 59 of size 2048 2021-03-06 13:52:23.170158: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd7600 next 60 of size 512 2021-03-06 13:52:23.170203: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd7800 next 61 of size 3538944 2021-03-06 13:52:23.170248: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037800 next 62 of size 512 2021-03-06 13:52:23.170292: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037a00 next 63 of size 1024 2021-03-06 13:52:23.170340: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037e00 next 64 of size 512 2021-03-06 13:52:23.170398: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038000 next 65 of size 512 2021-03-06 13:52:23.170457: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038200 next 66 of size 256 2021-03-06 13:52:23.170515: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038300 next 67 of size 1024 2021-03-06 13:52:23.170575: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038700 next 68 of size 1024 2021-03-06 13:52:23.170639: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038b00 next 69 of size 522240 2021-03-06 13:52:23.170699: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf030b8300 next 70 of size 1048576 2021-03-06 13:52:23.170758: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf031b8300 next 71 of size 2048 2021-03-06 13:52:23.170826: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf031b8b00 next 72 of size 4718592 2021-03-06 13:52:23.170893: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03638b00 next 73 of size 4718592 2021-03-06 13:52:23.170959: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab8b00 next 74 of size 1024 2021-03-06 13:52:23.171029: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab8f00 next 75 of size 2048 2021-03-06 13:52:23.171087: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab9700 next 76 of size 1024 2021-03-06 13:52:23.171128: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab9b00 next 77 of size 131072 2021-03-06 13:52:23.171169: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ad9b00 next 81 of size 18874368 2021-03-06 13:52:23.171206: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf04cd9b00 next 112 of size 2768896 2021-03-06 13:52:23.171243: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf04f7db00 next 115 of size 19546112 2021-03-06 13:52:23.171279: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf06221b00 next 18446744073709551615 of size 548365568 2021-03-06 13:52:23.171313: I tensorflow/core/common_runtime/bfc_allocator.cc:917] Summary of in-use Chunks by size: 2021-03-06 13:52:23.171353: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 50 Chunks of size 256 totalling 12.5KiB 2021-03-06 13:52:23.171396: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 10 Chunks of size 512 totalling 5.0KiB 2021-03-06 13:52:23.171440: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 17 Chunks of size 1024 totalling 17.0KiB 2021-03-06 13:52:23.171484: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1280 totalling 1.2KiB 2021-03-06 13:52:23.171529: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1792 totalling 1.8KiB 2021-03-06 13:52:23.171578: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 10 Chunks of size 2048 totalling 20.0KiB 2021-03-06 13:52:23.171633: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 6 Chunks of size 4096 totalling 24.0KiB 2021-03-06 13:52:23.171689: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 18432 totalling 18.0KiB 2021-03-06 13:52:23.171749: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 104448 totalling 102.0KiB 2021-03-06 13:52:23.171804: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 131072 totalling 128.0KiB 2021-03-06 13:52:23.171863: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 261120 totalling 255.0KiB 2021-03-06 13:52:23.171927: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 294912 totalling 288.0KiB 2021-03-06 13:52:23.171986: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 522240 totalling 510.0KiB 2021-03-06 13:52:23.172048: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1048576 totalling 1.00MiB 2021-03-06 13:52:23.172120: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1179648 totalling 1.12MiB 2021-03-06 13:52:23.172164: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1384448 totalling 1.32MiB 2021-03-06 13:52:23.172202: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 2399488 totalling 2.29MiB 2021-03-06 13:52:23.172240: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 2 Chunks of size 2768896 totalling 5.28MiB 2021-03-06 13:52:23.172277: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 3538944 totalling 3.38MiB 2021-03-06 13:52:23.172312: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 2 Chunks of size 4718592 totalling 9.00MiB 2021-03-06 13:52:23.172348: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 4931328 totalling 4.70MiB 2021-03-06 13:52:23.172386: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 18874368 totalling 18.00MiB 2021-03-06 13:52:23.172424: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 19546112 totalling 18.64MiB 2021-03-06 13:52:23.172462: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 21188096 totalling 20.21MiB 2021-03-06 13:52:23.172501: I tensorflow/core/common_runtime/bfc_allocator.cc:924] Sum Total of in-use chunks: 86.29MiB 2021-03-06 13:52:23.172539: I tensorflow/core/common_runtime/bfc_allocator.cc:926] total_region_allocated_bytes_: 639664128 memory_limit_: 639664128 available bytes: 0 curr_region_allocation_bytes_: 1279328256 2021-03-06 13:52:23.172588: I tensorflow/core/common_runtime/bfc_allocator.cc:932] Stats: Limit: 639664128 InUse: 90481664 MaxInUse: 394892800 NumAllocs: 271 MaxAllocSize: 316997632 2021-03-06 13:52:23.172723: W tensorflow/core/common_runtime/bfc_allocator.cc:427] ***************_____________________________________________________________________________________ 2021-03-06 13:52:33.962368: W tensorflow/core/common_runtime/bfc_allocator.cc:419] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.12GiB (rounded to 3355181056). If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation. Current allocation summary follows. 2021-03-06 13:52:33.963270: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (256): Total Chunks: 50, Chunks in use: 50. 12.5KiB allocated for chunks. 12.5KiB in use in bin. 2.8KiB client-requested in use in bin. 2021-03-06 13:52:33.963884: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (512): Total Chunks: 10, Chunks in use: 10. 5.0KiB allocated for chunks. 5.0KiB in use in bin. 5.0KiB client-requested in use in bin. 2021-03-06 13:52:33.964386: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (1024): Total Chunks: 19, Chunks in use: 19. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 19.7KiB client-requested in use in bin. 2021-03-06 13:52:33.964875: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (2048): Total Chunks: 10, Chunks in use: 10. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin. 2021-03-06 13:52:33.965344: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (4096): Total Chunks: 6, Chunks in use: 6. 24.0KiB allocated for chunks. 24.0KiB in use in bin. 24.0KiB client-requested in use in bin. 2021-03-06 13:52:33.965794: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:33.966598: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (16384): Total Chunks: 1, Chunks in use: 1. 18.0KiB allocated for chunks. 18.0KiB in use in bin. 18.0KiB client-requested in use in bin. 2021-03-06 13:52:33.966912: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:33.967135: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (65536): Total Chunks: 1, Chunks in use: 1. 102.0KiB allocated for chunks. 102.0KiB in use in bin. 72.0KiB client-requested in use in bin. 2021-03-06 13:52:33.967390: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (131072): Total Chunks: 2, Chunks in use: 2. 383.0KiB allocated for chunks. 383.0KiB in use in bin. 383.0KiB client-requested in use in bin. 2021-03-06 13:52:33.967617: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (262144): Total Chunks: 2, Chunks in use: 2. 798.0KiB allocated for chunks. 798.0KiB in use in bin. 798.0KiB client-requested in use in bin. 2021-03-06 13:52:33.967848: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (524288): Total Chunks: 2, Chunks in use: 1. 1.44MiB allocated for chunks. 673.5KiB in use in bin. 673.4KiB client-requested in use in bin. 2021-03-06 13:52:33.968041: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (1048576): Total Chunks: 2, Chunks in use: 2. 2.12MiB allocated for chunks. 2.12MiB in use in bin. 2.12MiB client-requested in use in bin. 2021-03-06 13:52:33.968265: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (2097152): Total Chunks: 5, Chunks in use: 5. 16.30MiB allocated for chunks. 16.30MiB in use in bin. 15.99MiB client-requested in use in bin. 2021-03-06 13:52:33.968461: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (4194304): Total Chunks: 3, Chunks in use: 3. 16.34MiB allocated for chunks. 16.34MiB in use in bin. 12.96MiB client-requested in use in bin. 2021-03-06 13:52:33.968678: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (8388608): Total Chunks: 1, Chunks in use: 0. 10.23MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:33.968876: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (16777216): Total Chunks: 2, Chunks in use: 2. 36.64MiB allocated for chunks. 36.64MiB in use in bin. 36.64MiB client-requested in use in bin. 2021-03-06 13:52:33.969093: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:33.969273: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:33.969483: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:33.969699: I tensorflow/core/common_runtime/bfc_allocator.cc:872] Bin (268435456): Total Chunks: 1, Chunks in use: 0. 525.60MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2021-03-06 13:52:33.969890: I tensorflow/core/common_runtime/bfc_allocator.cc:888] Bin for 3.12GiB was 256.00MiB, Chunk State: 2021-03-06 13:52:33.970288: I tensorflow/core/common_runtime/bfc_allocator.cc:894] Size: 525.60MiB | Requested Size: 51.65MiB | in_use: 0 | bin_num: 20, prev: Size: 18.64MiB | Requested Size: 18.64MiB | in_use: 1 | bin_num: -1 2021-03-06 13:52:33.970494: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Next region of size 639664128 2021-03-06 13:52:33.970618: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10000 next 1 of size 1280 2021-03-06 13:52:33.970694: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10500 next 2 of size 256 2021-03-06 13:52:33.970758: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10600 next 3 of size 256 2021-03-06 13:52:33.970832: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10700 next 4 of size 256 2021-03-06 13:52:33.970914: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10800 next 5 of size 256 2021-03-06 13:52:33.970986: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10900 next 42 of size 256 2021-03-06 13:52:33.971061: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10a00 next 51 of size 256 2021-03-06 13:52:33.971177: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10b00 next 52 of size 256 2021-03-06 13:52:33.971287: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10c00 next 46 of size 256 2021-03-06 13:52:33.971377: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10d00 next 44 of size 256 2021-03-06 13:52:33.971468: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10e00 next 110 of size 256 2021-03-06 13:52:33.971581: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b10f00 next 6 of size 256 2021-03-06 13:52:33.971675: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11000 next 7 of size 256 2021-03-06 13:52:33.971771: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11100 next 8 of size 1024 2021-03-06 13:52:33.971886: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11500 next 9 of size 256 2021-03-06 13:52:33.971983: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11600 next 10 of size 1024 2021-03-06 13:52:33.972075: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11a00 next 11 of size 1024 2021-03-06 13:52:33.972195: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b11e00 next 12 of size 512 2021-03-06 13:52:33.972315: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12000 next 13 of size 256 2021-03-06 13:52:33.972409: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12100 next 14 of size 256 2021-03-06 13:52:33.972502: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12200 next 15 of size 2048 2021-03-06 13:52:33.972626: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12a00 next 16 of size 256 2021-03-06 13:52:33.972720: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12b00 next 17 of size 256 2021-03-06 13:52:33.972815: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b12c00 next 18 of size 4096 2021-03-06 13:52:33.972930: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b13c00 next 19 of size 2048 2021-03-06 13:52:33.973047: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b14400 next 20 of size 4096 2021-03-06 13:52:33.973142: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15400 next 21 of size 256 2021-03-06 13:52:33.973235: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15500 next 22 of size 256 2021-03-06 13:52:33.973347: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15600 next 23 of size 256 2021-03-06 13:52:33.973459: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15700 next 24 of size 256 2021-03-06 13:52:33.973575: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15800 next 25 of size 512 2021-03-06 13:52:33.973690: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15a00 next 26 of size 256 2021-03-06 13:52:33.973805: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15b00 next 27 of size 256 2021-03-06 13:52:33.973922: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15c00 next 28 of size 256 2021-03-06 13:52:33.974118: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15d00 next 29 of size 256 2021-03-06 13:52:33.974242: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15e00 next 30 of size 256 2021-03-06 13:52:33.974297: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b15f00 next 31 of size 256 2021-03-06 13:52:33.974346: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16000 next 32 of size 256 2021-03-06 13:52:33.974392: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16100 next 33 of size 256 2021-03-06 13:52:33.974436: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16200 next 34 of size 256 2021-03-06 13:52:33.974480: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16300 next 35 of size 256 2021-03-06 13:52:33.974525: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16400 next 36 of size 256 2021-03-06 13:52:33.974569: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16500 next 37 of size 256 2021-03-06 13:52:33.974616: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16600 next 38 of size 256 2021-03-06 13:52:33.974667: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16700 next 39 of size 256 2021-03-06 13:52:33.974725: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16800 next 40 of size 256 2021-03-06 13:52:33.974782: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16900 next 41 of size 256 2021-03-06 13:52:33.974842: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00b16a00 next 113 of size 256 2021-03-06 13:52:33.974906: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf00b16b00 next 43 of size 816896 2021-03-06 13:52:33.974971: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00bde200 next 78 of size 1024 2021-03-06 13:52:33.975038: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00bde600 next 79 of size 261120 2021-03-06 13:52:33.975106: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c1e200 next 80 of size 256 2021-03-06 13:52:33.975175: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c1e300 next 95 of size 294912 2021-03-06 13:52:33.975245: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00c66300 next 97 of size 1179648 2021-03-06 13:52:33.975315: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf00d86300 next 45 of size 7700224 2021-03-06 13:52:33.975384: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de200 next 50 of size 256 2021-03-06 13:52:33.975457: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de300 next 103 of size 256 2021-03-06 13:52:33.975528: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014de400 next 104 of size 4096 2021-03-06 13:52:33.975615: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014df400 next 105 of size 4096 2021-03-06 13:52:33.975684: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e0400 next 106 of size 256 2021-03-06 13:52:33.975752: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e0500 next 107 of size 4096 2021-03-06 13:52:33.975819: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e1500 next 108 of size 2048 2021-03-06 13:52:33.975887: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e1d00 next 109 of size 1024 2021-03-06 13:52:33.975956: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf014e2100 next 120 of size 689664 2021-03-06 13:52:33.976026: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0158a700 next 47 of size 3094272 2021-03-06 13:52:33.976095: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187de00 next 82 of size 2048 2021-03-06 13:52:33.976152: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187e600 next 83 of size 512 2021-03-06 13:52:33.976221: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187e800 next 84 of size 1024 2021-03-06 13:52:33.976280: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ec00 next 85 of size 256 2021-03-06 13:52:33.976365: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ed00 next 86 of size 256 2021-03-06 13:52:33.976436: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187ee00 next 87 of size 1024 2021-03-06 13:52:33.976495: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187f200 next 88 of size 1024 2021-03-06 13:52:33.976554: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187f600 next 89 of size 1792 2021-03-06 13:52:33.976611: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0187fd00 next 90 of size 18432 2021-03-06 13:52:33.976668: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01884500 next 91 of size 256 2021-03-06 13:52:33.976724: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01884600 next 48 of size 104448 2021-03-06 13:52:33.976793: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189de00 next 49 of size 1024 2021-03-06 13:52:33.976850: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e200 next 92 of size 256 2021-03-06 13:52:33.976917: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e300 next 93 of size 256 2021-03-06 13:52:33.976973: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e400 next 94 of size 256 2021-03-06 13:52:33.977027: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e500 next 96 of size 1024 2021-03-06 13:52:33.977094: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189e900 next 98 of size 512 2021-03-06 13:52:33.977163: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189eb00 next 99 of size 256 2021-03-06 13:52:33.977220: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189ec00 next 100 of size 2048 2021-03-06 13:52:33.977275: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f400 next 101 of size 512 2021-03-06 13:52:33.977343: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f600 next 102 of size 512 2021-03-06 13:52:33.977415: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf0189f800 next 118 of size 4153344 2021-03-06 13:52:33.977488: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01c95800 next 111 of size 3538944 2021-03-06 13:52:33.977565: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf01ff5800 next 115 of size 2768896 2021-03-06 13:52:33.977637: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf02299800 next 53 of size 10726912 2021-03-06 13:52:33.977706: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd4600 next 54 of size 4096 2021-03-06 13:52:33.977774: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd5600 next 55 of size 1024 2021-03-06 13:52:33.977840: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd5a00 next 56 of size 2048 2021-03-06 13:52:33.977907: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6200 next 57 of size 2048 2021-03-06 13:52:33.977974: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6a00 next 58 of size 1024 2021-03-06 13:52:33.978093: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd6e00 next 59 of size 2048 2021-03-06 13:52:33.978165: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd7600 next 60 of size 512 2021-03-06 13:52:33.978204: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf02cd7800 next 61 of size 3538944 2021-03-06 13:52:33.978236: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037800 next 62 of size 512 2021-03-06 13:52:33.978266: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037a00 next 63 of size 1024 2021-03-06 13:52:33.978295: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03037e00 next 64 of size 512 2021-03-06 13:52:33.978322: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038000 next 65 of size 512 2021-03-06 13:52:33.978350: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038200 next 66 of size 256 2021-03-06 13:52:33.978377: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038300 next 67 of size 1024 2021-03-06 13:52:33.978404: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038700 next 68 of size 1024 2021-03-06 13:52:33.978432: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03038b00 next 69 of size 522240 2021-03-06 13:52:33.978463: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf030b8300 next 70 of size 1048576 2021-03-06 13:52:33.978492: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf031b8300 next 71 of size 2048 2021-03-06 13:52:33.978522: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf031b8b00 next 72 of size 4718592 2021-03-06 13:52:33.978553: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03638b00 next 73 of size 4718592 2021-03-06 13:52:33.978585: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab8b00 next 74 of size 1024 2021-03-06 13:52:33.978618: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab8f00 next 75 of size 2048 2021-03-06 13:52:33.978652: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab9700 next 76 of size 1024 2021-03-06 13:52:33.978689: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ab9b00 next 77 of size 131072 2021-03-06 13:52:33.978727: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf03ad9b00 next 81 of size 18874368 2021-03-06 13:52:33.978766: I tensorflow/core/common_runtime/bfc_allocator.cc:908] InUse at 0xf04cd9b00 next 119 of size 19546112 2021-03-06 13:52:33.978806: I tensorflow/core/common_runtime/bfc_allocator.cc:908] Free at 0xf05f7db00 next 18446744073709551615 of size 551134464 2021-03-06 13:52:33.978844: I tensorflow/core/common_runtime/bfc_allocator.cc:917] Summary of in-use Chunks by size: 2021-03-06 13:52:33.978893: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 50 Chunks of size 256 totalling 12.5KiB 2021-03-06 13:52:33.978938: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 10 Chunks of size 512 totalling 5.0KiB 2021-03-06 13:52:33.978984: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 17 Chunks of size 1024 totalling 17.0KiB 2021-03-06 13:52:33.979031: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1280 totalling 1.2KiB 2021-03-06 13:52:33.979078: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1792 totalling 1.8KiB 2021-03-06 13:52:33.979125: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 10 Chunks of size 2048 totalling 20.0KiB 2021-03-06 13:52:33.979172: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 6 Chunks of size 4096 totalling 24.0KiB 2021-03-06 13:52:33.979221: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 18432 totalling 18.0KiB 2021-03-06 13:52:33.979269: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 104448 totalling 102.0KiB 2021-03-06 13:52:33.979318: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 131072 totalling 128.0KiB 2021-03-06 13:52:33.979366: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 261120 totalling 255.0KiB 2021-03-06 13:52:33.979414: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 294912 totalling 288.0KiB 2021-03-06 13:52:33.979462: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 522240 totalling 510.0KiB 2021-03-06 13:52:33.979510: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 689664 totalling 673.5KiB 2021-03-06 13:52:33.979557: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1048576 totalling 1.00MiB 2021-03-06 13:52:33.979604: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 1179648 totalling 1.12MiB 2021-03-06 13:52:33.979651: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 2768896 totalling 2.64MiB 2021-03-06 13:52:33.979699: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 3094272 totalling 2.95MiB 2021-03-06 13:52:33.979746: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 2 Chunks of size 3538944 totalling 6.75MiB 2021-03-06 13:52:33.979792: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 4153344 totalling 3.96MiB 2021-03-06 13:52:33.979839: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 2 Chunks of size 4718592 totalling 9.00MiB 2021-03-06 13:52:33.979886: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 7700224 totalling 7.34MiB 2021-03-06 13:52:33.979934: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 18874368 totalling 18.00MiB 2021-03-06 13:52:33.979982: I tensorflow/core/common_runtime/bfc_allocator.cc:920] 1 Chunks of size 19546112 totalling 18.64MiB 2021-03-06 13:52:33.980029: I tensorflow/core/common_runtime/bfc_allocator.cc:924] Sum Total of in-use chunks: 73.42MiB 2021-03-06 13:52:33.980072: I tensorflow/core/common_runtime/bfc_allocator.cc:926] total_region_allocated_bytes_: 639664128 memory_limit_: 639664128 available bytes: 0 curr_region_allocation_bytes_: 1279328256 2021-03-06 13:52:33.980124: I tensorflow/core/common_runtime/bfc_allocator.cc:932] Stats: Limit: 639664128 InUse: 76985856 MaxInUse: 394892800 NumAllocs: 341 MaxAllocSize: 316997632 2021-03-06 13:52:33.980202: W tensorflow/core/common_runtime/bfc_allocator.cc:427] ****_*********______________________________________________________________________________________