NVIDIA CUDA v9.0
NVIDIA cuDNN v7.1.4
NVIDIA Video Car GTX650
NVIDA Video Card Driver 24.21.13.9826 dated 6/12/2018
Windows 10 : with latest updates
RAM 16 GB
GPU RAM; 8 GB
Python Conda
Tensorflow 1.9
Setuptools 39.1.0
############Convolution Neural Network model fitting#############
history<- model %>%
-
fit(train, -
trainLabels, -
epochs=60, -
batch_size=32, -
validation_split=0.20, -
validation_data =list(test,testLabels) -
)
##########Compilation Error
2018-08-11 18:06:14.131528: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2018-08-11 18:06:14.399912: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1392] Found device 0 with properties:
name: GeForce GTX 650 major: 3 minor: 0 memoryClockRate(GHz): 1.202
pciBusID: 0000:01:00.0
totalMemory: 2.00GiB freeMemory: 1.65GiB
2018-08-11 18:06:14.400413: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1471] Adding visible gpu devices: 0
2018-08-11 18:06:16.311180: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-08-11 18:06:16.312805: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:958] 0
2018-08-11 18:06:16.313982: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:971] 0: N
2018-08-11 18:06:16.341990: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1468 MB memory) → physical GPU (device: 0, name: GeForce GTX 650, pci bus id: 0000:01:00.0, compute capability: 3.0)
2018-08-11 18:06:28.214222: W C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 109.02MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-08-11 18:06:28.237589: W C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 436.88MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-08-11 18:06:28.387031: W C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 384.53MiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-08-11 18:06:38.406105: W C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 126.62MiB. Current allocation summary follows.
2018-08-11 18:06:38.408184: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (256): Total Chunks: 57, Chunks in use: 56. 14.3KiB allocated for chunks. 14.0KiB in use in bin. 3.1KiB client-requested in use in bin.
2018-08-11 18:06:38.410308: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (512): Total Chunks: 1, Chunks in use: 1. 512B allocated for chunks. 512B in use in bin. 324B client-requested in use in bin.
2018-08-11 18:06:38.412429: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.3KiB allocated for chunks. 1.3KiB in use in bin. 1.0KiB client-requested in use in bin.
2018-08-11 18:06:38.414595: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (2048): Total Chunks: 8, Chunks in use: 8. 22.0KiB allocated for chunks. 22.0KiB in use in bin. 21.5KiB client-requested in use in bin.
2018-08-11 18:06:38.416730: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (4096): Total Chunks: 4, Chunks in use: 4. 24.0KiB allocated for chunks. 24.0KiB in use in bin. 24.0KiB client-requested in use in bin.
2018-08-11 18:06:38.419297: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-08-11 18:06:38.421343: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-08-11 18:06:38.423488: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (32768): Total Chunks: 5, Chunks in use: 5. 180.0KiB allocated for chunks. 180.0KiB in use in bin. 180.0KiB client-requested in use in bin.
2018-08-11 18:06:38.425586: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (65536): Total Chunks: 4, Chunks in use: 4. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2018-08-11 18:06:38.427716: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (131072): Total Chunks: 4, Chunks in use: 4. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2018-08-11 18:06:38.429793: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-08-11 18:06:38.432039: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (524288): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-08-11 18:06:38.434009: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-08-11 18:06:38.436187: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-08-11 18:06:38.438311: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-08-11 18:06:38.440164: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (8388608): Total Chunks: 2, Chunks in use: 1. 24.68MiB allocated for chunks. 12.36MiB in use in bin. 12.36MiB client-requested in use in bin.
2018-08-11 18:06:38.442382: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-08-11 18:06:38.444533: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-08-11 18:06:38.446592: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2018-08-11 18:06:38.448453: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (134217728): Total Chunks: 2, Chunks in use: 2. 338.04MiB allocated for chunks. 338.04MiB in use in bin. 255.83MiB client-requested in use in bin.
2018-08-11 18:06:38.450420: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:630] Bin (268435456): Total Chunks: 4, Chunks in use: 4. 1.08GiB allocated for chunks. 1.08GiB in use in bin. 1.08GiB client-requested in use in bin.
2018-08-11 18:06:38.451732: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:646] Bin for 126.62MiB was 64.00MiB, Chunk State:
2018-08-11 18:06:38.463128: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700B60000 of size 1280
2018-08-11 18:06:38.463791: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700B60500 of size 256
2018-08-11 18:06:38.464404: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700B60600 of size 256
2018-08-11 18:06:38.464955: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700B60700 of size 256
2018-08-11 18:06:38.465427: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700B60800 of size 256
2018-08-11 18:06:38.466035: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700B60900 of size 256
2018-08-11 18:06:38.466677: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700B60A00 of size 256
2018-08-11 18:06:38.467258: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700B60B00 of size 256
2018-08-11 18:06:38.467790: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700B60C00 of size 256
2018-08-11 18:06:38.468353: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700B60D00 of size 256
2018-08-11 18:06:38.468960: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700B60E00 of size 256
2018-08-11 18:06:38.469539: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700B60F00 of size 256
2018-08-11 18:06:38.470145: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700B61000 of size 256
2018-08-11 18:06:38.470666: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700B61100 of size 256
2018-08-11 18:06:38.471201: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700B61200 of size 256
2018-08-11 18:06:38.471788: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700B61300 of size 256
2018-08-11 18:06:38.472313: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700B61400 of size 256
2018-08-11 18:06:38.472844: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700B61500 of size 3584
2018-08-11 18:06:38.473418: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700B62300 of size 6144
2018-08-11 18:06:38.474028: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700B63B00 of size 256
2018-08-11 18:06:38.474478: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700B63C00 of size 36864
2018-08-11 18:06:38.475040: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700B6CC00 of size 256
2018-08-11 18:06:38.475501: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700B6CD00 of size 73728
2018-08-11 18:06:38.476007: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700B7ED00 of size 147456
2018-08-11 18:06:38.476648: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700BA2D00 of size 256
2018-08-11 18:06:38.477154: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000700BA2E00 of size 289538048
2018-08-11 18:06:38.477743: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000711FC2E00 of size 2048
2018-08-11 18:06:38.478235: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000711FC3600 of size 256
2018-08-11 18:06:38.478782: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000711FC3700 of size 256
2018-08-11 18:06:38.479292: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000711FC3800 of size 256
2018-08-11 18:06:38.479781: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000711FC3900 of size 256
2018-08-11 18:06:38.480344: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000711FC3A00 of size 3584
2018-08-11 18:06:38.481033: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000711FC4800 of size 36864
2018-08-11 18:06:38.481313: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000711FCD800 of size 73728
2018-08-11 18:06:38.481557: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000711FDF800 of size 147456
2018-08-11 18:06:38.481712: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000712003800 of size 289538048
2018-08-11 18:06:38.482058: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000723423800 of size 6144
2018-08-11 18:06:38.482379: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000723425000 of size 3584
2018-08-11 18:06:38.482686: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000723425E00 of size 6144
2018-08-11 18:06:38.482968: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000723427600 of size 256
2018-08-11 18:06:38.483264: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000723427700 of size 256
2018-08-11 18:06:38.483531: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000723427800 of size 36864
2018-08-11 18:06:38.483815: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000723430800 of size 256
2018-08-11 18:06:38.484085: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000723430900 of size 256
2018-08-11 18:06:38.484367: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000723430A00 of size 256
2018-08-11 18:06:38.484663: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000723430B00 of size 256
2018-08-11 18:06:38.484943: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000723430C00 of size 73728
2018-08-11 18:06:38.485224: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000723442C00 of size 147456
2018-08-11 18:06:38.485450: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000723466C00 of size 256
2018-08-11 18:06:38.485696: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000723466D00 of size 256
2018-08-11 18:06:38.485840: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000723466E00 of size 256
2018-08-11 18:06:38.486096: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000723466F00 of size 256
2018-08-11 18:06:38.486318: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000723467000 of size 289538048
2018-08-11 18:06:38.486577: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000734887000 of size 2048
2018-08-11 18:06:38.486802: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000734887800 of size 2048
2018-08-11 18:06:38.487040: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000734888000 of size 256
2018-08-11 18:06:38.487296: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000734888100 of size 256
2018-08-11 18:06:38.487542: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000734888200 of size 256
2018-08-11 18:06:38.487770: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000734888300 of size 256
2018-08-11 18:06:38.488017: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000734888400 of size 256
2018-08-11 18:06:38.488320: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000734888500 of size 256
2018-08-11 18:06:38.488580: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000734888600 of size 256
2018-08-11 18:06:38.488809: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000734888700 of size 256
2018-08-11 18:06:38.489072: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000734888800 of size 256
2018-08-11 18:06:38.489331: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000734888900 of size 256
2018-08-11 18:06:38.489582: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000734888A00 of size 256
2018-08-11 18:06:38.489901: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000734888B00 of size 256
2018-08-11 18:06:38.490201: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000734888C00 of size 256
2018-08-11 18:06:38.490488: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000734888D00 of size 256
2018-08-11 18:06:38.490722: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000734888E00 of size 36864
2018-08-11 18:06:38.490979: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 0000000734891E00 of size 12923136
2018-08-11 18:06:38.491219: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007354E4F00 of size 512
2018-08-11 18:06:38.491463: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007354E5100 of size 256
2018-08-11 18:06:38.491670: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007354E5200 of size 3584
2018-08-11 18:06:38.491941: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007354E6000 of size 256
2018-08-11 18:06:38.492094: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007354E6100 of size 147456
2018-08-11 18:06:38.492293: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073550A100 of size 256
2018-08-11 18:06:38.492714: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000073550A200 of size 289538048
2018-08-11 18:06:38.493033: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000074692A200 of size 2048
2018-08-11 18:06:38.493319: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000074692AA00 of size 256
2018-08-11 18:06:38.493596: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000074692AB00 of size 6144
2018-08-11 18:06:38.493886: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000074692C300 of size 256
2018-08-11 18:06:38.494166: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Free at 000000074692C400 of size 256
2018-08-11 18:06:38.494459: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000074692C500 of size 256
2018-08-11 18:06:38.494735: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000074692C600 of size 36864
2018-08-11 18:06:38.494971: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000746935600 of size 256
2018-08-11 18:06:38.495117: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000746935700 of size 73728
2018-08-11 18:06:38.495441: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000746947700 of size 256
2018-08-11 18:06:38.495657: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 0000000746947800 of size 12960000
2018-08-11 18:06:38.495972: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007475A3900 of size 256
2018-08-11 18:06:38.496293: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 00000007475A3A00 of size 135489024
2018-08-11 18:06:38.496597: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:665] Chunk at 000000074F6DA000 of size 218973696
2018-08-11 18:06:38.496859: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:671] Summary of in-use Chunks by size:
2018-08-11 18:06:38.497193: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 56 Chunks of size 256 totalling 14.0KiB
2018-08-11 18:06:38.497497: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 512 totalling 512B
2018-08-11 18:06:38.497798: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 1280 totalling 1.3KiB
2018-08-11 18:06:38.498114: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 4 Chunks of size 2048 totalling 8.0KiB
2018-08-11 18:06:38.498428: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 4 Chunks of size 3584 totalling 14.0KiB
2018-08-11 18:06:38.498746: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 4 Chunks of size 6144 totalling 24.0KiB
2018-08-11 18:06:38.499041: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 5 Chunks of size 36864 totalling 180.0KiB
2018-08-11 18:06:38.499357: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 4 Chunks of size 73728 totalling 288.0KiB
2018-08-11 18:06:38.499658: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 4 Chunks of size 147456 totalling 576.0KiB
2018-08-11 18:06:38.499922: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 12960000 totalling 12.36MiB
2018-08-11 18:06:38.500223: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 135489024 totalling 129.21MiB
2018-08-11 18:06:38.500518: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 1 Chunks of size 218973696 totalling 208.83MiB
2018-08-11 18:06:38.500792: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:674] 4 Chunks of size 289538048 totalling 1.08GiB
2018-08-11 18:06:38.501147: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:678] Sum Total of in-use chunks: 1.42GiB
2018-08-11 18:06:38.501438: I C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:680] Stats:
Limit: 1539630694
InUse: 1526707200
MaxInUse: 1526707200
NumAllocs: 98
MaxAllocSize: 289538048
2018-08-11 18:06:38.501944: W C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:279] ***********************************************************************************************xxxxx
2018-08-11 18:06:38.502361: W C:\users\nwani_bazel_nwani\ujdkfsks\execroot\org_tensorflow\tensorflow\core\framework\op_kernel.cc:1318] OP_REQUIRES failed at conv_ops.cc:693 : Resource exhausted: OOM when allocating tensor with shape[27,32,196,196] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Error in py_call_impl(callable, dots$args, dots$keywords) :
ResourceExhaustedError: OOM when allocating tensor with shape[27,32,196,196] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: conv2d_2/convolution = Conv2D[T=DT_FLOAT, _class=[“loc:@training/SGD/gradients/conv2d_2/convolution_grad/Conv2DBackpropFilter”], data_format=“NCHW”, dilations=[1, 1, 1, 1], padding=“VALID”, strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device=“/job:localhost/replica:0/task:0/device:GPU:0”](conv2d_1/Relu, conv2d_2/kernel/read)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[Node: metrics/acc/Mean/_97 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_790_metrics/acc/Mean", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]