Hello,
I am having trouble with my GPU memory when running the example “Turbulent physics: Zero Equation Turbulance model”, i.e. ldc_2d_zeroEq.py.
When running the example code, everything works perfectly fine for about 5 time steps . During this stage around 2.2 Gb of the GPU memory is used. Then all the sudden, a spike occurs where the simulation start occupying the entire memory (3.7 Gb) and the run fails.
Can’t really see any reason for why the memory usages should increase so drastically and is wondering whether it is a bug or just lack of hardware capacity.
I’ve tried running the example “Lid Driven Cavity Flow”, and it works without any problems.
2021-11-01 17:04:49.237170: W tensorflow/core/common_runtime/bfc_allocator.cc:419] Allocator (GPU_0_bfc) ran out of memory trying to allocate 353.91MiB (rounded to 371101696). Current allocation summary follows.
2021-11-01 17:04:49.237316: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (256): Total Chunks: 128, Chunks in use: 128. 32.0KiB allocated for chunks. 32.0KiB in use in bin. 553B client-requested in use in bin.
2021-11-01 17:04:49.237360: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (512): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-01 17:04:49.237405: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2021-11-01 17:04:49.237451: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (2048): Total Chunks: 54, Chunks in use: 54. 108.0KiB allocated for chunks. 108.0KiB in use in bin. 108.0KiB client-requested in use in bin.
2021-11-01 17:04:49.237497: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (4096): Total Chunks: 14, Chunks in use: 13. 64.0KiB allocated for chunks. 60.0KiB in use in bin. 59.5KiB client-requested in use in bin.
2021-11-01 17:04:49.237526: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (8192): Total Chunks: 53, Chunks in use: 51. 814.8KiB allocated for chunks. 783.2KiB in use in bin. 777.2KiB client-requested in use in bin.
2021-11-01 17:04:49.237567: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (16384): Total Chunks: 9, Chunks in use: 7. 257.0KiB allocated for chunks. 203.5KiB in use in bin. 203.1KiB client-requested in use in bin.
2021-11-01 17:04:49.237610: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (32768): Total Chunks: 1, Chunks in use: 0. 47.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-01 17:04:49.237651: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-01 17:04:49.237692: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-01 17:04:49.237732: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-01 17:04:49.237777: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (524288): Total Chunks: 10, Chunks in use: 5. 6.94MiB allocated for chunks. 3.46MiB in use in bin. 3.46MiB client-requested in use in bin.
2021-11-01 17:04:49.237822: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (1048576): Total Chunks: 30, Chunks in use: 29. 33.05MiB allocated for chunks. 31.67MiB in use in bin. 31.67MiB client-requested in use in bin.
2021-11-01 17:04:49.237864: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-01 17:04:49.237914: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (4194304): Total Chunks: 52, Chunks in use: 51. 402.29MiB allocated for chunks. 396.43MiB in use in bin. 396.43MiB client-requested in use in bin.
2021-11-01 17:04:49.237966: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (8388608): Total Chunks: 3, Chunks in use: 3. 28.71MiB allocated for chunks. 28.71MiB in use in bin. 23.43MiB client-requested in use in bin.
2021-11-01 17:04:49.238010: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-01 17:04:49.238056: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (33554432): Total Chunks: 1, Chunks in use: 0. 38.71MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-01 17:04:49.238100: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-01 17:04:49.238146: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (134217728): Total Chunks: 1, Chunks in use: 0. 158.09MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-01 17:04:49.238196: I tensorflow/core/common_runtime/bfc_allocator.cc:869] Bin (268435456): Total Chunks: 7, Chunks in use: 6. 2.45GiB allocated for chunks. 2.14GiB in use in bin. 2.07GiB client-requested in use in bin.
2021-11-01 17:04:49.238244: I tensorflow/core/common_runtime/bfc_allocator.cc:885] Bin for 353.91MiB was 256.00MiB, Chunk State:
2021-11-01 17:04:49.238305: I tensorflow/core/common_runtime/bfc_allocator.cc:891] Size: 316.18MiB | Requested Size: 7.81MiB | in_use: 0 | bin_num: 20, prev: Size: 353.91MiB | Requested Size: 353.91MiB | in_use: 1 | bin_num: -1
2021-11-01 17:04:49.238345: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 1180827648
2021-11-01 17:04:49.238386: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f15e2000000 next 245 of size 371101696
2021-11-01 17:04:49.238423: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f15f81e9000 next 342 of size 371101696
2021-11-01 17:04:49.238459: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f160e3d2000 next 18446744073709551615 of size 438624256
2021-11-01 17:04:49.238493: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 1073741824
2021-11-01 17:04:49.238528: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f162e000000 next 472 of size 371101696
2021-11-01 17:04:49.238566: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16441e9000 next 298 of size 371101696
2021-11-01 17:04:49.238599: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x7f165a3d2000 next 18446744073709551615 of size 331538432
2021-11-01 17:04:49.238632: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 536870912
2021-11-01 17:04:49.238665: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f166e000000 next 469 of size 371101696
2021-11-01 17:04:49.238695: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x7f16841e9000 next 18446744073709551615 of size 165769216
2021-11-01 17:04:49.238727: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 268435456
2021-11-01 17:04:49.238754: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f168e000000 next 247 of size 8192000
2021-11-01 17:04:49.238773: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f168e7d0000 next 484 of size 8192000
2021-11-01 17:04:49.238793: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f168efa0000 next 454 of size 8192000
2021-11-01 17:04:49.238820: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f168f770000 next 363 of size 8192000
2021-11-01 17:04:49.238854: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f168ff40000 next 476 of size 8189952
2021-11-01 17:04:49.238890: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f169070f800 next 440 of size 8189952
2021-11-01 17:04:49.238925: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f1690edf000 next 441 of size 8189952
2021-11-01 17:04:49.238959: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16916ae800 next 240 of size 8189952
2021-11-01 17:04:49.238991: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f1691e7e000 next 213 of size 8192000
2021-11-01 17:04:49.239025: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f169264e000 next 219 of size 8192000
2021-11-01 17:04:49.239050: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f1692e1e000 next 457 of size 8192000
2021-11-01 17:04:49.239065: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16935ee000 next 401 of size 8192000
2021-11-01 17:04:49.239085: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x7f1693dbe000 next 324 of size 1449728
2021-11-01 17:04:49.239115: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f1693f1ff00 next 301 of size 1048576
2021-11-01 17:04:49.239149: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f169401ff00 next 380 of size 1048576
2021-11-01 17:04:49.239170: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f169411ff00 next 221 of size 1048576
2021-11-01 17:04:49.239186: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f169421ff00 next 253 of size 1048576
2021-11-01 17:04:49.239206: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f169431ff00 next 433 of size 1048576
2021-11-01 17:04:49.239233: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x7f169441ff00 next 426 of size 6144000
2021-11-01 17:04:49.239265: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16949fbf00 next 473 of size 8189952
2021-11-01 17:04:49.239299: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16951cb700 next 366 of size 8192000
2021-11-01 17:04:49.239331: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f169599b700 next 229 of size 8192000
2021-11-01 17:04:49.239364: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f169616b700 next 450 of size 8192000
2021-11-01 17:04:49.239396: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f169693b700 next 420 of size 8189952
2021-11-01 17:04:49.239428: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f169710af00 next 353 of size 8189952
2021-11-01 17:04:49.239455: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16978da700 next 361 of size 2048000
2021-11-01 17:04:49.239473: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f1697ace700 next 280 of size 6144000
2021-11-01 17:04:49.239494: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16980aa700 next 260 of size 8189952
2021-11-01 17:04:49.239524: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f1698879f00 next 442 of size 8192000
2021-11-01 17:04:49.239557: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f1699049f00 next 428 of size 2048000
2021-11-01 17:04:49.239590: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f169923df00 next 443 of size 8189952
2021-11-01 17:04:49.239616: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f1699a0d700 next 288 of size 8189952
2021-11-01 17:04:49.239632: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f169a1dcf00 next 311 of size 8189952
2021-11-01 17:04:49.239647: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f169a9ac700 next 220 of size 8189952
2021-11-01 17:04:49.239668: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f169b17bf00 next 302 of size 8189952
2021-11-01 17:04:49.239698: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x7f169b94b700 next 18446744073709551615 of size 40585472
2021-11-01 17:04:49.239730: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 134217728
2021-11-01 17:04:49.239764: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f169e000000 next 194 of size 8192000
2021-11-01 17:04:49.239798: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f169e7d0000 next 295 of size 8192000
2021-11-01 17:04:49.239832: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f169efa0000 next 336 of size 8192000
2021-11-01 17:04:49.239869: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f169f770000 next 389 of size 8192000
2021-11-01 17:04:49.239904: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f169ff40000 next 382 of size 8189952
2021-11-01 17:04:49.239940: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16a070f800 next 316 of size 8189952
2021-11-01 17:04:49.239973: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16a0edf000 next 215 of size 8189952
2021-11-01 17:04:49.240008: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16a16ae800 next 416 of size 8189952
2021-11-01 17:04:49.240042: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16a1e7e000 next 445 of size 8192000
2021-11-01 17:04:49.240077: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16a264e000 next 258 of size 8192000
2021-11-01 17:04:49.240110: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16a2e1e000 next 230 of size 8192000
2021-11-01 17:04:49.240146: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16a35ee000 next 368 of size 8192000
2021-11-01 17:04:49.240170: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16a3dbe000 next 281 of size 8189952
2021-11-01 17:04:49.240186: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16a458d800 next 371 of size 8189952
2021-11-01 17:04:49.240208: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16a4d5d000 next 319 of size 8189952
2021-11-01 17:04:49.240241: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16a552c800 next 18446744073709551615 of size 11352064
2021-11-01 17:04:49.240302: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 67108864
2021-11-01 17:04:49.240336: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16a6000000 next 436 of size 8192000
2021-11-01 17:04:49.240371: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16a67d0000 next 452 of size 8192000
2021-11-01 17:04:49.240406: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16a6fa0000 next 327 of size 8192000
2021-11-01 17:04:49.240440: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16a7770000 next 386 of size 8192000
2021-11-01 17:04:49.240473: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16a7f40000 next 210 of size 8189952
2021-11-01 17:04:49.240507: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16a870f800 next 405 of size 8189952
2021-11-01 17:04:49.240543: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16a8edf000 next 400 of size 8189952
2021-11-01 17:04:49.240580: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16a96ae800 next 18446744073709551615 of size 9771008
2021-11-01 17:04:49.240614: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 33554432
2021-11-01 17:04:49.240646: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16aa000000 next 398 of size 8189952
2021-11-01 17:04:49.240659: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16aa7cf800 next 478 of size 8189952
2021-11-01 17:04:49.240670: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16aaf9f000 next 315 of size 8189952
2021-11-01 17:04:49.240682: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16ab76e800 next 18446744073709551615 of size 8984576
2021-11-01 17:04:49.240693: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 8388608
2021-11-01 17:04:49.240704: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d2000000 next 43 of size 1048576
2021-11-01 17:04:49.240716: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d2100000 next 45 of size 1048576
2021-11-01 17:04:49.240728: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d2200000 next 46 of size 1048576
2021-11-01 17:04:49.240739: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d2300000 next 47 of size 1048576
2021-11-01 17:04:49.240750: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d2400000 next 58 of size 1048576
2021-11-01 17:04:49.240761: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d2500000 next 69 of size 1048576
2021-11-01 17:04:49.240773: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d2600000 next 73 of size 1048576
2021-11-01 17:04:49.240784: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d2700000 next 18446744073709551615 of size 1048576
2021-11-01 17:04:49.240795: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 16777216
2021-11-01 17:04:49.240806: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d2800000 next 95 of size 1048576
2021-11-01 17:04:49.240817: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d2900000 next 96 of size 1048576
2021-11-01 17:04:49.240828: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d2a00000 next 99 of size 1048576
2021-11-01 17:04:49.240839: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d2b00000 next 100 of size 1048576
2021-11-01 17:04:49.240851: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d2c00000 next 101 of size 1048576
2021-11-01 17:04:49.240862: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d2d00000 next 102 of size 1048576
2021-11-01 17:04:49.240873: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d2e00000 next 259 of size 724992
2021-11-01 17:04:49.240885: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d2eb1000 next 286 of size 1449728
2021-11-01 17:04:49.240896: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d3012f00 next 344 of size 1449728
2021-11-01 17:04:49.240908: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x7f16d3174e00 next 232 of size 741120
2021-11-01 17:04:49.240919: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d3229d00 next 241 of size 16128
2021-11-01 17:04:49.240931: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d322dc00 next 196 of size 12032
2021-11-01 17:04:49.240942: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x7f16d3230b00 next 370 of size 737024
2021-11-01 17:04:49.240951: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d32e4a00 next 379 of size 16128
2021-11-01 17:04:49.240959: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d32e8900 next 388 of size 16128
2021-11-01 17:04:49.240967: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d32ec800 next 209 of size 16128
2021-11-01 17:04:49.240975: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x7f16d32f0700 next 303 of size 4096
2021-11-01 17:04:49.240984: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d32f1700 next 343 of size 729088
2021-11-01 17:04:49.240992: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d33a3700 next 326 of size 724992
2021-11-01 17:04:49.241000: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x7f16d3454700 next 238 of size 724992
2021-11-01 17:04:49.241008: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d3505700 next 294 of size 724992
2021-11-01 17:04:49.241016: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x7f16d35b6700 next 404 of size 724992
2021-11-01 17:04:49.241024: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d3667700 next 270 of size 4096
2021-11-01 17:04:49.241033: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d3668700 next 206 of size 256
2021-11-01 17:04:49.241041: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d3668800 next 474 of size 256
2021-11-01 17:04:49.241049: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d3668900 next 332 of size 724992
2021-11-01 17:04:49.241057: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d3719900 next 475 of size 12032
2021-11-01 17:04:49.241065: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d371c800 next 195 of size 16128
2021-11-01 17:04:49.241073: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x7f16d3720700 next 214 of size 16128
2021-11-01 17:04:49.241081: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d3724600 next 237 of size 12032
2021-11-01 17:04:49.241089: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d3727500 next 470 of size 16128
2021-11-01 17:04:49.241097: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x7f16d372b400 next 340 of size 16128
2021-11-01 17:04:49.241105: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d372f300 next 464 of size 4096
2021-11-01 17:04:49.241113: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d3730300 next 273 of size 16128
2021-11-01 17:04:49.241121: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d3734200 next 274 of size 16128
2021-11-01 17:04:49.241129: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d3738100 next 418 of size 256
2021-11-01 17:04:49.241137: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d3738200 next 226 of size 16384
2021-11-01 17:04:49.241145: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d373c200 next 268 of size 16128
2021-11-01 17:04:49.241153: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x7f16d3740100 next 279 of size 48128
2021-11-01 17:04:49.241161: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f16d374bd00 next 378 of size 16128
2021-11-01 17:04:49.241169: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x7f16d374fc00 next 18446744073709551615 of size 721920
2021-11-01 17:04:49.241177: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 1048576
2021-11-01 17:04:49.241185: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd600000 next 1 of size 256
2021-11-01 17:04:49.241193: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd600100 next 2 of size 2048
2021-11-01 17:04:49.241201: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd600900 next 3 of size 2048
2021-11-01 17:04:49.241209: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd601100 next 4 of size 2048
2021-11-01 17:04:49.241217: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd601900 next 5 of size 256
2021-11-01 17:04:49.241225: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd601a00 next 6 of size 2048
2021-11-01 17:04:49.241233: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd602200 next 7 of size 256
2021-11-01 17:04:49.241241: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd602300 next 8 of size 256
2021-11-01 17:04:49.241249: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd602400 next 9 of size 2048
2021-11-01 17:04:49.241257: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd602c00 next 10 of size 2048
2021-11-01 17:04:49.241265: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd603400 next 11 of size 2048
2021-11-01 17:04:49.241273: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd603c00 next 12 of size 256
2021-11-01 17:04:49.241281: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd603d00 next 13 of size 256
2021-11-01 17:04:49.241289: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd603e00 next 14 of size 2048
2021-11-01 17:04:49.241297: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd604600 next 15 of size 2048
2021-11-01 17:04:49.241305: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd604e00 next 16 of size 256
2021-11-01 17:04:49.241313: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd604f00 next 17 of size 2048
2021-11-01 17:04:49.241321: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd605700 next 18 of size 2048
2021-11-01 17:04:49.241329: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd605f00 next 19 of size 2048
2021-11-01 17:04:49.241337: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd606700 next 20 of size 256
2021-11-01 17:04:49.241345: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd606800 next 21 of size 2048
2021-11-01 17:04:49.241353: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd607000 next 22 of size 256
2021-11-01 17:04:49.241361: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd607100 next 23 of size 256
2021-11-01 17:04:49.241369: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd607200 next 24 of size 256
2021-11-01 17:04:49.241377: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd607300 next 25 of size 2048
2021-11-01 17:04:49.241385: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd607b00 next 26 of size 256
2021-11-01 17:04:49.241393: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd607c00 next 27 of size 2048
2021-11-01 17:04:49.241401: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd608400 next 28 of size 2048
2021-11-01 17:04:49.241409: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd608c00 next 29 of size 2048
2021-11-01 17:04:49.241417: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd609400 next 30 of size 256
2021-11-01 17:04:49.241425: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd609500 next 31 of size 256
2021-11-01 17:04:49.241433: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd609600 next 32 of size 2048
2021-11-01 17:04:49.241441: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd609e00 next 35 of size 6144
2021-11-01 17:04:49.241449: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd60b600 next 38 of size 4096
2021-11-01 17:04:49.241457: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd60c600 next 41 of size 1280
2021-11-01 17:04:49.241466: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd60cb00 next 44 of size 2048
2021-11-01 17:04:49.241473: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd60d300 next 48 of size 2048
2021-11-01 17:04:49.241481: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd60db00 next 49 of size 2048
2021-11-01 17:04:49.241490: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd60e300 next 50 of size 4096
2021-11-01 17:04:49.241497: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd60f300 next 51 of size 2048
2021-11-01 17:04:49.241505: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd60fb00 next 52 of size 2048
2021-11-01 17:04:49.241513: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd610300 next 53 of size 2048
2021-11-01 17:04:49.241521: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd610b00 next 54 of size 2048
2021-11-01 17:04:49.241530: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd611300 next 55 of size 256
2021-11-01 17:04:49.241537: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd611400 next 56 of size 256
2021-11-01 17:04:49.241546: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd611500 next 57 of size 2048
2021-11-01 17:04:49.241568: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd611d00 next 59 of size 2048
2021-11-01 17:04:49.241575: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd612500 next 60 of size 4096
2021-11-01 17:04:49.241583: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd613500 next 61 of size 2048
2021-11-01 17:04:49.241591: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd613d00 next 62 of size 2048
2021-11-01 17:04:49.241598: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd614500 next 63 of size 2048
2021-11-01 17:04:49.241606: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd614d00 next 64 of size 2048
2021-11-01 17:04:49.241613: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd615500 next 65 of size 2048
2021-11-01 17:04:49.241621: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd615d00 next 66 of size 2048
2021-11-01 17:04:49.241628: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd616500 next 67 of size 4096
2021-11-01 17:04:49.241636: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd617500 next 68 of size 2048
2021-11-01 17:04:49.241644: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd617d00 next 70 of size 2048
2021-11-01 17:04:49.241651: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd618500 next 71 of size 2048
2021-11-01 17:04:49.241659: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd618d00 next 72 of size 2048
2021-11-01 17:04:49.241667: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd619500 next 74 of size 2048
2021-11-01 17:04:49.241675: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd619d00 next 75 of size 2048
2021-11-01 17:04:49.241683: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd61a500 next 76 of size 256
2021-11-01 17:04:49.241690: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd61a600 next 77 of size 256
2021-11-01 17:04:49.241698: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd61a700 next 78 of size 2048
2021-11-01 17:04:49.241706: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd61af00 next 79 of size 256
2021-11-01 17:04:49.241713: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd61b000 next 80 of size 2048
2021-11-01 17:04:49.241722: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd61b800 next 81 of size 2048
2021-11-01 17:04:49.241729: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd61c000 next 82 of size 2048
2021-11-01 17:04:49.241737: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd61c800 next 83 of size 2048
2021-11-01 17:04:49.241744: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd61d000 next 84 of size 2048
2021-11-01 17:04:49.241752: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd61d800 next 85 of size 2048
2021-11-01 17:04:49.241760: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd61e000 next 86 of size 2048
2021-11-01 17:04:49.241767: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd61e800 next 87 of size 2048
2021-11-01 17:04:49.241775: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd61f000 next 88 of size 256
2021-11-01 17:04:49.241782: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd61f100 next 89 of size 2048
2021-11-01 17:04:49.241790: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd61f900 next 90 of size 6144
2021-11-01 17:04:49.241798: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd621100 next 91 of size 2048
2021-11-01 17:04:49.241806: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd621900 next 92 of size 2048
2021-11-01 17:04:49.241813: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd622100 next 93 of size 2048
2021-11-01 17:04:49.241821: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd622900 next 97 of size 2048
2021-11-01 17:04:49.241828: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd623100 next 98 of size 2048
2021-11-01 17:04:49.241836: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd623900 next 103 of size 6144
2021-11-01 17:04:49.241844: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd625100 next 104 of size 6144
2021-11-01 17:04:49.241851: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd626900 next 118 of size 256
2021-11-01 17:04:49.241859: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd626a00 next 115 of size 256
2021-11-01 17:04:49.241867: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd626b00 next 119 of size 256
2021-11-01 17:04:49.241874: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd626c00 next 117 of size 256
2021-11-01 17:04:49.241882: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd626d00 next 124 of size 256
2021-11-01 17:04:49.241890: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd626e00 next 106 of size 256
2021-11-01 17:04:49.241898: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd626f00 next 107 of size 256
2021-11-01 17:04:49.241905: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd627000 next 113 of size 256
2021-11-01 17:04:49.241913: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd627100 next 131 of size 256
2021-11-01 17:04:49.241920: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd627200 next 123 of size 256
2021-11-01 17:04:49.241928: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd627300 next 112 of size 256
2021-11-01 17:04:49.241935: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd627400 next 114 of size 256
2021-11-01 17:04:49.241943: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd627500 next 105 of size 256
2021-11-01 17:04:49.241950: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd627600 next 137 of size 256
2021-11-01 17:04:49.241958: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd627700 next 138 of size 256
2021-11-01 17:04:49.252116: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd627800 next 135 of size 256
2021-11-01 17:04:49.252135: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd627900 next 136 of size 16128
2021-11-01 17:04:49.252143: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd62b800 next 109 of size 32000
2021-11-01 17:04:49.252150: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd633500 next 126 of size 32000
2021-11-01 17:04:49.252156: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd63b200 next 132 of size 16128
2021-11-01 17:04:49.252162: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd63f100 next 133 of size 16128
2021-11-01 17:04:49.252169: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd643000 next 110 of size 16128
2021-11-01 17:04:49.252175: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd646f00 next 111 of size 16128
2021-11-01 17:04:49.252181: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd64ae00 next 120 of size 16128
2021-11-01 17:04:49.252188: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd64ed00 next 122 of size 16128
2021-11-01 17:04:49.252194: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd652c00 next 134 of size 16128
2021-11-01 17:04:49.252200: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd656b00 next 139 of size 16128
2021-11-01 17:04:49.252207: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd65aa00 next 108 of size 16128
2021-11-01 17:04:49.252213: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd65e900 next 129 of size 16128
2021-11-01 17:04:49.252219: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd662800 next 130 of size 256
2021-11-01 17:04:49.252226: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd662900 next 128 of size 4096
2021-11-01 17:04:49.252232: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd663900 next 127 of size 12032
2021-11-01 17:04:49.252238: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd666800 next 121 of size 256
2021-11-01 17:04:49.252244: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd666900 next 125 of size 256
2021-11-01 17:04:49.252260: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd666a00 next 116 of size 16128
2021-11-01 17:04:49.252267: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66a900 next 140 of size 256
2021-11-01 17:04:49.252273: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66aa00 next 141 of size 256
2021-11-01 17:04:49.252279: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66ab00 next 142 of size 256
2021-11-01 17:04:49.252286: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66ac00 next 143 of size 256
2021-11-01 17:04:49.252292: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66ad00 next 144 of size 256
2021-11-01 17:04:49.252298: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66ae00 next 145 of size 256
2021-11-01 17:04:49.252304: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66af00 next 146 of size 256
2021-11-01 17:04:49.252311: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66b000 next 147 of size 256
2021-11-01 17:04:49.252317: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66b100 next 148 of size 256
2021-11-01 17:04:49.252323: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66b200 next 149 of size 256
2021-11-01 17:04:49.252329: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66b300 next 150 of size 256
2021-11-01 17:04:49.252336: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66b400 next 151 of size 256
2021-11-01 17:04:49.252342: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66b500 next 152 of size 256
2021-11-01 17:04:49.252348: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66b600 next 153 of size 256
2021-11-01 17:04:49.252355: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66b700 next 154 of size 256
2021-11-01 17:04:49.252361: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66b800 next 155 of size 256
2021-11-01 17:04:49.252367: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66b900 next 156 of size 256
2021-11-01 17:04:49.252373: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66ba00 next 157 of size 256
2021-11-01 17:04:49.252379: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66bb00 next 158 of size 256
2021-11-01 17:04:49.252386: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66bc00 next 159 of size 256
2021-11-01 17:04:49.252392: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66bd00 next 160 of size 256
2021-11-01 17:04:49.252398: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66be00 next 161 of size 256
2021-11-01 17:04:49.252404: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66bf00 next 162 of size 256
2021-11-01 17:04:49.252410: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66c000 next 163 of size 256
2021-11-01 17:04:49.252416: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66c100 next 164 of size 256
2021-11-01 17:04:49.252422: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66c200 next 165 of size 256
2021-11-01 17:04:49.252428: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66c300 next 166 of size 256
2021-11-01 17:04:49.252435: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66c400 next 167 of size 256
2021-11-01 17:04:49.252441: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66c500 next 168 of size 256
2021-11-01 17:04:49.252447: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66c600 next 169 of size 256
2021-11-01 17:04:49.252453: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66c700 next 170 of size 256
2021-11-01 17:04:49.252459: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66c800 next 171 of size 256
2021-11-01 17:04:49.252465: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66c900 next 172 of size 256
2021-11-01 17:04:49.252472: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66ca00 next 173 of size 256
2021-11-01 17:04:49.252478: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66cb00 next 174 of size 256
2021-11-01 17:04:49.252484: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66cc00 next 175 of size 256
2021-11-01 17:04:49.252490: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66cd00 next 176 of size 256
2021-11-01 17:04:49.252496: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66ce00 next 177 of size 256
2021-11-01 17:04:49.252502: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66cf00 next 178 of size 256
2021-11-01 17:04:49.252508: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66d000 next 179 of size 256
2021-11-01 17:04:49.252515: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66d100 next 180 of size 256
2021-11-01 17:04:49.252521: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66d200 next 181 of size 256
2021-11-01 17:04:49.252527: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66d300 next 182 of size 256
2021-11-01 17:04:49.252533: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66d400 next 183 of size 256
2021-11-01 17:04:49.252540: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66d500 next 184 of size 256
2021-11-01 17:04:49.252546: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66d600 next 185 of size 256
2021-11-01 17:04:49.252553: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66d700 next 186 of size 256
2021-11-01 17:04:49.252559: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66d800 next 187 of size 256
2021-11-01 17:04:49.252565: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66d900 next 188 of size 256
2021-11-01 17:04:49.252571: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66da00 next 189 of size 256
2021-11-01 17:04:49.252577: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66db00 next 190 of size 256
2021-11-01 17:04:49.252584: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66dc00 next 191 of size 256
2021-11-01 17:04:49.252590: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66dd00 next 192 of size 256
2021-11-01 17:04:49.252596: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66de00 next 369 of size 256
2021-11-01 17:04:49.252602: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66df00 next 365 of size 256
2021-11-01 17:04:49.252609: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66e000 next 487 of size 256
2021-11-01 17:04:49.252615: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66e100 next 422 of size 256
2021-11-01 17:04:49.252621: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66e200 next 423 of size 256
2021-11-01 17:04:49.252628: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66e300 next 334 of size 256
2021-11-01 17:04:49.252634: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66e400 next 468 of size 256
2021-11-01 17:04:49.252640: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66e500 next 318 of size 256
2021-11-01 17:04:49.252646: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66e600 next 267 of size 256
2021-11-01 17:04:49.252653: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66e700 next 425 of size 256
2021-11-01 17:04:49.252659: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66e800 next 216 of size 256
2021-11-01 17:04:49.252666: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66e900 next 349 of size 256
2021-11-01 17:04:49.252672: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66ea00 next 287 of size 256
2021-11-01 17:04:49.252678: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd66eb00 next 481 of size 16128
2021-11-01 17:04:49.252684: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd672a00 next 290 of size 16128
2021-11-01 17:04:49.252691: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd676900 next 390 of size 32000
2021-11-01 17:04:49.252697: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd67e600 next 323 of size 32000
2021-11-01 17:04:49.252704: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd686300 next 309 of size 32000
2021-11-01 17:04:49.252710: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd68e000 next 364 of size 32000
2021-11-01 17:04:49.252717: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd695d00 next 227 of size 16128
2021-11-01 17:04:49.252723: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd699c00 next 373 of size 16128
2021-11-01 17:04:49.252729: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd69db00 next 278 of size 16128
2021-11-01 17:04:49.252735: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6a1a00 next 360 of size 16128
2021-11-01 17:04:49.252742: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6a5900 next 346 of size 16128
2021-11-01 17:04:49.252748: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6a9800 next 449 of size 16128
2021-11-01 17:04:49.252755: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6ad700 next 233 of size 16128
2021-11-01 17:04:49.252761: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6b1600 next 341 of size 16128
2021-11-01 17:04:49.252767: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6b5500 next 284 of size 16128
2021-11-01 17:04:49.252773: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6b9400 next 211 of size 16128
2021-11-01 17:04:49.252780: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6bd300 next 471 of size 16128
2021-11-01 17:04:49.252786: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6c1200 next 201 of size 16128
2021-11-01 17:04:49.252793: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6c5100 next 419 of size 16128
2021-11-01 17:04:49.252799: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6c9000 next 458 of size 16128
2021-11-01 17:04:49.252805: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6ccf00 next 299 of size 16128
2021-11-01 17:04:49.252811: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6d0e00 next 251 of size 16128
2021-11-01 17:04:49.252818: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6d4d00 next 465 of size 16128
2021-11-01 17:04:49.252824: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6d8c00 next 312 of size 16128
2021-11-01 17:04:49.252831: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6dcb00 next 339 of size 256
2021-11-01 17:04:49.252837: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6dcc00 next 427 of size 256
2021-11-01 17:04:49.252843: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6dcd00 next 197 of size 256
2021-11-01 17:04:49.252849: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6dce00 next 335 of size 256
2021-11-01 17:04:49.252856: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6dcf00 next 461 of size 256
2021-11-01 17:04:49.252862: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6dd000 next 432 of size 256
2021-11-01 17:04:49.252868: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6dd100 next 444 of size 256
2021-11-01 17:04:49.252874: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6dd200 next 477 of size 256
2021-11-01 17:04:49.252881: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6dd300 next 212 of size 256
2021-11-01 17:04:49.252887: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6dd400 next 377 of size 256
2021-11-01 17:04:49.252893: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6dd500 next 243 of size 256
2021-11-01 17:04:49.252899: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6dd600 next 337 of size 256
2021-11-01 17:04:49.252905: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6dd700 next 394 of size 256
2021-11-01 17:04:49.252911: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6dd800 next 223 of size 256
2021-11-01 17:04:49.252917: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6dd900 next 385 of size 256
2021-11-01 17:04:49.252924: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6dda00 next 310 of size 256
2021-11-01 17:04:49.252930: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6ddb00 next 421 of size 256
2021-11-01 17:04:49.252936: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6ddc00 next 218 of size 256
2021-11-01 17:04:49.252942: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6ddd00 next 396 of size 256
2021-11-01 17:04:49.252948: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6dde00 next 397 of size 256
2021-11-01 17:04:49.252954: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6ddf00 next 322 of size 4096
2021-11-01 17:04:49.252961: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6def00 next 297 of size 16128
2021-11-01 17:04:49.252967: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6e2e00 next 222 of size 16128
2021-11-01 17:04:49.252974: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6e6d00 next 362 of size 4096
2021-11-01 17:04:49.252980: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6e7d00 next 482 of size 12032
2021-11-01 17:04:49.252986: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6eac00 next 261 of size 16128
2021-11-01 17:04:49.252992: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x7f17cd6eeb00 next 271 of size 28160
2021-11-01 17:04:49.252999: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cd6f5900 next 409 of size 16128
2021-11-01 17:04:49.253005: I tensorflow/core/common_runtime/bfc_allocator.cc:905] Free at 0x7f17cd6f9800 next 18446744073709551615 of size 26624
2021-11-01 17:04:49.253011: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 2097152
2021-11-01 17:04:49.253018: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cda00000 next 34 of size 1048576
2021-11-01 17:04:49.253025: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cdb00000 next 18446744073709551615 of size 1048576
2021-11-01 17:04:49.253031: I tensorflow/core/common_runtime/bfc_allocator.cc:898] Next region of size 4194304
2021-11-01 17:04:49.253038: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cdc00000 next 37 of size 1048576
2021-11-01 17:04:49.253044: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cdd00000 next 39 of size 1048576
2021-11-01 17:04:49.253051: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cde00000 next 40 of size 1048576
2021-11-01 17:04:49.253057: I tensorflow/core/common_runtime/bfc_allocator.cc:905] InUse at 0x7f17cdf00000 next 18446744073709551615 of size 1048576
2021-11-01 17:04:49.253063: I tensorflow/core/common_runtime/bfc_allocator.cc:914] Summary of in-use Chunks by size:
2021-11-01 17:04:49.253074: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 128 Chunks of size 256 totalling 32.0KiB
2021-11-01 17:04:49.253081: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 1280 totalling 1.2KiB
2021-11-01 17:04:49.253089: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 54 Chunks of size 2048 totalling 108.0KiB
2021-11-01 17:04:49.253096: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 9 Chunks of size 4096 totalling 36.0KiB
2021-11-01 17:04:49.253103: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 4 Chunks of size 6144 totalling 24.0KiB
2021-11-01 17:04:49.253111: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 5 Chunks of size 12032 totalling 58.8KiB
2021-11-01 17:04:49.253118: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 46 Chunks of size 16128 totalling 724.5KiB
2021-11-01 17:04:49.253125: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 16384 totalling 16.0KiB
2021-11-01 17:04:49.253132: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 6 Chunks of size 32000 totalling 187.5KiB
2021-11-01 17:04:49.253139: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 4 Chunks of size 724992 totalling 2.77MiB
2021-11-01 17:04:49.253147: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 729088 totalling 712.0KiB
2021-11-01 17:04:49.253154: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 25 Chunks of size 1048576 totalling 25.00MiB
2021-11-01 17:04:49.253161: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 1449728 totalling 2.76MiB
2021-11-01 17:04:49.253168: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 2 Chunks of size 2048000 totalling 3.91MiB
2021-11-01 17:04:49.253175: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 6144000 totalling 5.86MiB
2021-11-01 17:04:49.253182: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 26 Chunks of size 8189952 totalling 203.07MiB
2021-11-01 17:04:49.253190: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 24 Chunks of size 8192000 totalling 187.50MiB
2021-11-01 17:04:49.253197: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 8984576 totalling 8.57MiB
2021-11-01 17:04:49.253204: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 9771008 totalling 9.32MiB
2021-11-01 17:04:49.253211: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 11352064 totalling 10.83MiB
2021-11-01 17:04:49.253218: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 5 Chunks of size 371101696 totalling 1.73GiB
2021-11-01 17:04:49.253225: I tensorflow/core/common_runtime/bfc_allocator.cc:917] 1 Chunks of size 438624256 totalling 418.30MiB
2021-11-01 17:04:49.253231: I tensorflow/core/common_runtime/bfc_allocator.cc:921] Sum Total of in-use chunks: 2.59GiB
2021-11-01 17:04:49.253238: I tensorflow/core/common_runtime/bfc_allocator.cc:923] total_region_allocated_bytes_: 3327262720 memory_limit_: 3327262720 available bytes: 0 curr_region_allocation_bytes_: 4294967296
2021-11-01 17:04:49.253248: I tensorflow/core/common_runtime/bfc_allocator.cc:929] Stats:
Limit: 3327262720
InUse: 2777986560
MaxInUse: 2777986560
NumAllocs: 604462
MaxAllocSize: 438624256
2021-11-01 17:04:49.253278: W tensorflow/core/common_runtime/bfc_allocator.cc:424] x_____________****************
2021-11-01 17:04:49.253307: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at cwise_ops_common.cc:82 : Resource exhausted: OOM when allocating tensor with shape[181202,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
*** Process received signal ***
Signal: Segmentation fault (11)
Signal code: Address not mapped (1)
Failing at address: 0x10
[ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f1901b7b3c0]
[ 2] /home/apollo/miniconda3/envs/SimNet_conda/lib/python3.6/site-packages/tensorflow_core/python/…/libtensorflow_framework.so.1(_ZN10tensorflow13BaseGPUDevice7ComputeEPNS_8OpKernelEPNS_15OpKernelContextE+0x512)[0x7f18c4b64ef2]
[ 3] /home/apollo/miniconda3/envs/SimNet_conda/lib/python3.6/site-packages/tensorflow_core/python/…/libtensorflow_framework.so.1(+0xfaf19e)[0x7f18c4bbf19e]
[ 4] /home/apollo/miniconda3/envs/SimNet_conda/lib/python3.6/site-packages/tensorflow_core/python/…/libtensorflow_framework.so.1(+0xfaf64f)[0x7f18c4bbf64f]
[ 5] /home/apollo/miniconda3/envs/SimNet_conda/lib/python3.6/site-packages/tensorflow_core/python/…/libtensorflow_framework.so.1(_ZN5Eigen15ThreadPoolTemplIN10tensorflow6thread16EigenEnvironmentEE10WorkerLoopEi+0x291)[0x7f18c4c64e21]
[ 6] /home/apollo/miniconda3/envs/SimNet_conda/lib/python3.6/site-packages/tensorflow_core/python/…/libtensorflow_framework.so.1(_ZNSt17_Function_handlerIFvvEZN10tensorflow6thread16EigenEnvironment12CreateThreadESt8functionIS0_EEUlvE_E9_M_invokeERKSt9_Any_data+0x48)[0x7f18c4c624b8]
[ 7] /home/apollo/miniconda3/envs/SimNet_conda/lib/python3.6/site-packages/tensorflow_core/python/…/…/…/…/libstdc++.so.6(+0xc819d)[0x7f18bd14119d]
[ 8] /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609)[0x7f1901b6f609]
[ 9] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f1901a96293]
*** End of error message ***
Segmentation fault (core dumped)