Hi!
I’m using 3566 images to train convolutional neural network implemented with tensorFlow + Keras, but I have some problems.
I have configurated 4GB of swapfile.
An error that Jetson returns to me is that I have excedeed 10% of system memory, while I still have 11GB of free memory. So, I don’t know wich memory is the message refering to.
Sometimes I am able to execute my code without problems, so I don’t know wich is the exact problem.
Is someone experiencing the same problems as me?
I would appreciate if someone could lend me a hand.
I also have a problem with the histoy parameter that the fit functions returns to me.
All errors are printed in the message down below when I have an error in my execution:
MEMORY ERROR*****
2019-11-19 13:01:15.026448: W tensorflow/core/common_runtime/bfc_allocator.cc:314] Allocator (GPU_0_bfc) ran out of memory trying to allocate 72.0KiB (rounded to 73728). Current allocation summary follows.
2019-11-19 13:01:15.026624: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (256): Total Chunks: 68, Chunks in use: 68. 17.0KiB allocated for chunks. 17.0KiB in use in bin. 2.7KiB client-requested in use in bin.
2019-11-19 13:01:15.026738: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (512): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.026850: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2019-11-19 13:01:15.026968: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2048): Total Chunks: 4, Chunks in use: 4. 13.0KiB allocated for chunks. 13.0KiB in use in bin. 12.5KiB client-requested in use in bin.
2019-11-19 13:01:15.027078: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.027196: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8192): Total Chunks: 4, Chunks in use: 4. 32.0KiB allocated for chunks. 32.0KiB in use in bin. 32.0KiB client-requested in use in bin.
2019-11-19 13:01:15.027273: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.027368: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.027445: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (65536): Total Chunks: 3, Chunks in use: 3. 216.0KiB allocated for chunks. 216.0KiB in use in bin. 216.0KiB client-requested in use in bin.
2019-11-19 13:01:15.027509: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.027573: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.027644: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (524288): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.027725: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1048576): Total Chunks: 1, Chunks in use: 1. 1.22MiB allocated for chunks. 1.22MiB in use in bin. 1.22MiB client-requested in use in bin.
2019-11-19 13:01:15.027807: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.027891: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.027988: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8388608): Total Chunks: 3, Chunks in use: 3. 24.80MiB allocated for chunks. 24.80MiB in use in bin. 24.80MiB client-requested in use in bin.
2019-11-19 13:01:15.028078: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.028176: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (33554432): Total Chunks: 1, Chunks in use: 1. 58.11MiB allocated for chunks. 58.11MiB in use in bin. 36.00MiB client-requested in use in bin.
2019-11-19 13:01:15.028265: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.028356: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.028448: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.028554: I tensorflow/core/common_runtime/bfc_allocator.cc:780] Bin for 72.0KiB was 64.0KiB, Chunk State:
2019-11-19 13:01:15.028623: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 88502272
2019-11-19 13:01:15.028677: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850000 next 1 of size 1280
2019-11-19 13:01:15.028720: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850500 next 2 of size 256
2019-11-19 13:01:15.028763: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850600 next 3 of size 256
2019-11-19 13:01:15.028808: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850700 next 4 of size 256
2019-11-19 13:01:15.028849: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850800 next 5 of size 256
2019-11-19 13:01:15.028894: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850900 next 6 of size 256
2019-11-19 13:01:15.028936: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850a00 next 7 of size 256
2019-11-19 13:01:15.028981: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850b00 next 8 of size 256
2019-11-19 13:01:15.029030: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850c00 next 9 of size 256
2019-11-19 13:01:15.029083: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850d00 next 10 of size 256
2019-11-19 13:01:15.029137: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850e00 next 11 of size 256
2019-11-19 13:01:15.029192: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850f00 next 12 of size 256
2019-11-19 13:01:15.029249: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00851000 next 13 of size 256
2019-11-19 13:01:15.029309: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00851100 next 14 of size 256
2019-11-19 13:01:15.029370: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00851200 next 15 of size 256
2019-11-19 13:01:15.029431: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00851300 next 16 of size 256
2019-11-19 13:01:15.029493: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00851400 next 17 of size 256
2019-11-19 13:01:15.029551: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00851500 next 18 of size 256
2019-11-19 13:01:15.029613: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00851600 next 19 of size 256
2019-11-19 13:01:15.029670: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00851700 next 20 of size 256
2019-11-19 13:01:15.029726: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00851800 next 21 of size 256
2019-11-19 13:01:15.029786: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00851900 next 22 of size 256
2019-11-19 13:01:15.029845: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00851a00 next 23 of size 8192
2019-11-19 13:01:15.029906: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00853a00 next 24 of size 8667136
2019-11-19 13:01:15.029963: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01097a00 next 25 of size 256
2019-11-19 13:01:15.030019: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01097b00 next 26 of size 73728
2019-11-19 13:01:15.030081: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf010a9b00 next 27 of size 3328
2019-11-19 13:01:15.030141: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf010aa800 next 28 of size 256
2019-11-19 13:01:15.030185: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf010aa900 next 29 of size 256
2019-11-19 13:01:15.030223: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf010aaa00 next 30 of size 8192
2019-11-19 13:01:15.030260: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf010aca00 next 31 of size 256
2019-11-19 13:01:15.030297: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf010acb00 next 32 of size 8667136
2019-11-19 13:01:15.030334: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf018f0b00 next 33 of size 73728
2019-11-19 13:01:15.030368: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01902b00 next 34 of size 3328
2019-11-19 13:01:15.030402: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01903800 next 35 of size 256
2019-11-19 13:01:15.030509: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01903900 next 36 of size 256
2019-11-19 13:01:15.030561: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01903a00 next 37 of size 256
2019-11-19 13:01:15.030602: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01903b00 next 38 of size 256
2019-11-19 13:01:15.030710: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01903c00 next 39 of size 256
2019-11-19 13:01:15.030746: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01903d00 next 40 of size 256
2019-11-19 13:01:15.030773: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01903e00 next 41 of size 256
2019-11-19 13:01:15.030798: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01903f00 next 42 of size 256
2019-11-19 13:01:15.030825: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01904000 next 43 of size 256
2019-11-19 13:01:15.030851: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01904100 next 44 of size 256
2019-11-19 13:01:15.030877: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01904200 next 45 of size 256
2019-11-19 13:01:15.030904: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01904300 next 46 of size 256
2019-11-19 13:01:15.030930: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01904400 next 47 of size 256
2019-11-19 13:01:15.030956: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01904500 next 48 of size 256
2019-11-19 13:01:15.030984: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01904600 next 49 of size 1280000
2019-11-19 13:01:15.031013: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3ce00 next 50 of size 256
2019-11-19 13:01:15.031041: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3cf00 next 51 of size 256
2019-11-19 13:01:15.031068: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3d000 next 52 of size 256
2019-11-19 13:01:15.031094: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3d100 next 53 of size 256
2019-11-19 13:01:15.031122: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3d200 next 54 of size 256
2019-11-19 13:01:15.031150: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3d300 next 55 of size 256
2019-11-19 13:01:15.031176: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3d400 next 56 of size 256
2019-11-19 13:01:15.031202: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3d500 next 57 of size 256
2019-11-19 13:01:15.031228: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3d600 next 58 of size 256
2019-11-19 13:01:15.031255: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3d700 next 59 of size 256
2019-11-19 13:01:15.031281: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3d800 next 60 of size 256
2019-11-19 13:01:15.031308: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3d900 next 61 of size 8192
2019-11-19 13:01:15.031392: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3f900 next 62 of size 256
2019-11-19 13:01:15.031427: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3fa00 next 63 of size 3328
2019-11-19 13:01:15.031448: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a40700 next 64 of size 256
2019-11-19 13:01:15.031471: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a40800 next 65 of size 3328
2019-11-19 13:01:15.031499: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a41500 next 66 of size 256
2019-11-19 13:01:15.031523: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a41600 next 67 of size 256
2019-11-19 13:01:15.031562: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a41700 next 68 of size 256
2019-11-19 13:01:15.031624: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a41800 next 69 of size 8667136
2019-11-19 13:01:15.031664: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02285800 next 70 of size 256
2019-11-19 13:01:15.031698: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02285900 next 71 of size 256
2019-11-19 13:01:15.031730: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02285a00 next 72 of size 8192
2019-11-19 13:01:15.031762: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02287a00 next 73 of size 256
2019-11-19 13:01:15.031795: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02287b00 next 74 of size 256
2019-11-19 13:01:15.031827: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02287c00 next 75 of size 256
2019-11-19 13:01:15.031859: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02287d00 next 76 of size 256
2019-11-19 13:01:15.031893: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02287e00 next 77 of size 256
2019-11-19 13:01:15.031926: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02287f00 next 78 of size 256
2019-11-19 13:01:15.031961: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02288000 next 79 of size 256
2019-11-19 13:01:15.031996: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02288100 next 80 of size 256
2019-11-19 13:01:15.032031: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02288200 next 81 of size 256
2019-11-19 13:01:15.032066: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02288300 next 82 of size 256
2019-11-19 13:01:15.032102: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02288400 next 83 of size 256
2019-11-19 13:01:15.032138: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02288500 next 84 of size 73728
2019-11-19 13:01:15.032176: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf0229a500 next 18446744073709551615 of size 60934912
2019-11-19 13:01:15.032213: I tensorflow/core/common_runtime/bfc_allocator.cc:809] Summary of in-use Chunks by size:
2019-11-19 13:01:15.032260: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 68 Chunks of size 256 totalling 17.0KiB
2019-11-19 13:01:15.032302: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 1280 totalling 1.2KiB
2019-11-19 13:01:15.032343: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 3328 totalling 13.0KiB
2019-11-19 13:01:15.032383: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 8192 totalling 32.0KiB
2019-11-19 13:01:15.032423: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 3 Chunks of size 73728 totalling 216.0KiB
2019-11-19 13:01:15.032463: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 1280000 totalling 1.22MiB
2019-11-19 13:01:15.032506: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 3 Chunks of size 8667136 totalling 24.80MiB
2019-11-19 13:01:15.032549: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 60934912 totalling 58.11MiB
2019-11-19 13:01:15.032591: I tensorflow/core/common_runtime/bfc_allocator.cc:816] Sum Total of in-use chunks: 84.40MiB
2019-11-19 13:01:15.032630: I tensorflow/core/common_runtime/bfc_allocator.cc:818] total_region_allocated_bytes_: 88502272 memory_limit_: 88502272 available bytes: 0 curr_region_allocation_bytes_: 177004544
2019-11-19 13:01:15.032678: I tensorflow/core/common_runtime/bfc_allocator.cc:824] Stats:
Limit: 88502272
InUse: 88502272
MaxInUse: 88502272
NumAllocs: 85
MaxAllocSize: 60934912
2019-11-19 13:01:15.032790: W tensorflow/core/common_runtime/bfc_allocator.cc:319] **************************************************************************xxxxxxxxxxxxxxxxxxxxxxxxxx
2019-11-19 13:01:15.032942: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at cwise_ops_common.cc:70 : Resource exhausted: OOM when allocating tensor with shape[3,3,32,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-11-19 13:01:15.032943: W tensorflow/core/common_runtime/bfc_allocator.cc:314] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.1KiB (rounded to 3328). Current allocation summary follows.
2019-11-19 13:01:15.033059: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (256): Total Chunks: 68, Chunks in use: 68. 17.0KiB allocated for chunks. 17.0KiB in use in bin. 2.7KiB client-requested in use in bin.
2019-11-19 13:01:15.033101: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (512): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.033134: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2019-11-19 13:01:15.033180: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2048): Total Chunks: 4, Chunks in use: 4. 13.0KiB allocated for chunks. 13.0KiB in use in bin. 12.5KiB client-requested in use in bin.
2019-11-19 13:01:15.033212: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.033243: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8192): Total Chunks: 4, Chunks in use: 4. 32.0KiB allocated for chunks. 32.0KiB in use in bin. 32.0KiB client-requested in use in bin.
2019-11-19 13:01:15.033270: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.033296: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.033328: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (65536): Total Chunks: 3, Chunks in use: 3. 216.0KiB allocated for chunks. 216.0KiB in use in bin. 216.0KiB client-requested in use in bin.
2019-11-19 13:01:15.033355: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.033382: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.033409: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (524288): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.033441: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1048576): Total Chunks: 1, Chunks in use: 1. 1.22MiB allocated for chunks. 1.22MiB in use in bin. 1.22MiB client-requested in use in bin.
2019-11-19 13:01:15.033471: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.033500: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.033534: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8388608): Total Chunks: 3, Chunks in use: 3. 24.80MiB allocated for chunks. 24.80MiB in use in bin. 24.80MiB client-requested in use in bin.
2019-11-19 13:01:15.033560: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.033618: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (33554432): Total Chunks: 1, Chunks in use: 1. 58.11MiB allocated for chunks. 58.11MiB in use in bin. 36.00MiB client-requested in use in bin.
2019-11-19 13:01:15.035194: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.035275: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.035377: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-11-19 13:01:15.035444: I tensorflow/core/common_runtime/bfc_allocator.cc:780] Bin for 3.2KiB was 2.0KiB, Chunk State:
2019-11-19 13:01:15.035495: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 88502272
2019-11-19 13:01:15.035548: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850000 next 1 of size 1280
2019-11-19 13:01:15.035594: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850500 next 2 of size 256
2019-11-19 13:01:15.035636: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850600 next 3 of size 256
2019-11-19 13:01:15.035679: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850700 next 4 of size 256
2019-11-19 13:01:15.035725: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850800 next 5 of size 256
2019-11-19 13:01:15.035767: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850900 next 6 of size 256
2019-11-19 13:01:15.035810: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850a00 next 7 of size 256
2019-11-19 13:01:15.035855: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850b00 next 8 of size 256
2019-11-19 13:01:15.035906: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850c00 next 9 of size 256
2019-11-19 13:01:15.035956: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850d00 next 10 of size 256
2019-11-19 13:01:15.036003: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850e00 next 11 of size 256
2019-11-19 13:01:15.036045: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850f00 next 12 of size 256
2019-11-19 13:01:15.036083: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00851000 next 13 of size 256
2019-11-19 13:01:15.036123: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00851100 next 14 of size 256
2019-11-19 13:01:15.036165: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00851200 next 15 of size 256
2019-11-19 13:01:15.036205: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00851300 next 16 of size 256
2019-11-19 13:01:15.036245: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00851400 next 17 of size 256
2019-11-19 13:01:15.036286: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00851500 next 18 of size 256
2019-11-19 13:01:15.036326: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00851600 next 19 of size 256
2019-11-19 13:01:15.036366: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00851700 next 20 of size 256
2019-11-19 13:01:15.036404: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00851800 next 21 of size 256
2019-11-19 13:01:15.036443: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00851900 next 22 of size 256
2019-11-19 13:01:15.036484: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00851a00 next 23 of size 8192
2019-11-19 13:01:15.579087: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00853a00 next 24 of size 8667136
2019-11-19 13:01:15.579175: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01097a00 next 25 of size 256
2019-11-19 13:01:15.579224: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01097b00 next 26 of size 73728
2019-11-19 13:01:15.579271: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf010a9b00 next 27 of size 3328
2019-11-19 13:01:15.579368: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf010aa800 next 28 of size 256
2019-11-19 13:01:15.579425: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf010aa900 next 29 of size 256
2019-11-19 13:01:15.579469: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf010aaa00 next 30 of size 8192
2019-11-19 13:01:15.579511: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf010aca00 next 31 of size 256
2019-11-19 13:01:15.579557: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf010acb00 next 32 of size 8667136
2019-11-19 13:01:15.579601: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf018f0b00 next 33 of size 73728
2019-11-19 13:01:15.579645: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01902b00 next 34 of size 3328
2019-11-19 13:01:15.579692: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01903800 next 35 of size 256
2019-11-19 13:01:15.579863: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01903900 next 36 of size 256
2019-11-19 13:01:15.579925: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01903a00 next 37 of size 256
2019-11-19 13:01:15.579968: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01903b00 next 38 of size 256
2019-11-19 13:01:15.580010: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01903c00 next 39 of size 256
2019-11-19 13:01:15.580057: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01903d00 next 40 of size 256
2019-11-19 13:01:15.580102: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01903e00 next 41 of size 256
2019-11-19 13:01:15.580146: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01903f00 next 42 of size 256
2019-11-19 13:01:15.580189: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01904000 next 43 of size 256
2019-11-19 13:01:15.580232: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01904100 next 44 of size 256
2019-11-19 13:01:15.580273: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01904200 next 45 of size 256
2019-11-19 13:01:15.584984: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01904300 next 46 of size 256
2019-11-19 13:01:15.585072: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01904400 next 47 of size 256
2019-11-19 13:01:15.585122: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01904500 next 48 of size 256
2019-11-19 13:01:15.585169: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01904600 next 49 of size 1280000
2019-11-19 13:01:15.585215: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3ce00 next 50 of size 256
2019-11-19 13:01:15.585256: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3cf00 next 51 of size 256
2019-11-19 13:01:15.586084: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3d000 next 52 of size 256
2019-11-19 13:01:15.591889: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3d100 next 53 of size 256
2019-11-19 13:01:15.591953: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3d200 next 54 of size 256
2019-11-19 13:01:15.591985: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3d300 next 55 of size 256
2019-11-19 13:01:15.996727: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3d400 next 56 of size 256
2019-11-19 13:01:15.996824: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3d500 next 57 of size 256
2019-11-19 13:01:15.996873: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3d600 next 58 of size 256
2019-11-19 13:01:15.996914: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3d700 next 59 of size 256
2019-11-19 13:01:15.996952: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3d800 next 60 of size 256
2019-11-19 13:01:15.996989: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3d900 next 61 of size 8192
2019-11-19 13:01:15.997021: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3f900 next 62 of size 256
2019-11-19 13:01:15.997055: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a3fa00 next 63 of size 3328
2019-11-19 13:01:15.997089: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a40700 next 64 of size 256
2019-11-19 13:01:15.997122: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a40800 next 65 of size 3328
2019-11-19 13:01:15.997151: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a41500 next 66 of size 256
2019-11-19 13:01:15.997179: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a41600 next 67 of size 256
2019-11-19 13:01:15.997207: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a41700 next 68 of size 256
2019-11-19 13:01:15.997236: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01a41800 next 69 of size 8667136
2019-11-19 13:01:15.997265: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02285800 next 70 of size 256
2019-11-19 13:01:15.997292: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02285900 next 71 of size 256
2019-11-19 13:01:15.997321: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02285a00 next 72 of size 8192
2019-11-19 13:01:15.997348: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02287a00 next 73 of size 256
2019-11-19 13:01:15.997375: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02287b00 next 74 of size 256
2019-11-19 13:01:15.997402: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02287c00 next 75 of size 256
2019-11-19 13:01:15.997430: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02287d00 next 76 of size 256
2019-11-19 13:01:15.997457: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02287e00 next 77 of size 256
2019-11-19 13:01:15.997485: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02287f00 next 78 of size 256
2019-11-19 13:01:15.997512: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02288000 next 79 of size 256
2019-11-19 13:01:15.997541: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02288100 next 80 of size 256
2019-11-19 13:01:15.997568: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02288200 next 81 of size 256
2019-11-19 13:01:15.997596: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02288300 next 82 of size 256
2019-11-19 13:01:15.997626: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02288400 next 83 of size 256
2019-11-19 13:01:15.997655: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02288500 next 84 of size 73728
2019-11-19 13:01:15.997685: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf0229a500 next 18446744073709551615 of size 60934912
2019-11-19 13:01:15.997713: I tensorflow/core/common_runtime/bfc_allocator.cc:809] Summary of in-use Chunks by size:
2019-11-19 13:01:15.997756: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 68 Chunks of size 256 totalling 17.0KiB
2019-11-19 13:01:15.997793: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 1280 totalling 1.2KiB
2019-11-19 13:01:15.997824: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 3328 totalling 13.0KiB
2019-11-19 13:01:15.997856: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 8192 totalling 32.0KiB
2019-11-19 13:01:16.034331: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 3 Chunks of size 73728 totalling 216.0KiB
2019-11-19 13:01:16.034400: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 1280000 totalling 1.22MiB
2019-11-19 13:01:16.034436: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 3 Chunks of size 8667136 totalling 24.80MiB
2019-11-19 13:01:16.034469: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 60934912 totalling 58.11MiB
2019-11-19 13:01:16.034501: I tensorflow/core/common_runtime/bfc_allocator.cc:816] Sum Total of in-use chunks: 84.40MiB
2019-11-19 13:01:16.034532: I tensorflow/core/common_runtime/bfc_allocator.cc:818] total_region_allocated_bytes_: 88502272 memory_limit_: 88502272 available bytes: 0 curr_region_allocation_bytes_: 177004544
2019-11-19 13:01:16.034569: I tensorflow/core/common_runtime/bfc_allocator.cc:824] Stats:
Limit: 88502272
InUse: 88502272
MaxInUse: 88502272
NumAllocs: 85
MaxAllocSize: 60934912
2019-11-19 13:01:16.034626: W tensorflow/core/common_runtime/bfc_allocator.cc:319] **************************************************************************xxxxxxxxxxxxxxxxxxxxxxxxxx
2019-11-19 13:01:16.034759: W tensorflow/core/framework/op_kernel.cc:1502] OP_REQUIRES failed at conv_ops.cc:881 : Resource exhausted: OOM when allocating tensor with shape[32,1,5,5] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
HISTORY ERROR*******
Traceback (most recent call last):
File “/home/victor/.vscode-oss/extensions/ms-python.python-2019.3.6558/pythonFiles/ptvsd_launcher.py”, line 45, in
main(ptvsdArgs)
File “/home/victor/.vscode-oss/extensions/ms-python.python-2019.3.6558/pythonFiles/lib/python/ptvsd/main.py”, line 391, in main
run()
File “/home/victor/.vscode-oss/extensions/ms-python.python-2019.3.6558/pythonFiles/lib/python/ptvsd/main.py”, line 272, in run_file
runpy.run_path(target, run_name=‘main’)
File “/usr/lib/python3.6/runpy.py”, line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File “/usr/lib/python3.6/runpy.py”, line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File “/usr/lib/python3.6/runpy.py”, line 85, in _run_code
exec(code, run_globals)
File “/home/victor/Desktop/Jetson_Nano/Version_final_V0.6/Main.py”, line 97, in
history, trainTime = cnn.trainModel(model)
File “/home/victor/Desktop/Jetson_Nano/Version_final_V0.6/NeuralNetwork.py”, line 144, in trainModel
history = model.fit(trainImages,outTrainVector,batch_size=32,epochs=gv.epochs,validation_data=(validationImages, outValidationVector))
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py”, line 780, in fit
steps_name=‘steps_per_epoch’)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training_arrays.py”, line 363, in model_iteration
batch_outs = f(ins_batch)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/backend.py”, line 3292, in call
run_metadata=self.run_metadata)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py”, line 1458, in call
run_metadata_ptr)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) Resource exhausted: OOM when allocating tensor with shape[3,3,32,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node Nadam/Nadam/update_conv2d_1/kernel/mul}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[metrics/mean_absolute_error/Identity/_111]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
(1) Resource exhausted: OOM when allocating tensor with shape[3,3,32,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node Nadam/Nadam/update_conv2d_1/kernel/mul}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
0 successful operations.
0 derived errors ignored.
Terminated