Hy,
I’m trying to run a tutorial (GitHub - theAIGuysCode/Object-Detection-API: Yolov3 Object Detection implemented as APIs, using TensorFlow and Flask) on my jetson Nano (jetpack 4.3) but when I’m ready to run
python app.py
an error occurs (see an extract below).
I have read something about a relation between the error and a random function. But I don’t know what to do with this information. Maybe it does not have anything to do with my problem.
Extract of the output after running python app.py
(You can find the complete output at the bottom.)
Does anyone have an idea what I can do?
Best regards
chris
xxxx*************************_____***************
2020-09-16 09:51:50.107477: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at cwise_ops_common.cc:82 : Resource exhausted: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File “app.py”, line 30, in
yolo = YoloV3(classes=num_classes)
File “/home/christopher/aiguyyolotest1/Object-Detection-API/yolov3_tf2/models.py”, line 217, in YoloV3
output_0 = YoloOutput(512, len(masks[0]), classes, name=‘yolo_output_0’)(x)
File “/home/christopher/aiguyyolotest1/Object-Detection-API/yolov3_tf2/models.py”, line 146, in yolo_output
x = DarknetConv(x, filters * 2, 3)
File “/home/christopher/aiguyyolotest1/Object-Detection-API/yolov3_tf2/models.py”, line 52, in DarknetConv
use_bias=not batch_norm, kernel_regularizer=l2(0.0005))(x)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py”, line 748, in call
self._maybe_build(inputs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py”, line 2116, in _maybe_build
self.build(input_shapes)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/layers/convolutional.py”, line 161, in build
dtype=self.dtype)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py”, line 446, in add_weight
caching_device=caching_device)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/tracking/base.py”, line 744, in _add_variable_with_custom_getter
**kwargs_for_getter)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer_utils.py”, line 142, in make_variable
shape=variable_shape if variable_shape else None)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py”, line 258, in call
return cls._variable_v1_call(*args, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py”, line 219, in _variable_v1_call
shape=shape)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py”, line 197, in
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variable_scope.py”, line 2596, in default_variable_creator
shape=shape)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py”, line 262, in call
return super(VariableMetaclass, cls).call(*args, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py”, line 1411, in init
distribute_strategy=distribute_strategy)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py”, line 1542, in _init_from_args
initial_value() if init_from_fn else initial_value,
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer_utils.py”, line 122, in
init_val = lambda: initializer(shape, dtype=dtype)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops_v2.py”, line 425, in call
return self._random_generator.random_uniform(shape, -limit, limit, dtype)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops_v2.py”, line 788, in random_uniform
shape=shape, minval=minval, maxval=maxval, dtype=dtype, seed=self.seed)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/random_ops.py”, line 273, in random_uniform
result = math_ops.add(rnd * (maxval - minval), minval, name=name)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_math_ops.py”, line 343, in add
_ops.raise_from_not_ok_status(e, name)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py”, line 6606, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File “”, line 3, in raise_from
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Add] name: conv2d_57/kernel/Initializer/random_uniform/
Complete output
(aiguyyolotest1) christopher@ccz:~/aiguyyolotest1/Object-Detection-API$ python app.py
2020-09-16 09:50:52.475423: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-16 09:50:55.385885: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.6
2020-09-16 09:50:55.387874: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.6
2020-09-16 09:51:02.271231: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-09-16 09:51:02.311970: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-16 09:51:02.312145: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 23.84GiB/s
2020-09-16 09:51:02.312289: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-16 09:51:02.312452: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-09-16 09:51:02.394153: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-09-16 09:51:02.509485: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-09-16 09:51:02.621084: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-09-16 09:51:02.685989: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-09-16 09:51:02.686363: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-09-16 09:51:02.686911: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-16 09:51:02.687411: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-16 09:51:02.687513: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-09-16 09:51:02.802625: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2020-09-16 09:51:02.803601: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x316eaee0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-09-16 09:51:02.803663: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-09-16 09:51:02.905778: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-16 09:51:02.906103: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x306105f0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-09-16 09:51:02.906164: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
2020-09-16 09:51:02.906975: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-16 09:51:02.907127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 23.84GiB/s
2020-09-16 09:51:02.907328: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-16 09:51:02.907445: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-09-16 09:51:02.907612: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-09-16 09:51:02.907756: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-09-16 09:51:02.907882: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-09-16 09:51:02.908010: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-09-16 09:51:02.908104: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-09-16 09:51:02.908530: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-16 09:51:02.908987: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-16 09:51:02.909086: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-09-16 09:51:02.909272: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-16 09:51:13.777986: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-09-16 09:51:13.778172: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0
2020-09-16 09:51:13.778208: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N
2020-09-16 09:51:13.812381: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-16 09:51:13.812952: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-16 09:51:13.828170: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 275 MB memory) → physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2020-09-16 09:51:50.018283: W tensorflow/core/common_runtime/bfc_allocator.cc:424] Allocator (GPU_0_bfc) ran out of memory trying to allocate 18.00MiB (rounded to 18874368). Current allocation summary follows.
2020-09-16 09:51:50.018457: I tensorflow/core/common_runtime/bfc_allocator.cc:894] BFCAllocator dump for GPU_0_bfc
2020-09-16 09:51:50.018526: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (256): Total Chunks: 25, Chunks in use: 25. 6.2KiB allocated for chunks. 6.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2020-09-16 09:51:50.018611: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (512): Total Chunks: 44, Chunks in use: 44. 22.0KiB allocated for chunks. 22.0KiB in use in bin. 22.0KiB client-requested in use in bin.
2020-09-16 09:51:50.018730: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (1024): Total Chunks: 69, Chunks in use: 69. 69.2KiB allocated for chunks. 69.2KiB in use in bin. 69.0KiB client-requested in use in bin.
2020-09-16 09:51:50.018835: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (2048): Total Chunks: 65, Chunks in use: 65. 133.0KiB allocated for chunks. 133.0KiB in use in bin. 131.4KiB client-requested in use in bin.
2020-09-16 09:51:50.019167: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (4096): Total Chunks: 28, Chunks in use: 28. 114.0KiB allocated for chunks. 114.0KiB in use in bin. 112.0KiB client-requested in use in bin.
2020-09-16 09:51:50.019651: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (8192): Total Chunks: 1, Chunks in use: 1. 8.0KiB allocated for chunks. 8.0KiB in use in bin. 8.0KiB client-requested in use in bin.
2020-09-16 09:51:50.020119: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-16 09:51:50.020370: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (32768): Total Chunks: 2, Chunks in use: 2. 64.0KiB allocated for chunks. 64.0KiB in use in bin. 64.0KiB client-requested in use in bin.
2020-09-16 09:51:50.020676: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (65536): Total Chunks: 3, Chunks in use: 2. 223.5KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2020-09-16 09:51:50.021014: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (131072): Total Chunks: 8, Chunks in use: 8. 1.00MiB allocated for chunks. 1.00MiB in use in bin. 1.00MiB client-requested in use in bin.
2020-09-16 09:51:50.021218: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (262144): Total Chunks: 3, Chunks in use: 3. 864.0KiB allocated for chunks. 864.0KiB in use in bin. 864.0KiB client-requested in use in bin.
2020-09-16 09:51:50.021436: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (524288): Total Chunks: 10, Chunks in use: 8. 5.28MiB allocated for chunks. 4.28MiB in use in bin. 4.00MiB client-requested in use in bin.
2020-09-16 09:51:50.021801: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (1048576): Total Chunks: 9, Chunks in use: 9. 10.88MiB allocated for chunks. 10.88MiB in use in bin. 10.12MiB client-requested in use in bin.
2020-09-16 09:51:50.022083: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (2097152): Total Chunks: 9, Chunks in use: 7. 19.50MiB allocated for chunks. 15.00MiB in use in bin. 14.00MiB client-requested in use in bin.
2020-09-16 09:51:50.022247: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (4194304): Total Chunks: 9, Chunks in use: 9. 42.38MiB allocated for chunks. 42.38MiB in use in bin. 40.50MiB client-requested in use in bin.
2020-09-16 09:51:50.022412: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-16 09:51:50.022600: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (16777216): Total Chunks: 10, Chunks in use: 9. 194.75MiB allocated for chunks. 178.75MiB in use in bin. 162.00MiB client-requested in use in bin.
2020-09-16 09:51:50.022855: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-16 09:51:50.023353: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-16 09:51:50.023601: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-16 09:51:50.023997: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-16 09:51:50.024224: I tensorflow/core/common_runtime/bfc_allocator.cc:917] Bin for 18.00MiB was 16.00MiB, Chunk State:
2020-09-16 09:51:50.024420: I tensorflow/core/common_runtime/bfc_allocator.cc:923] Size: 16.00MiB | Requested Size: 4B | in_use: 0 | bin_num: 16, prev: Size: 2.00MiB | Requested Size: 2.00MiB | in_use: 1 | bin_num: -1, next: Size: 20.00MiB | Requested Size: 18.00MiB | in_use: 1 | bin_num: -1
2020-09-16 09:51:50.024713: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 1048576
2020-09-16 09:51:50.024879: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0000 of size 1280 next 1
2020-09-16 09:51:50.025027: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0500 of size 256 next 2
2020-09-16 09:51:50.025380: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0600 of size 256 next 5
2020-09-16 09:51:50.025526: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0700 of size 256 next 4
2020-09-16 09:51:50.025928: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0800 of size 256 next 7
2020-09-16 09:51:50.026086: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0900 of size 256 next 8
2020-09-16 09:51:50.026240: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0a00 of size 256 next 9
2020-09-16 09:51:50.026440: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0b00 of size 256 next 11
2020-09-16 09:51:50.026616: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0c00 of size 256 next 13
2020-09-16 09:51:50.026909: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0d00 of size 256 next 14
2020-09-16 09:51:50.027054: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0e00 of size 256 next 15
2020-09-16 09:51:50.027190: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0f00 of size 256 next 17
2020-09-16 09:51:50.027493: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1000 of size 256 next 19
2020-09-16 09:51:50.027581: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1100 of size 256 next 20
2020-09-16 09:51:50.027659: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1200 of size 256 next 22
2020-09-16 09:51:50.027746: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1300 of size 256 next 24
2020-09-16 09:51:50.027840: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1400 of size 256 next 25
2020-09-16 09:51:50.028016: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1500 of size 256 next 26
2020-09-16 09:51:50.028126: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1600 of size 512 next 27
2020-09-16 09:51:50.028231: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1800 of size 512 next 30
2020-09-16 09:51:50.028352: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1a00 of size 512 next 31
2020-09-16 09:51:50.028456: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1c00 of size 512 next 32
2020-09-16 09:51:50.028572: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1e00 of size 256 next 33
2020-09-16 09:51:50.028682: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1f00 of size 256 next 35
2020-09-16 09:51:50.028794: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f2000 of size 256 next 36
2020-09-16 09:51:50.028905: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f2100 of size 256 next 37
2020-09-16 09:51:50.029125: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f2200 of size 512 next 3
2020-09-16 09:51:50.029246: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f2400 of size 3584 next 6
2020-09-16 09:51:50.029358: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3200 of size 512 next 38
2020-09-16 09:51:50.029483: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3400 of size 512 next 40
2020-09-16 09:51:50.029559: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3600 of size 512 next 41
2020-09-16 09:51:50.029627: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3800 of size 256 next 42
2020-09-16 09:51:50.029719: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3900 of size 256 next 44
2020-09-16 09:51:50.029782: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3a00 of size 256 next 45
2020-09-16 09:51:50.029842: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3b00 of size 256 next 46
2020-09-16 09:51:50.029898: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3c00 of size 512 next 47
2020-09-16 09:51:50.029954: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3e00 of size 512 next 49
2020-09-16 09:51:50.030009: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4000 of size 512 next 50
2020-09-16 09:51:50.030190: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4200 of size 512 next 51
2020-09-16 09:51:50.030260: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4400 of size 1024 next 53
2020-09-16 09:51:50.030329: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4800 of size 1024 next 56
2020-09-16 09:51:50.030399: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4c00 of size 1024 next 57
2020-09-16 09:51:50.030471: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5000 of size 1024 next 58
2020-09-16 09:51:50.030546: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5400 of size 512 next 59
2020-09-16 09:51:50.030621: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5600 of size 512 next 61
2020-09-16 09:51:50.030697: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5800 of size 512 next 63
2020-09-16 09:51:50.030773: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5a00 of size 512 next 64
2020-09-16 09:51:50.030849: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5c00 of size 1024 next 65
2020-09-16 09:51:50.030925: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6000 of size 1024 next 66
2020-09-16 09:51:50.031001: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6400 of size 1024 next 67
2020-09-16 09:51:50.031077: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6800 of size 1024 next 68
2020-09-16 09:51:50.031153: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6c00 of size 512 next 69
2020-09-16 09:51:50.031229: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6e00 of size 512 next 71
2020-09-16 09:51:50.031393: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f7000 of size 512 next 18
2020-09-16 09:51:50.031477: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f7200 of size 8192 next 16
2020-09-16 09:51:50.031556: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f9200 of size 32768 next 34
2020-09-16 09:51:50.031654: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00901200 of size 32768 next 43
2020-09-16 09:51:50.031729: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00909200 of size 512 next 72
2020-09-16 09:51:50.031803: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00909400 of size 1024 next 75
2020-09-16 09:51:50.031877: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00909800 of size 1024 next 76
2020-09-16 09:51:50.031951: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00909c00 of size 1024 next 77
2020-09-16 09:51:50.032097: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090a000 of size 1024 next 78
2020-09-16 09:51:50.032173: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090a400 of size 512 next 79
2020-09-16 09:51:50.032248: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090a600 of size 512 next 80
2020-09-16 09:51:50.032321: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090a800 of size 512 next 82
2020-09-16 09:51:50.032395: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090aa00 of size 512 next 83
2020-09-16 09:51:50.032469: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090ac00 of size 1024 next 85
2020-09-16 09:51:50.032550: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090b000 of size 1024 next 86
2020-09-16 09:51:50.032711: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090b400 of size 1024 next 87
2020-09-16 09:51:50.032770: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090b800 of size 1024 next 88
2020-09-16 09:51:50.032819: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090bc00 of size 512 next 90
2020-09-16 09:51:50.032865: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090be00 of size 512 next 91
2020-09-16 09:51:50.032910: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090c000 of size 512 next 92
2020-09-16 09:51:50.032952: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090c200 of size 512 next 93
2020-09-16 09:51:50.032995: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090c400 of size 1024 next 95
2020-09-16 09:51:50.033040: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090c800 of size 1024 next 97
2020-09-16 09:51:50.033086: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090cc00 of size 1024 next 98
2020-09-16 09:51:50.033135: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090d000 of size 1024 next 99
2020-09-16 09:51:50.033188: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090d400 of size 512 next 101
2020-09-16 09:51:50.033242: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090d600 of size 512 next 102
2020-09-16 09:51:50.033363: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090d800 of size 512 next 103
2020-09-16 09:51:50.033423: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090da00 of size 512 next 104
2020-09-16 09:51:50.033484: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090dc00 of size 1024 next 106
2020-09-16 09:51:50.033545: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090e000 of size 1024 next 107
2020-09-16 09:51:50.033608: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090e400 of size 1024 next 108
2020-09-16 09:51:50.033697: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090e800 of size 1024 next 109
2020-09-16 09:51:50.033768: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090ec00 of size 512 next 111
2020-09-16 09:51:50.033832: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090ee00 of size 512 next 112
2020-09-16 09:51:50.033962: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090f000 of size 512 next 113
2020-09-16 09:51:50.034029: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090f200 of size 512 next 114
2020-09-16 09:51:50.034094: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090f400 of size 1024 next 116
2020-09-16 09:51:50.034158: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090f800 of size 1024 next 117
2020-09-16 09:51:50.034222: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090fc00 of size 1024 next 118
2020-09-16 09:51:50.034285: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910000 of size 1024 next 119
2020-09-16 09:51:50.034349: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910400 of size 512 next 121
2020-09-16 09:51:50.034413: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910600 of size 512 next 122
2020-09-16 09:51:50.034476: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910800 of size 512 next 123
2020-09-16 09:51:50.034615: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910a00 of size 512 next 124
2020-09-16 09:51:50.034683: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910c00 of size 1024 next 125
2020-09-16 09:51:50.034747: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911000 of size 1024 next 126
2020-09-16 09:51:50.034810: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911400 of size 1024 next 127
2020-09-16 09:51:50.034883: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911800 of size 1024 next 128
2020-09-16 09:51:50.034930: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911c00 of size 512 next 130
2020-09-16 09:51:50.034970: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911e00 of size 512 next 131
2020-09-16 09:51:50.035009: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912000 of size 512 next 132
2020-09-16 09:51:50.035049: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912200 of size 512 next 133
2020-09-16 09:51:50.035090: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912400 of size 1024 next 136
2020-09-16 09:51:50.035130: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912800 of size 1024 next 137
2020-09-16 09:51:50.035170: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912c00 of size 1024 next 138
2020-09-16 09:51:50.035285: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00913000 of size 1024 next 139
2020-09-16 09:51:50.035333: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00913400 of size 2048 next 140
2020-09-16 09:51:50.035382: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00913c00 of size 2048 next 141
2020-09-16 09:51:50.035432: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00914400 of size 2048 next 142
2020-09-16 09:51:50.035483: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00914c00 of size 2048 next 144
2020-09-16 09:51:50.035535: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00915400 of size 1024 next 145
2020-09-16 09:51:50.035588: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00915800 of size 1024 next 146
2020-09-16 09:51:50.035642: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00915c00 of size 1024 next 148
2020-09-16 09:51:50.035695: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00916000 of size 1024 next 149
2020-09-16 09:51:50.035753: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00916400 of size 3584 next 12
2020-09-16 09:51:50.035811: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00917200 of size 73728 next 10
2020-09-16 09:51:50.035872: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00929200 of size 2048 next 150
2020-09-16 09:51:50.036002: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00929a00 of size 2048 next 153
2020-09-16 09:51:50.036047: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092a200 of size 2048 next 154
2020-09-16 09:51:50.036084: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092aa00 of size 1024 next 156
2020-09-16 09:51:50.036121: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092ae00 of size 1024 next 157
2020-09-16 09:51:50.036157: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092b200 of size 1024 next 158
2020-09-16 09:51:50.036195: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092b600 of size 1024 next 159
2020-09-16 09:51:50.036229: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092ba00 of size 2048 next 160
2020-09-16 09:51:50.036262: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092c200 of size 2048 next 162
2020-09-16 09:51:50.036297: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092ca00 of size 2048 next 163
2020-09-16 09:51:50.036334: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092d200 of size 2048 next 164
2020-09-16 09:51:50.036372: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092da00 of size 1024 next 165
2020-09-16 09:51:50.036411: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092de00 of size 1024 next 166
2020-09-16 09:51:50.036452: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092e200 of size 1024 next 167
2020-09-16 09:51:50.036495: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092e600 of size 1024 next 168
2020-09-16 09:51:50.036539: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092ea00 of size 2048 next 169
2020-09-16 09:51:50.036584: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092f200 of size 2048 next 171
2020-09-16 09:51:50.036630: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092fa00 of size 2048 next 172
2020-09-16 09:51:50.036677: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00930200 of size 2048 next 173
2020-09-16 09:51:50.036724: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00930a00 of size 1024 next 175
2020-09-16 09:51:50.036772: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00930e00 of size 1024 next 176
2020-09-16 09:51:50.036905: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00931200 of size 1024 next 177
2020-09-16 09:51:50.036964: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00931600 of size 1024 next 178
2020-09-16 09:51:50.037016: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00931a00 of size 2048 next 180
2020-09-16 09:51:50.037073: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00932200 of size 2048 next 182
2020-09-16 09:51:50.037110: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00932a00 of size 2048 next 183
2020-09-16 09:51:50.037136: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00933200 of size 2048 next 184
2020-09-16 09:51:50.037161: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00933a00 of size 1024 next 186
2020-09-16 09:51:50.037184: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00933e00 of size 1024 next 187
2020-09-16 09:51:50.037207: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00934200 of size 1024 next 188
2020-09-16 09:51:50.037230: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00934600 of size 1024 next 189
2020-09-16 09:51:50.037253: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00934a00 of size 2048 next 190
2020-09-16 09:51:50.037275: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00935200 of size 2048 next 191
2020-09-16 09:51:50.037297: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00935a00 of size 2048 next 193
2020-09-16 09:51:50.037318: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00936200 of size 2048 next 194
2020-09-16 09:51:50.037340: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00936a00 of size 1024 next 196
2020-09-16 09:51:50.037361: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00936e00 of size 1024 next 197
2020-09-16 09:51:50.037383: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00937200 of size 1024 next 198
2020-09-16 09:51:50.037406: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00937600 of size 1024 next 199
2020-09-16 09:51:50.037482: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00937a00 of size 2048 next 202
2020-09-16 09:51:50.037509: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00938200 of size 2048 next 203
2020-09-16 09:51:50.037535: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00938a00 of size 2048 next 204
2020-09-16 09:51:50.037563: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00939200 of size 2048 next 205
2020-09-16 09:51:50.037590: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00939a00 of size 1024 next 207
2020-09-16 09:51:50.037618: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00939e00 of size 1024 next 208
2020-09-16 09:51:50.037646: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0093a200 of size 1024 next 209
2020-09-16 09:51:50.037693: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0093a600 of size 1024 next 210
2020-09-16 09:51:50.037729: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0093aa00 of size 2048 next 23
2020-09-16 09:51:50.037757: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0093b200 of size 73728 next 21
2020-09-16 09:51:50.037782: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094d200 of size 2048 next 212
2020-09-16 09:51:50.037807: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094da00 of size 2048 next 213
2020-09-16 09:51:50.037831: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094e200 of size 2048 next 214
2020-09-16 09:51:50.037854: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094ea00 of size 1024 next 216
2020-09-16 09:51:50.037878: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094ee00 of size 1024 next 217
2020-09-16 09:51:50.037903: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094f200 of size 1024 next 218
2020-09-16 09:51:50.037927: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094f600 of size 1024 next 219
2020-09-16 09:51:50.037951: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094fa00 of size 2048 next 221
2020-09-16 09:51:50.037976: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00950200 of size 2048 next 222
2020-09-16 09:51:50.045954: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00950a00 of size 2048 next 223
2020-09-16 09:51:50.046025: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00951200 of size 2048 next 224
2020-09-16 09:51:50.046055: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00951a00 of size 4096 next 225
2020-09-16 09:51:50.046085: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00952a00 of size 4096 next 228
2020-09-16 09:51:50.046112: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00953a00 of size 4096 next 229
2020-09-16 09:51:50.046140: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00954a00 of size 4096 next 230
2020-09-16 09:51:50.046169: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00955a00 of size 2048 next 231
2020-09-16 09:51:50.046198: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00956200 of size 2048 next 232
2020-09-16 09:51:50.046229: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00956a00 of size 2048 next 234
2020-09-16 09:51:50.046259: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00957200 of size 2048 next 235
2020-09-16 09:51:50.046298: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00957a00 of size 4096 next 236
2020-09-16 09:51:50.046330: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00958a00 of size 4096 next 238
2020-09-16 09:51:50.046356: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00959a00 of size 4096 next 239
2020-09-16 09:51:50.046383: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095aa00 of size 4096 next 240
2020-09-16 09:51:50.046419: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095ba00 of size 2048 next 242
2020-09-16 09:51:50.046450: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095c200 of size 2048 next 243
2020-09-16 09:51:50.046478: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095ca00 of size 2048 next 244
2020-09-16 09:51:50.046524: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095d200 of size 2048 next 245
2020-09-16 09:51:50.046559: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095da00 of size 4096 next 246
2020-09-16 09:51:50.046587: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095ea00 of size 4096 next 248
2020-09-16 09:51:50.046614: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095fa00 of size 4096 next 249
2020-09-16 09:51:50.046648: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00960a00 of size 4096 next 250
2020-09-16 09:51:50.046676: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00961a00 of size 2048 next 252
2020-09-16 09:51:50.046704: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00962200 of size 2048 next 253
2020-09-16 09:51:50.046738: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00962a00 of size 2048 next 254
2020-09-16 09:51:50.046766: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00963200 of size 2048 next 255
2020-09-16 09:51:50.046794: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00963a00 of size 4096 next 256
2020-09-16 09:51:50.046828: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00964a00 of size 4096 next 258
2020-09-16 09:51:50.046863: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00965a00 of size 4096 next 259
2020-09-16 09:51:50.046892: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00966a00 of size 4096 next 260
2020-09-16 09:51:50.046918: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00967a00 of size 2048 next 262
2020-09-16 09:51:50.046951: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00968200 of size 2048 next 263
2020-09-16 09:51:50.046986: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00968a00 of size 2048 next 264
2020-09-16 09:51:50.047020: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00969200 of size 2048 next 265
2020-09-16 09:51:50.047054: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00969a00 of size 4096 next 266
2020-09-16 09:51:50.047088: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0096aa00 of size 4096 next 268
2020-09-16 09:51:50.047123: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0096ba00 of size 6144 next 81
2020-09-16 09:51:50.047159: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0096d200 of size 131072 next 62
2020-09-16 09:51:50.047194: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0098d200 of size 131072 next 60
2020-09-16 09:51:50.047229: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009ad200 of size 131072 next 70
2020-09-16 09:51:50.047264: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009cd200 of size 4096 next 269
2020-09-16 09:51:50.047298: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009ce200 of size 2048 next 270
2020-09-16 09:51:50.047331: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009cea00 of size 2048 next 271
2020-09-16 09:51:50.047365: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009cf200 of size 2048 next 272
2020-09-16 09:51:50.047399: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009cfa00 of size 2048 next 273
2020-09-16 09:51:50.047433: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d0200 of size 4096 next 274
2020-09-16 09:51:50.047467: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d1200 of size 4096 next 276
2020-09-16 09:51:50.047503: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d2200 of size 4096 next 277
2020-09-16 09:51:50.047538: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d3200 of size 4096 next 278
2020-09-16 09:51:50.047572: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d4200 of size 2048 next 280
2020-09-16 09:51:50.047606: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d4a00 of size 2048 next 281
2020-09-16 09:51:50.047640: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d5200 of size 2048 next 282
2020-09-16 09:51:50.047674: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d5a00 of size 2048 next 283
2020-09-16 09:51:50.047708: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d6200 of size 4096 next 284
2020-09-16 09:51:50.047741: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d7200 of size 4096 next 285
2020-09-16 09:51:50.047775: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d8200 of size 4096 next 287
2020-09-16 09:51:50.047809: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d9200 of size 4096 next 288
2020-09-16 09:51:50.047843: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009da200 of size 2048 next 290
2020-09-16 09:51:50.047876: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009daa00 of size 2048 next 291
2020-09-16 09:51:50.047910: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009db200 of size 2048 next 292
2020-09-16 09:51:50.047944: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009dba00 of size 2048 next 293
2020-09-16 09:51:50.047978: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f009dc200 of size 81408 next 18446744073709551615
2020-09-16 09:51:50.048013: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 2097152
2020-09-16 09:51:50.048048: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009f0000 of size 294912 next 29
2020-09-16 09:51:50.048084: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00a38000 of size 294912 next 39
2020-09-16 09:51:50.048118: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00a80000 of size 294912 next 48
2020-09-16 09:51:50.048152: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00ac8000 of size 524288 next 155
2020-09-16 09:51:50.048188: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00b48000 of size 688128 next 18446744073709551615
2020-09-16 09:51:50.048222: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 4194304
2020-09-16 09:51:50.048256: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00bf0000 of size 131072 next 89
2020-09-16 09:51:50.048291: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00c10000 of size 131072 next 100
2020-09-16 09:51:50.048325: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00c30000 of size 131072 next 110
2020-09-16 09:51:50.048360: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00c50000 of size 131072 next 120
2020-09-16 09:51:50.048394: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00c70000 of size 131072 next 129
2020-09-16 09:51:50.048428: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f00c90000 of size 524288 next 55
2020-09-16 09:51:50.048463: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00d10000 of size 1179648 next 52
2020-09-16 09:51:50.048499: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00e30000 of size 1835008 next 18446744073709551615
2020-09-16 09:51:50.048533: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 8388608
2020-09-16 09:51:50.048566: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00ff0000 of size 1179648 next 74
2020-09-16 09:51:50.048602: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01110000 of size 1179648 next 84
2020-09-16 09:51:50.048637: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f01230000 of size 524288 next 147
2020-09-16 09:51:50.048672: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f012b0000 of size 655360 next 96
2020-09-16 09:51:50.048709: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01350000 of size 1179648 next 94
2020-09-16 09:51:50.048744: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01470000 of size 1179648 next 105
2020-09-16 09:51:50.048779: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01590000 of size 1179648 next 115
2020-09-16 09:51:50.048816: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f016b0000 of size 1310720 next 18446744073709551615
2020-09-16 09:51:50.048849: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 16777216
2020-09-16 09:51:50.048884: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f017f0000 of size 1179648 next 135
2020-09-16 09:51:50.048919: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01910000 of size 524288 next 174
2020-09-16 09:51:50.048954: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01990000 of size 2097152 next 241
2020-09-16 09:51:50.048989: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01b90000 of size 2097152 next 251
2020-09-16 09:51:50.049024: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01d90000 of size 2097152 next 261
2020-09-16 09:51:50.049057: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f01f90000 of size 2621440 next 143
2020-09-16 09:51:50.049092: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f02210000 of size 6160384 next 18446744073709551615
2020-09-16 09:51:50.049125: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 33554432
2020-09-16 09:51:50.049162: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f027f0000 of size 4718592 next 152
2020-09-16 09:51:50.049197: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f02c70000 of size 4718592 next 161
2020-09-16 09:51:50.049231: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f030f0000 of size 4718592 next 170
2020-09-16 09:51:50.049265: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f03570000 of size 524288 next 185
2020-09-16 09:51:50.049299: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f035f0000 of size 524288 next 195
2020-09-16 09:51:50.049334: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f03670000 of size 524288 next 206
2020-09-16 09:51:50.049367: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f036f0000 of size 524288 next 215
2020-09-16 09:51:50.049402: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f03770000 of size 2621440 next 181
2020-09-16 09:51:50.049438: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f039f0000 of size 4718592 next 179
2020-09-16 09:51:50.049472: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f03e70000 of size 2097152 next 233
2020-09-16 09:51:50.049506: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f04070000 of size 2621440 next 192
2020-09-16 09:51:50.049541: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f042f0000 of size 5242880 next 18446744073709551615
2020-09-16 09:51:50.049574: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 67108864
2020-09-16 09:51:50.049609: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f047f0000 of size 4718592 next 201
2020-09-16 09:51:50.049644: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f04c70000 of size 4718592 next 211
2020-09-16 09:51:50.049733: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f050f0000 of size 4718592 next 220
2020-09-16 09:51:50.049773: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f05570000 of size 18874368 next 275
2020-09-16 09:51:50.049809: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f06770000 of size 2097152 next 279
2020-09-16 09:51:50.049844: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f06970000 of size 31981568 next 18446744073709551615
2020-09-16 09:51:50.049879: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 134217728
2020-09-16 09:51:50.049916: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f087f0000 of size 18874368 next 227
2020-09-16 09:51:50.049952: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f099f0000 of size 18874368 next 237
2020-09-16 09:51:50.049977: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0abf0000 of size 18874368 next 247
2020-09-16 09:51:50.050000: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0bdf0000 of size 18874368 next 257
2020-09-16 09:51:50.050023: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0cff0000 of size 18874368 next 267
2020-09-16 09:51:50.050045: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0e1f0000 of size 2097152 next 289
2020-09-16 09:51:50.050066: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f0e3f0000 of size 16777216 next 286
2020-09-16 09:51:50.050089: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0f3f0000 of size 20971520 next 18446744073709551615
2020-09-16 09:51:50.050110: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 21233664
2020-09-16 09:51:50.050133: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f107f0000 of size 21233664 next 18446744073709551615
2020-09-16 09:51:50.050154: I tensorflow/core/common_runtime/bfc_allocator.cc:955] Summary of in-use Chunks by size:
2020-09-16 09:51:50.050185: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 25 Chunks of size 256 totalling 6.2KiB
2020-09-16 09:51:50.050212: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 44 Chunks of size 512 totalling 22.0KiB
2020-09-16 09:51:50.050237: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 68 Chunks of size 1024 totalling 68.0KiB
2020-09-16 09:51:50.050261: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 1280 totalling 1.2KiB
2020-09-16 09:51:50.050286: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 63 Chunks of size 2048 totalling 126.0KiB
2020-09-16 09:51:50.050309: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 3584 totalling 7.0KiB
2020-09-16 09:51:50.050333: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 27 Chunks of size 4096 totalling 108.0KiB
2020-09-16 09:51:50.050357: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 6144 totalling 6.0KiB
2020-09-16 09:51:50.050382: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 8192 totalling 8.0KiB
2020-09-16 09:51:50.050406: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 32768 totalling 64.0KiB
2020-09-16 09:51:50.050432: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 73728 totalling 144.0KiB
2020-09-16 09:51:50.050456: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 8 Chunks of size 131072 totalling 1.00MiB
2020-09-16 09:51:50.050482: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 3 Chunks of size 294912 totalling 864.0KiB
2020-09-16 09:51:50.050507: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 6 Chunks of size 524288 totalling 3.00MiB
2020-09-16 09:51:50.050533: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 655360 totalling 640.0KiB
2020-09-16 09:51:50.050560: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 688128 totalling 672.0KiB
2020-09-16 09:51:50.050586: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 7 Chunks of size 1179648 totalling 7.88MiB
2020-09-16 09:51:50.050611: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 1310720 totalling 1.25MiB
2020-09-16 09:51:50.050642: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 1835008 totalling 1.75MiB
2020-09-16 09:51:50.050669: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 5 Chunks of size 2097152 totalling 10.00MiB
2020-09-16 09:51:50.050699: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 2621440 totalling 5.00MiB
2020-09-16 09:51:50.050721: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 7 Chunks of size 4718592 totalling 31.50MiB
2020-09-16 09:51:50.050746: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 5242880 totalling 5.00MiB
2020-09-16 09:51:50.050773: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 6160384 totalling 5.88MiB
2020-09-16 09:51:50.052009: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 6 Chunks of size 18874368 totalling 108.00MiB
2020-09-16 09:51:50.052068: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 20971520 totalling 20.00MiB
2020-09-16 09:51:50.052106: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 21233664 totalling 20.25MiB
2020-09-16 09:51:50.052146: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 31981568 totalling 30.50MiB
2020-09-16 09:51:50.052182: I tensorflow/core/common_runtime/bfc_allocator.cc:962] Sum Total of in-use chunks: 253.67MiB
2020-09-16 09:51:50.052215: I tensorflow/core/common_runtime/bfc_allocator.cc:964] total_region_allocated_bytes_: 288620544 memory_limit_: 288620544 available bytes: 0 curr_region_allocation_bytes_: 536870912
2020-09-16 09:51:50.071114: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Stats:
Limit: 288620544
InUse: 265994752
MaxInUse: 265994752
NumAllocs: 1432
MaxAllocSize: 340787202020-09-16 09:51:50.071237: W tensorflow/core/common_runtime/bfc_allocator.cc:429] xxxx_____*******
2020-09-16 09:51:50.107477: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at cwise_ops_common.cc:82 : Resource exhausted: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File “app.py”, line 30, in
yolo = YoloV3(classes=num_classes)
File “/home/christopher/aiguyyolotest1/Object-Detection-API/yolov3_tf2/models.py”, line 217, in YoloV3
output_0 = YoloOutput(512, len(masks[0]), classes, name=‘yolo_output_0’)(x)
File “/home/christopher/aiguyyolotest1/Object-Detection-API/yolov3_tf2/models.py”, line 146, in yolo_output
x = DarknetConv(x, filters * 2, 3)
File “/home/christopher/aiguyyolotest1/Object-Detection-API/yolov3_tf2/models.py”, line 52, in DarknetConv
use_bias=not batch_norm, kernel_regularizer=l2(0.0005))(x)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py”, line 748, in call
self._maybe_build(inputs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py”, line 2116, in _maybe_build
self.build(input_shapes)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/layers/convolutional.py”, line 161, in build
dtype=self.dtype)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py”, line 446, in add_weight
caching_device=caching_device)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/tracking/base.py”, line 744, in _add_variable_with_custom_getter
**kwargs_for_getter)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer_utils.py”, line 142, in make_variable
shape=variable_shape if variable_shape else None)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py”, line 258, in call
return cls._variable_v1_call(*args, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py”, line 219, in _variable_v1_call
shape=shape)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py”, line 197, in
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variable_scope.py”, line 2596, in default_variable_creator
shape=shape)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py”, line 262, in call
return super(VariableMetaclass, cls).call(*args, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py”, line 1411, in init
distribute_strategy=distribute_strategy)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py”, line 1542, in _init_from_args
initial_value() if init_from_fn else initial_value,
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer_utils.py”, line 122, in
init_val = lambda: initializer(shape, dtype=dtype)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops_v2.py”, line 425, in call
return self._random_generator.random_uniform(shape, -limit, limit, dtype)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops_v2.py”, line 788, in random_uniform
shape=shape, minval=minval, maxval=maxval, dtype=dtype, seed=self.seed)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/random_ops.py”, line 273, in random_uniform
result = math_ops.add(rnd * (maxval - minval), minval, name=name)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_math_ops.py”, line 343, in add
_ops.raise_from_not_ok_status(e, name)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py”, line 6606, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File “”, line 3, in raise_from
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Add] name: conv2d_57/kernel/Initializer/random_uniform/