Jetson Nano running out of memory: ResourceExhaustedError: OOM when allocating tensor with shape[3,3,512,1024

Hy,

I’m trying to run a tutorial (GitHub - theAIGuysCode/Object-Detection-API: Yolov3 Object Detection implemented as APIs, using TensorFlow and Flask) on my jetson Nano (jetpack 4.3) but when I’m ready to run
python app.py
an error occurs (see an extract below).

I have read something about a relation between the error and a random function. But I don’t know what to do with this information. Maybe it does not have anything to do with my problem.

Extract of the output after running python app.py
(You can find the complete output at the bottom.)

Does anyone have an idea what I can do?

Best regards
chris

xxxx*************************_____***************
2020-09-16 09:51:50.107477: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at cwise_ops_common.cc:82 : Resource exhausted: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File “app.py”, line 30, in
yolo = YoloV3(classes=num_classes)
File “/home/christopher/aiguyyolotest1/Object-Detection-API/yolov3_tf2/models.py”, line 217, in YoloV3
output_0 = YoloOutput(512, len(masks[0]), classes, name=‘yolo_output_0’)(x)
File “/home/christopher/aiguyyolotest1/Object-Detection-API/yolov3_tf2/models.py”, line 146, in yolo_output
x = DarknetConv(x, filters * 2, 3)
File “/home/christopher/aiguyyolotest1/Object-Detection-API/yolov3_tf2/models.py”, line 52, in DarknetConv
use_bias=not batch_norm, kernel_regularizer=l2(0.0005))(x)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py”, line 748, in call
self._maybe_build(inputs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py”, line 2116, in _maybe_build
self.build(input_shapes)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/layers/convolutional.py”, line 161, in build
dtype=self.dtype)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py”, line 446, in add_weight
caching_device=caching_device)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/tracking/base.py”, line 744, in _add_variable_with_custom_getter
**kwargs_for_getter)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer_utils.py”, line 142, in make_variable
shape=variable_shape if variable_shape else None)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py”, line 258, in call
return cls._variable_v1_call(*args, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py”, line 219, in _variable_v1_call
shape=shape)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py”, line 197, in
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variable_scope.py”, line 2596, in default_variable_creator
shape=shape)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py”, line 262, in call
return super(VariableMetaclass, cls).call(*args, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py”, line 1411, in init
distribute_strategy=distribute_strategy)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py”, line 1542, in _init_from_args
initial_value() if init_from_fn else initial_value,
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer_utils.py”, line 122, in
init_val = lambda: initializer(shape, dtype=dtype)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops_v2.py”, line 425, in call
return self._random_generator.random_uniform(shape, -limit, limit, dtype)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops_v2.py”, line 788, in random_uniform
shape=shape, minval=minval, maxval=maxval, dtype=dtype, seed=self.seed)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/random_ops.py”, line 273, in random_uniform
result = math_ops.add(rnd * (maxval - minval), minval, name=name)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_math_ops.py”, line 343, in add
_ops.raise_from_not_ok_status(e, name)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py”, line 6606, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File “”, line 3, in raise_from
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Add] name: conv2d_57/kernel/Initializer/random_uniform/

Complete output

(aiguyyolotest1) christopher@ccz:~/aiguyyolotest1/Object-Detection-API$ python app.py
2020-09-16 09:50:52.475423: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-16 09:50:55.385885: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.6
2020-09-16 09:50:55.387874: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.6
2020-09-16 09:51:02.271231: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-09-16 09:51:02.311970: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-16 09:51:02.312145: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 23.84GiB/s
2020-09-16 09:51:02.312289: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-16 09:51:02.312452: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-09-16 09:51:02.394153: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-09-16 09:51:02.509485: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-09-16 09:51:02.621084: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-09-16 09:51:02.685989: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-09-16 09:51:02.686363: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-09-16 09:51:02.686911: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-16 09:51:02.687411: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-16 09:51:02.687513: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-09-16 09:51:02.802625: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2020-09-16 09:51:02.803601: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x316eaee0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-09-16 09:51:02.803663: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-09-16 09:51:02.905778: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-16 09:51:02.906103: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x306105f0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-09-16 09:51:02.906164: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
2020-09-16 09:51:02.906975: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-16 09:51:02.907127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 23.84GiB/s
2020-09-16 09:51:02.907328: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-16 09:51:02.907445: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-09-16 09:51:02.907612: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-09-16 09:51:02.907756: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-09-16 09:51:02.907882: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-09-16 09:51:02.908010: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-09-16 09:51:02.908104: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-09-16 09:51:02.908530: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-16 09:51:02.908987: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-16 09:51:02.909086: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-09-16 09:51:02.909272: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-16 09:51:13.777986: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-09-16 09:51:13.778172: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0
2020-09-16 09:51:13.778208: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N
2020-09-16 09:51:13.812381: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-16 09:51:13.812952: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-16 09:51:13.828170: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 275 MB memory) → physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2020-09-16 09:51:50.018283: W tensorflow/core/common_runtime/bfc_allocator.cc:424] Allocator (GPU_0_bfc) ran out of memory trying to allocate 18.00MiB (rounded to 18874368). Current allocation summary follows.
2020-09-16 09:51:50.018457: I tensorflow/core/common_runtime/bfc_allocator.cc:894] BFCAllocator dump for GPU_0_bfc
2020-09-16 09:51:50.018526: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (256): Total Chunks: 25, Chunks in use: 25. 6.2KiB allocated for chunks. 6.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2020-09-16 09:51:50.018611: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (512): Total Chunks: 44, Chunks in use: 44. 22.0KiB allocated for chunks. 22.0KiB in use in bin. 22.0KiB client-requested in use in bin.
2020-09-16 09:51:50.018730: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (1024): Total Chunks: 69, Chunks in use: 69. 69.2KiB allocated for chunks. 69.2KiB in use in bin. 69.0KiB client-requested in use in bin.
2020-09-16 09:51:50.018835: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (2048): Total Chunks: 65, Chunks in use: 65. 133.0KiB allocated for chunks. 133.0KiB in use in bin. 131.4KiB client-requested in use in bin.
2020-09-16 09:51:50.019167: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (4096): Total Chunks: 28, Chunks in use: 28. 114.0KiB allocated for chunks. 114.0KiB in use in bin. 112.0KiB client-requested in use in bin.
2020-09-16 09:51:50.019651: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (8192): Total Chunks: 1, Chunks in use: 1. 8.0KiB allocated for chunks. 8.0KiB in use in bin. 8.0KiB client-requested in use in bin.
2020-09-16 09:51:50.020119: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-16 09:51:50.020370: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (32768): Total Chunks: 2, Chunks in use: 2. 64.0KiB allocated for chunks. 64.0KiB in use in bin. 64.0KiB client-requested in use in bin.
2020-09-16 09:51:50.020676: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (65536): Total Chunks: 3, Chunks in use: 2. 223.5KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2020-09-16 09:51:50.021014: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (131072): Total Chunks: 8, Chunks in use: 8. 1.00MiB allocated for chunks. 1.00MiB in use in bin. 1.00MiB client-requested in use in bin.
2020-09-16 09:51:50.021218: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (262144): Total Chunks: 3, Chunks in use: 3. 864.0KiB allocated for chunks. 864.0KiB in use in bin. 864.0KiB client-requested in use in bin.
2020-09-16 09:51:50.021436: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (524288): Total Chunks: 10, Chunks in use: 8. 5.28MiB allocated for chunks. 4.28MiB in use in bin. 4.00MiB client-requested in use in bin.
2020-09-16 09:51:50.021801: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (1048576): Total Chunks: 9, Chunks in use: 9. 10.88MiB allocated for chunks. 10.88MiB in use in bin. 10.12MiB client-requested in use in bin.
2020-09-16 09:51:50.022083: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (2097152): Total Chunks: 9, Chunks in use: 7. 19.50MiB allocated for chunks. 15.00MiB in use in bin. 14.00MiB client-requested in use in bin.
2020-09-16 09:51:50.022247: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (4194304): Total Chunks: 9, Chunks in use: 9. 42.38MiB allocated for chunks. 42.38MiB in use in bin. 40.50MiB client-requested in use in bin.
2020-09-16 09:51:50.022412: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-16 09:51:50.022600: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (16777216): Total Chunks: 10, Chunks in use: 9. 194.75MiB allocated for chunks. 178.75MiB in use in bin. 162.00MiB client-requested in use in bin.
2020-09-16 09:51:50.022855: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-16 09:51:50.023353: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-16 09:51:50.023601: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-16 09:51:50.023997: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-16 09:51:50.024224: I tensorflow/core/common_runtime/bfc_allocator.cc:917] Bin for 18.00MiB was 16.00MiB, Chunk State:
2020-09-16 09:51:50.024420: I tensorflow/core/common_runtime/bfc_allocator.cc:923] Size: 16.00MiB | Requested Size: 4B | in_use: 0 | bin_num: 16, prev: Size: 2.00MiB | Requested Size: 2.00MiB | in_use: 1 | bin_num: -1, next: Size: 20.00MiB | Requested Size: 18.00MiB | in_use: 1 | bin_num: -1
2020-09-16 09:51:50.024713: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 1048576
2020-09-16 09:51:50.024879: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0000 of size 1280 next 1
2020-09-16 09:51:50.025027: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0500 of size 256 next 2
2020-09-16 09:51:50.025380: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0600 of size 256 next 5
2020-09-16 09:51:50.025526: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0700 of size 256 next 4
2020-09-16 09:51:50.025928: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0800 of size 256 next 7
2020-09-16 09:51:50.026086: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0900 of size 256 next 8
2020-09-16 09:51:50.026240: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0a00 of size 256 next 9
2020-09-16 09:51:50.026440: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0b00 of size 256 next 11
2020-09-16 09:51:50.026616: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0c00 of size 256 next 13
2020-09-16 09:51:50.026909: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0d00 of size 256 next 14
2020-09-16 09:51:50.027054: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0e00 of size 256 next 15
2020-09-16 09:51:50.027190: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0f00 of size 256 next 17
2020-09-16 09:51:50.027493: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1000 of size 256 next 19
2020-09-16 09:51:50.027581: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1100 of size 256 next 20
2020-09-16 09:51:50.027659: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1200 of size 256 next 22
2020-09-16 09:51:50.027746: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1300 of size 256 next 24
2020-09-16 09:51:50.027840: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1400 of size 256 next 25
2020-09-16 09:51:50.028016: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1500 of size 256 next 26
2020-09-16 09:51:50.028126: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1600 of size 512 next 27
2020-09-16 09:51:50.028231: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1800 of size 512 next 30
2020-09-16 09:51:50.028352: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1a00 of size 512 next 31
2020-09-16 09:51:50.028456: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1c00 of size 512 next 32
2020-09-16 09:51:50.028572: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1e00 of size 256 next 33
2020-09-16 09:51:50.028682: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1f00 of size 256 next 35
2020-09-16 09:51:50.028794: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f2000 of size 256 next 36
2020-09-16 09:51:50.028905: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f2100 of size 256 next 37
2020-09-16 09:51:50.029125: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f2200 of size 512 next 3
2020-09-16 09:51:50.029246: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f2400 of size 3584 next 6
2020-09-16 09:51:50.029358: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3200 of size 512 next 38
2020-09-16 09:51:50.029483: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3400 of size 512 next 40
2020-09-16 09:51:50.029559: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3600 of size 512 next 41
2020-09-16 09:51:50.029627: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3800 of size 256 next 42
2020-09-16 09:51:50.029719: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3900 of size 256 next 44
2020-09-16 09:51:50.029782: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3a00 of size 256 next 45
2020-09-16 09:51:50.029842: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3b00 of size 256 next 46
2020-09-16 09:51:50.029898: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3c00 of size 512 next 47
2020-09-16 09:51:50.029954: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3e00 of size 512 next 49
2020-09-16 09:51:50.030009: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4000 of size 512 next 50
2020-09-16 09:51:50.030190: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4200 of size 512 next 51
2020-09-16 09:51:50.030260: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4400 of size 1024 next 53
2020-09-16 09:51:50.030329: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4800 of size 1024 next 56
2020-09-16 09:51:50.030399: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4c00 of size 1024 next 57
2020-09-16 09:51:50.030471: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5000 of size 1024 next 58
2020-09-16 09:51:50.030546: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5400 of size 512 next 59
2020-09-16 09:51:50.030621: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5600 of size 512 next 61
2020-09-16 09:51:50.030697: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5800 of size 512 next 63
2020-09-16 09:51:50.030773: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5a00 of size 512 next 64
2020-09-16 09:51:50.030849: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5c00 of size 1024 next 65
2020-09-16 09:51:50.030925: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6000 of size 1024 next 66
2020-09-16 09:51:50.031001: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6400 of size 1024 next 67
2020-09-16 09:51:50.031077: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6800 of size 1024 next 68
2020-09-16 09:51:50.031153: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6c00 of size 512 next 69
2020-09-16 09:51:50.031229: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6e00 of size 512 next 71
2020-09-16 09:51:50.031393: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f7000 of size 512 next 18
2020-09-16 09:51:50.031477: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f7200 of size 8192 next 16
2020-09-16 09:51:50.031556: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f9200 of size 32768 next 34
2020-09-16 09:51:50.031654: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00901200 of size 32768 next 43
2020-09-16 09:51:50.031729: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00909200 of size 512 next 72
2020-09-16 09:51:50.031803: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00909400 of size 1024 next 75
2020-09-16 09:51:50.031877: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00909800 of size 1024 next 76
2020-09-16 09:51:50.031951: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00909c00 of size 1024 next 77
2020-09-16 09:51:50.032097: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090a000 of size 1024 next 78
2020-09-16 09:51:50.032173: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090a400 of size 512 next 79
2020-09-16 09:51:50.032248: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090a600 of size 512 next 80
2020-09-16 09:51:50.032321: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090a800 of size 512 next 82
2020-09-16 09:51:50.032395: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090aa00 of size 512 next 83
2020-09-16 09:51:50.032469: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090ac00 of size 1024 next 85
2020-09-16 09:51:50.032550: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090b000 of size 1024 next 86
2020-09-16 09:51:50.032711: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090b400 of size 1024 next 87
2020-09-16 09:51:50.032770: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090b800 of size 1024 next 88
2020-09-16 09:51:50.032819: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090bc00 of size 512 next 90
2020-09-16 09:51:50.032865: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090be00 of size 512 next 91
2020-09-16 09:51:50.032910: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090c000 of size 512 next 92
2020-09-16 09:51:50.032952: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090c200 of size 512 next 93
2020-09-16 09:51:50.032995: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090c400 of size 1024 next 95
2020-09-16 09:51:50.033040: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090c800 of size 1024 next 97
2020-09-16 09:51:50.033086: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090cc00 of size 1024 next 98
2020-09-16 09:51:50.033135: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090d000 of size 1024 next 99
2020-09-16 09:51:50.033188: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090d400 of size 512 next 101
2020-09-16 09:51:50.033242: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090d600 of size 512 next 102
2020-09-16 09:51:50.033363: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090d800 of size 512 next 103
2020-09-16 09:51:50.033423: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090da00 of size 512 next 104
2020-09-16 09:51:50.033484: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090dc00 of size 1024 next 106
2020-09-16 09:51:50.033545: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090e000 of size 1024 next 107
2020-09-16 09:51:50.033608: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090e400 of size 1024 next 108
2020-09-16 09:51:50.033697: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090e800 of size 1024 next 109
2020-09-16 09:51:50.033768: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090ec00 of size 512 next 111
2020-09-16 09:51:50.033832: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090ee00 of size 512 next 112
2020-09-16 09:51:50.033962: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090f000 of size 512 next 113
2020-09-16 09:51:50.034029: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090f200 of size 512 next 114
2020-09-16 09:51:50.034094: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090f400 of size 1024 next 116
2020-09-16 09:51:50.034158: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090f800 of size 1024 next 117
2020-09-16 09:51:50.034222: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090fc00 of size 1024 next 118
2020-09-16 09:51:50.034285: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910000 of size 1024 next 119
2020-09-16 09:51:50.034349: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910400 of size 512 next 121
2020-09-16 09:51:50.034413: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910600 of size 512 next 122
2020-09-16 09:51:50.034476: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910800 of size 512 next 123
2020-09-16 09:51:50.034615: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910a00 of size 512 next 124
2020-09-16 09:51:50.034683: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910c00 of size 1024 next 125
2020-09-16 09:51:50.034747: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911000 of size 1024 next 126
2020-09-16 09:51:50.034810: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911400 of size 1024 next 127
2020-09-16 09:51:50.034883: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911800 of size 1024 next 128
2020-09-16 09:51:50.034930: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911c00 of size 512 next 130
2020-09-16 09:51:50.034970: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911e00 of size 512 next 131
2020-09-16 09:51:50.035009: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912000 of size 512 next 132
2020-09-16 09:51:50.035049: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912200 of size 512 next 133
2020-09-16 09:51:50.035090: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912400 of size 1024 next 136
2020-09-16 09:51:50.035130: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912800 of size 1024 next 137
2020-09-16 09:51:50.035170: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912c00 of size 1024 next 138
2020-09-16 09:51:50.035285: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00913000 of size 1024 next 139
2020-09-16 09:51:50.035333: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00913400 of size 2048 next 140
2020-09-16 09:51:50.035382: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00913c00 of size 2048 next 141
2020-09-16 09:51:50.035432: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00914400 of size 2048 next 142
2020-09-16 09:51:50.035483: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00914c00 of size 2048 next 144
2020-09-16 09:51:50.035535: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00915400 of size 1024 next 145
2020-09-16 09:51:50.035588: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00915800 of size 1024 next 146
2020-09-16 09:51:50.035642: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00915c00 of size 1024 next 148
2020-09-16 09:51:50.035695: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00916000 of size 1024 next 149
2020-09-16 09:51:50.035753: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00916400 of size 3584 next 12
2020-09-16 09:51:50.035811: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00917200 of size 73728 next 10
2020-09-16 09:51:50.035872: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00929200 of size 2048 next 150
2020-09-16 09:51:50.036002: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00929a00 of size 2048 next 153
2020-09-16 09:51:50.036047: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092a200 of size 2048 next 154
2020-09-16 09:51:50.036084: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092aa00 of size 1024 next 156
2020-09-16 09:51:50.036121: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092ae00 of size 1024 next 157
2020-09-16 09:51:50.036157: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092b200 of size 1024 next 158
2020-09-16 09:51:50.036195: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092b600 of size 1024 next 159
2020-09-16 09:51:50.036229: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092ba00 of size 2048 next 160
2020-09-16 09:51:50.036262: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092c200 of size 2048 next 162
2020-09-16 09:51:50.036297: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092ca00 of size 2048 next 163
2020-09-16 09:51:50.036334: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092d200 of size 2048 next 164
2020-09-16 09:51:50.036372: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092da00 of size 1024 next 165
2020-09-16 09:51:50.036411: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092de00 of size 1024 next 166
2020-09-16 09:51:50.036452: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092e200 of size 1024 next 167
2020-09-16 09:51:50.036495: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092e600 of size 1024 next 168
2020-09-16 09:51:50.036539: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092ea00 of size 2048 next 169
2020-09-16 09:51:50.036584: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092f200 of size 2048 next 171
2020-09-16 09:51:50.036630: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092fa00 of size 2048 next 172
2020-09-16 09:51:50.036677: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00930200 of size 2048 next 173
2020-09-16 09:51:50.036724: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00930a00 of size 1024 next 175
2020-09-16 09:51:50.036772: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00930e00 of size 1024 next 176
2020-09-16 09:51:50.036905: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00931200 of size 1024 next 177
2020-09-16 09:51:50.036964: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00931600 of size 1024 next 178
2020-09-16 09:51:50.037016: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00931a00 of size 2048 next 180
2020-09-16 09:51:50.037073: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00932200 of size 2048 next 182
2020-09-16 09:51:50.037110: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00932a00 of size 2048 next 183
2020-09-16 09:51:50.037136: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00933200 of size 2048 next 184
2020-09-16 09:51:50.037161: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00933a00 of size 1024 next 186
2020-09-16 09:51:50.037184: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00933e00 of size 1024 next 187
2020-09-16 09:51:50.037207: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00934200 of size 1024 next 188
2020-09-16 09:51:50.037230: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00934600 of size 1024 next 189
2020-09-16 09:51:50.037253: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00934a00 of size 2048 next 190
2020-09-16 09:51:50.037275: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00935200 of size 2048 next 191
2020-09-16 09:51:50.037297: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00935a00 of size 2048 next 193
2020-09-16 09:51:50.037318: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00936200 of size 2048 next 194
2020-09-16 09:51:50.037340: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00936a00 of size 1024 next 196
2020-09-16 09:51:50.037361: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00936e00 of size 1024 next 197
2020-09-16 09:51:50.037383: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00937200 of size 1024 next 198
2020-09-16 09:51:50.037406: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00937600 of size 1024 next 199
2020-09-16 09:51:50.037482: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00937a00 of size 2048 next 202
2020-09-16 09:51:50.037509: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00938200 of size 2048 next 203
2020-09-16 09:51:50.037535: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00938a00 of size 2048 next 204
2020-09-16 09:51:50.037563: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00939200 of size 2048 next 205
2020-09-16 09:51:50.037590: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00939a00 of size 1024 next 207
2020-09-16 09:51:50.037618: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00939e00 of size 1024 next 208
2020-09-16 09:51:50.037646: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0093a200 of size 1024 next 209
2020-09-16 09:51:50.037693: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0093a600 of size 1024 next 210
2020-09-16 09:51:50.037729: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0093aa00 of size 2048 next 23
2020-09-16 09:51:50.037757: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0093b200 of size 73728 next 21
2020-09-16 09:51:50.037782: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094d200 of size 2048 next 212
2020-09-16 09:51:50.037807: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094da00 of size 2048 next 213
2020-09-16 09:51:50.037831: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094e200 of size 2048 next 214
2020-09-16 09:51:50.037854: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094ea00 of size 1024 next 216
2020-09-16 09:51:50.037878: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094ee00 of size 1024 next 217
2020-09-16 09:51:50.037903: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094f200 of size 1024 next 218
2020-09-16 09:51:50.037927: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094f600 of size 1024 next 219
2020-09-16 09:51:50.037951: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094fa00 of size 2048 next 221
2020-09-16 09:51:50.037976: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00950200 of size 2048 next 222
2020-09-16 09:51:50.045954: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00950a00 of size 2048 next 223
2020-09-16 09:51:50.046025: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00951200 of size 2048 next 224
2020-09-16 09:51:50.046055: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00951a00 of size 4096 next 225
2020-09-16 09:51:50.046085: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00952a00 of size 4096 next 228
2020-09-16 09:51:50.046112: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00953a00 of size 4096 next 229
2020-09-16 09:51:50.046140: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00954a00 of size 4096 next 230
2020-09-16 09:51:50.046169: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00955a00 of size 2048 next 231
2020-09-16 09:51:50.046198: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00956200 of size 2048 next 232
2020-09-16 09:51:50.046229: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00956a00 of size 2048 next 234
2020-09-16 09:51:50.046259: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00957200 of size 2048 next 235
2020-09-16 09:51:50.046298: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00957a00 of size 4096 next 236
2020-09-16 09:51:50.046330: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00958a00 of size 4096 next 238
2020-09-16 09:51:50.046356: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00959a00 of size 4096 next 239
2020-09-16 09:51:50.046383: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095aa00 of size 4096 next 240
2020-09-16 09:51:50.046419: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095ba00 of size 2048 next 242
2020-09-16 09:51:50.046450: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095c200 of size 2048 next 243
2020-09-16 09:51:50.046478: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095ca00 of size 2048 next 244
2020-09-16 09:51:50.046524: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095d200 of size 2048 next 245
2020-09-16 09:51:50.046559: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095da00 of size 4096 next 246
2020-09-16 09:51:50.046587: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095ea00 of size 4096 next 248
2020-09-16 09:51:50.046614: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095fa00 of size 4096 next 249
2020-09-16 09:51:50.046648: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00960a00 of size 4096 next 250
2020-09-16 09:51:50.046676: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00961a00 of size 2048 next 252
2020-09-16 09:51:50.046704: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00962200 of size 2048 next 253
2020-09-16 09:51:50.046738: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00962a00 of size 2048 next 254
2020-09-16 09:51:50.046766: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00963200 of size 2048 next 255
2020-09-16 09:51:50.046794: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00963a00 of size 4096 next 256
2020-09-16 09:51:50.046828: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00964a00 of size 4096 next 258
2020-09-16 09:51:50.046863: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00965a00 of size 4096 next 259
2020-09-16 09:51:50.046892: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00966a00 of size 4096 next 260
2020-09-16 09:51:50.046918: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00967a00 of size 2048 next 262
2020-09-16 09:51:50.046951: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00968200 of size 2048 next 263
2020-09-16 09:51:50.046986: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00968a00 of size 2048 next 264
2020-09-16 09:51:50.047020: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00969200 of size 2048 next 265
2020-09-16 09:51:50.047054: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00969a00 of size 4096 next 266
2020-09-16 09:51:50.047088: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0096aa00 of size 4096 next 268
2020-09-16 09:51:50.047123: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0096ba00 of size 6144 next 81
2020-09-16 09:51:50.047159: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0096d200 of size 131072 next 62
2020-09-16 09:51:50.047194: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0098d200 of size 131072 next 60
2020-09-16 09:51:50.047229: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009ad200 of size 131072 next 70
2020-09-16 09:51:50.047264: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009cd200 of size 4096 next 269
2020-09-16 09:51:50.047298: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009ce200 of size 2048 next 270
2020-09-16 09:51:50.047331: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009cea00 of size 2048 next 271
2020-09-16 09:51:50.047365: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009cf200 of size 2048 next 272
2020-09-16 09:51:50.047399: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009cfa00 of size 2048 next 273
2020-09-16 09:51:50.047433: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d0200 of size 4096 next 274
2020-09-16 09:51:50.047467: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d1200 of size 4096 next 276
2020-09-16 09:51:50.047503: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d2200 of size 4096 next 277
2020-09-16 09:51:50.047538: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d3200 of size 4096 next 278
2020-09-16 09:51:50.047572: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d4200 of size 2048 next 280
2020-09-16 09:51:50.047606: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d4a00 of size 2048 next 281
2020-09-16 09:51:50.047640: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d5200 of size 2048 next 282
2020-09-16 09:51:50.047674: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d5a00 of size 2048 next 283
2020-09-16 09:51:50.047708: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d6200 of size 4096 next 284
2020-09-16 09:51:50.047741: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d7200 of size 4096 next 285
2020-09-16 09:51:50.047775: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d8200 of size 4096 next 287
2020-09-16 09:51:50.047809: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d9200 of size 4096 next 288
2020-09-16 09:51:50.047843: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009da200 of size 2048 next 290
2020-09-16 09:51:50.047876: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009daa00 of size 2048 next 291
2020-09-16 09:51:50.047910: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009db200 of size 2048 next 292
2020-09-16 09:51:50.047944: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009dba00 of size 2048 next 293
2020-09-16 09:51:50.047978: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f009dc200 of size 81408 next 18446744073709551615
2020-09-16 09:51:50.048013: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 2097152
2020-09-16 09:51:50.048048: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009f0000 of size 294912 next 29
2020-09-16 09:51:50.048084: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00a38000 of size 294912 next 39
2020-09-16 09:51:50.048118: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00a80000 of size 294912 next 48
2020-09-16 09:51:50.048152: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00ac8000 of size 524288 next 155
2020-09-16 09:51:50.048188: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00b48000 of size 688128 next 18446744073709551615
2020-09-16 09:51:50.048222: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 4194304
2020-09-16 09:51:50.048256: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00bf0000 of size 131072 next 89
2020-09-16 09:51:50.048291: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00c10000 of size 131072 next 100
2020-09-16 09:51:50.048325: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00c30000 of size 131072 next 110
2020-09-16 09:51:50.048360: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00c50000 of size 131072 next 120
2020-09-16 09:51:50.048394: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00c70000 of size 131072 next 129
2020-09-16 09:51:50.048428: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f00c90000 of size 524288 next 55
2020-09-16 09:51:50.048463: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00d10000 of size 1179648 next 52
2020-09-16 09:51:50.048499: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00e30000 of size 1835008 next 18446744073709551615
2020-09-16 09:51:50.048533: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 8388608
2020-09-16 09:51:50.048566: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00ff0000 of size 1179648 next 74
2020-09-16 09:51:50.048602: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01110000 of size 1179648 next 84
2020-09-16 09:51:50.048637: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f01230000 of size 524288 next 147
2020-09-16 09:51:50.048672: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f012b0000 of size 655360 next 96
2020-09-16 09:51:50.048709: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01350000 of size 1179648 next 94
2020-09-16 09:51:50.048744: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01470000 of size 1179648 next 105
2020-09-16 09:51:50.048779: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01590000 of size 1179648 next 115
2020-09-16 09:51:50.048816: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f016b0000 of size 1310720 next 18446744073709551615
2020-09-16 09:51:50.048849: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 16777216
2020-09-16 09:51:50.048884: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f017f0000 of size 1179648 next 135
2020-09-16 09:51:50.048919: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01910000 of size 524288 next 174
2020-09-16 09:51:50.048954: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01990000 of size 2097152 next 241
2020-09-16 09:51:50.048989: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01b90000 of size 2097152 next 251
2020-09-16 09:51:50.049024: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01d90000 of size 2097152 next 261
2020-09-16 09:51:50.049057: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f01f90000 of size 2621440 next 143
2020-09-16 09:51:50.049092: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f02210000 of size 6160384 next 18446744073709551615
2020-09-16 09:51:50.049125: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 33554432
2020-09-16 09:51:50.049162: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f027f0000 of size 4718592 next 152
2020-09-16 09:51:50.049197: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f02c70000 of size 4718592 next 161
2020-09-16 09:51:50.049231: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f030f0000 of size 4718592 next 170
2020-09-16 09:51:50.049265: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f03570000 of size 524288 next 185
2020-09-16 09:51:50.049299: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f035f0000 of size 524288 next 195
2020-09-16 09:51:50.049334: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f03670000 of size 524288 next 206
2020-09-16 09:51:50.049367: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f036f0000 of size 524288 next 215
2020-09-16 09:51:50.049402: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f03770000 of size 2621440 next 181
2020-09-16 09:51:50.049438: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f039f0000 of size 4718592 next 179
2020-09-16 09:51:50.049472: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f03e70000 of size 2097152 next 233
2020-09-16 09:51:50.049506: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f04070000 of size 2621440 next 192
2020-09-16 09:51:50.049541: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f042f0000 of size 5242880 next 18446744073709551615
2020-09-16 09:51:50.049574: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 67108864
2020-09-16 09:51:50.049609: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f047f0000 of size 4718592 next 201
2020-09-16 09:51:50.049644: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f04c70000 of size 4718592 next 211
2020-09-16 09:51:50.049733: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f050f0000 of size 4718592 next 220
2020-09-16 09:51:50.049773: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f05570000 of size 18874368 next 275
2020-09-16 09:51:50.049809: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f06770000 of size 2097152 next 279
2020-09-16 09:51:50.049844: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f06970000 of size 31981568 next 18446744073709551615
2020-09-16 09:51:50.049879: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 134217728
2020-09-16 09:51:50.049916: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f087f0000 of size 18874368 next 227
2020-09-16 09:51:50.049952: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f099f0000 of size 18874368 next 237
2020-09-16 09:51:50.049977: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0abf0000 of size 18874368 next 247
2020-09-16 09:51:50.050000: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0bdf0000 of size 18874368 next 257
2020-09-16 09:51:50.050023: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0cff0000 of size 18874368 next 267
2020-09-16 09:51:50.050045: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0e1f0000 of size 2097152 next 289
2020-09-16 09:51:50.050066: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f0e3f0000 of size 16777216 next 286
2020-09-16 09:51:50.050089: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0f3f0000 of size 20971520 next 18446744073709551615
2020-09-16 09:51:50.050110: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 21233664
2020-09-16 09:51:50.050133: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f107f0000 of size 21233664 next 18446744073709551615
2020-09-16 09:51:50.050154: I tensorflow/core/common_runtime/bfc_allocator.cc:955] Summary of in-use Chunks by size:
2020-09-16 09:51:50.050185: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 25 Chunks of size 256 totalling 6.2KiB
2020-09-16 09:51:50.050212: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 44 Chunks of size 512 totalling 22.0KiB
2020-09-16 09:51:50.050237: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 68 Chunks of size 1024 totalling 68.0KiB
2020-09-16 09:51:50.050261: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 1280 totalling 1.2KiB
2020-09-16 09:51:50.050286: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 63 Chunks of size 2048 totalling 126.0KiB
2020-09-16 09:51:50.050309: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 3584 totalling 7.0KiB
2020-09-16 09:51:50.050333: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 27 Chunks of size 4096 totalling 108.0KiB
2020-09-16 09:51:50.050357: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 6144 totalling 6.0KiB
2020-09-16 09:51:50.050382: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 8192 totalling 8.0KiB
2020-09-16 09:51:50.050406: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 32768 totalling 64.0KiB
2020-09-16 09:51:50.050432: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 73728 totalling 144.0KiB
2020-09-16 09:51:50.050456: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 8 Chunks of size 131072 totalling 1.00MiB
2020-09-16 09:51:50.050482: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 3 Chunks of size 294912 totalling 864.0KiB
2020-09-16 09:51:50.050507: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 6 Chunks of size 524288 totalling 3.00MiB
2020-09-16 09:51:50.050533: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 655360 totalling 640.0KiB
2020-09-16 09:51:50.050560: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 688128 totalling 672.0KiB
2020-09-16 09:51:50.050586: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 7 Chunks of size 1179648 totalling 7.88MiB
2020-09-16 09:51:50.050611: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 1310720 totalling 1.25MiB
2020-09-16 09:51:50.050642: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 1835008 totalling 1.75MiB
2020-09-16 09:51:50.050669: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 5 Chunks of size 2097152 totalling 10.00MiB
2020-09-16 09:51:50.050699: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 2621440 totalling 5.00MiB
2020-09-16 09:51:50.050721: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 7 Chunks of size 4718592 totalling 31.50MiB
2020-09-16 09:51:50.050746: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 5242880 totalling 5.00MiB
2020-09-16 09:51:50.050773: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 6160384 totalling 5.88MiB
2020-09-16 09:51:50.052009: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 6 Chunks of size 18874368 totalling 108.00MiB
2020-09-16 09:51:50.052068: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 20971520 totalling 20.00MiB
2020-09-16 09:51:50.052106: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 21233664 totalling 20.25MiB
2020-09-16 09:51:50.052146: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 31981568 totalling 30.50MiB
2020-09-16 09:51:50.052182: I tensorflow/core/common_runtime/bfc_allocator.cc:962] Sum Total of in-use chunks: 253.67MiB
2020-09-16 09:51:50.052215: I tensorflow/core/common_runtime/bfc_allocator.cc:964] total_region_allocated_bytes_: 288620544 memory_limit_: 288620544 available bytes: 0 curr_region_allocation_bytes_: 536870912
2020-09-16 09:51:50.071114: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Stats:
Limit: 288620544
InUse: 265994752
MaxInUse: 265994752
NumAllocs: 1432
MaxAllocSize: 34078720

2020-09-16 09:51:50.071237: W tensorflow/core/common_runtime/bfc_allocator.cc:429] xxxx_____*******
2020-09-16 09:51:50.107477: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at cwise_ops_common.cc:82 : Resource exhausted: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File “app.py”, line 30, in
yolo = YoloV3(classes=num_classes)
File “/home/christopher/aiguyyolotest1/Object-Detection-API/yolov3_tf2/models.py”, line 217, in YoloV3
output_0 = YoloOutput(512, len(masks[0]), classes, name=‘yolo_output_0’)(x)
File “/home/christopher/aiguyyolotest1/Object-Detection-API/yolov3_tf2/models.py”, line 146, in yolo_output
x = DarknetConv(x, filters * 2, 3)
File “/home/christopher/aiguyyolotest1/Object-Detection-API/yolov3_tf2/models.py”, line 52, in DarknetConv
use_bias=not batch_norm, kernel_regularizer=l2(0.0005))(x)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py”, line 748, in call
self._maybe_build(inputs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py”, line 2116, in _maybe_build
self.build(input_shapes)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/layers/convolutional.py”, line 161, in build
dtype=self.dtype)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py”, line 446, in add_weight
caching_device=caching_device)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/tracking/base.py”, line 744, in _add_variable_with_custom_getter
**kwargs_for_getter)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer_utils.py”, line 142, in make_variable
shape=variable_shape if variable_shape else None)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py”, line 258, in call
return cls._variable_v1_call(*args, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py”, line 219, in _variable_v1_call
shape=shape)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py”, line 197, in
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variable_scope.py”, line 2596, in default_variable_creator
shape=shape)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py”, line 262, in call
return super(VariableMetaclass, cls).call(*args, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py”, line 1411, in init
distribute_strategy=distribute_strategy)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py”, line 1542, in _init_from_args
initial_value() if init_from_fn else initial_value,
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer_utils.py”, line 122, in
init_val = lambda: initializer(shape, dtype=dtype)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops_v2.py”, line 425, in call
return self._random_generator.random_uniform(shape, -limit, limit, dtype)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops_v2.py”, line 788, in random_uniform
shape=shape, minval=minval, maxval=maxval, dtype=dtype, seed=self.seed)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/random_ops.py”, line 273, in random_uniform
result = math_ops.add(rnd * (maxval - minval), minval, name=name)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_math_ops.py”, line 343, in add
_ops.raise_from_not_ok_status(e, name)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py”, line 6606, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File “”, line 3, in raise_from
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Add] name: conv2d_57/kernel/Initializer/random_uniform/

Hi,

Could you try the GPU configure shared in this comment first?

Thanks.

Hi,

I copied this

gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.6)
sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))
config = tf.ConfigProto()
config.gpu_options.allow_growth=True
sess = tf.Session(config=config)

into the python file.

Unfortunately there is a problem with a missing attribute:

gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.6)
AttributeError: module ‘tensorflow’ has no attribute ‘GPUOptions’

I also tried this

tf.config.gpu.set_per_process_memory_fraction(0.6)
tf.config.gpu.set_per_process_memory_growth(True)

But also with that i got an error message:

tf.config.gpu.set_per_process_memory_fraction(0.6)
AttributeError: module ‘tensorflow_core._api.v2.config’ has no attribute ‘gpu’

I’m using Tensorflow version 2.1.0

Hi,

For TensorFlow v2.x, please try the following:

device = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(device[0], True)
tf.config.experimental.set_virtual_device_configuration(device[0], [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=1024)])

Thanks.

1 Like

First things first. Thanks for your help. I’m really very pleased about this. Your advice seemed to help and solved the problem.

I’m really sorry for asking always new questions but there is a new problem. I already tried to solve the new problem on my own but I did not find a solution (unfortunatelly I have not much experience and still have to learn a lot …).

What I did is changing the code regarding the physical and virtual devices but nothing helped.

In the following you find the error description:

(aiguyyolotest1) christopher@ccz:~/aiguyyolotest1/Object-Detection-API$ python app.py
2020-09-18 08:52:14.003805: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-18 08:52:16.973431: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.6
2020-09-18 08:52:16.975454: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.6
2020-09-18 08:52:23.799496: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-09-18 08:52:23.812897: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-18 08:52:23.813114: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 23.84GiB/s
2020-09-18 08:52:23.813292: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-18 08:52:23.813477: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-09-18 08:52:23.816538: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-09-18 08:52:23.817772: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-09-18 08:52:23.821916: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-09-18 08:52:23.825056: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-09-18 08:52:23.825358: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-09-18 08:52:23.825841: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-18 08:52:23.826278: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-18 08:52:23.826374: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
Traceback (most recent call last):
File “app.py”, line 38, in
tf.config.experimental.set_memory_growth(physical_devices[0], True)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/config.py”, line 494, in set_memory_growth
context.context().set_memory_growth(device, enable)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/context.py”, line 1231, in set_memory_growth
“Cannot set memory growth on device when virtual devices configured”)
ValueError: Cannot set memory growth on device when virtual devices configured

This is the original python file into which i copied the lines of code you told me.

import time
from absl import app, logging
import cv2
import numpy as np
import tensorflow as tf
from yolov3_tf2.models import (
    YoloV3, YoloV3Tiny
)
from yolov3_tf2.dataset import transform_images, load_tfrecord_dataset
from yolov3_tf2.utils import draw_outputs
from flask import Flask, request, Response, jsonify, send_from_directory, abort
import os

# customize your API through the following parameters
classes_path = './data/labels/coco.names'
weights_path = './weights/yolov3.tf'
tiny = False                    # set to True if using a Yolov3 Tiny model
size = 416                      # size images are resized to for model
output_path = './detections/'   # path to output folder where images with detections are saved
num_classes = 80                # number of classes in model

# load in weights and classes
physical_devices = tf.config.experimental.list_physical_devices('GPU')
if len(physical_devices) > 0:
    tf.config.experimental.set_memory_growth(physical_devices[0], True)

if tiny:
    yolo = YoloV3Tiny(classes=num_classes)
else:
    yolo = YoloV3(classes=num_classes)

yolo.load_weights(weights_path).expect_partial()
print('weights loaded')

class_names = [c.strip() for c in open(classes_path).readlines()]
print('classes loaded')

# Initialize Flask application
app = Flask(__name__)

# API that returns JSON with classes found in images
@app.route('/detections', methods=['POST'])
def get_detections():
    raw_images = []
    images = request.files.getlist("images")
    image_names = []
    for image in images:
        image_name = image.filename
        image_names.append(image_name)
        image.save(os.path.join(os.getcwd(), image_name))
        img_raw = tf.image.decode_image(
            open(image_name, 'rb').read(), channels=3)
        raw_images.append(img_raw)
        
    num = 0
    
    # create list for final response
    response = []

    for j in range(len(raw_images)):
        # create list of responses for current image
        responses = []
        raw_img = raw_images[j]
        num+=1
        img = tf.expand_dims(raw_img, 0)
        img = transform_images(img, size)

        t1 = time.time()
        boxes, scores, classes, nums = yolo(img)
        t2 = time.time()
        print('time: {}'.format(t2 - t1))

        print('detections:')
        for i in range(nums[0]):
            print('\t{}, {}, {}'.format(class_names[int(classes[0][i])],
                                            np.array(scores[0][i]),
                                            np.array(boxes[0][i])))
            responses.append({
                "class": class_names[int(classes[0][i])],
                "confidence": float("{0:.2f}".format(np.array(scores[0][i])*100))
            })
        response.append({
            "image": image_names[j],
            "detections": responses
        })
        img = cv2.cvtColor(raw_img.numpy(), cv2.COLOR_RGB2BGR)
        img = draw_outputs(img, (boxes, scores, classes, nums), class_names)
        cv2.imwrite(output_path + 'detection' + str(num) + '.jpg', img)
        print('output saved to: {}'.format(output_path + 'detection' + str(num) + '.jpg'))

    #remove temporary images
    for name in image_names:
        os.remove(name)
    try:
        return jsonify({"response":response}), 200
    except FileNotFoundError:
        abort(404)

# API that returns image with detections on it
@app.route('/image', methods= ['POST'])
def get_image():
    image = request.files["images"]
    image_name = image.filename
    image.save(os.path.join(os.getcwd(), image_name))
    img_raw = tf.image.decode_image(
        open(image_name, 'rb').read(), channels=3)
    img = tf.expand_dims(img_raw, 0)
    img = transform_images(img, size)

    t1 = time.time()
    boxes, scores, classes, nums = yolo(img)
    t2 = time.time()
    print('time: {}'.format(t2 - t1))

    print('detections:')
    for i in range(nums[0]):
        print('\t{}, {}, {}'.format(class_names[int(classes[0][i])],
                                        np.array(scores[0][i]),
                                        np.array(boxes[0][i])))
    img = cv2.cvtColor(img_raw.numpy(), cv2.COLOR_RGB2BGR)
    img = draw_outputs(img, (boxes, scores, classes, nums), class_names)
    cv2.imwrite(output_path + 'detection.jpg', img)
    print('output saved to: {}'.format(output_path + 'detection.jpg'))
    
    # prepare image for response
    _, img_encoded = cv2.imencode('.png', img)
    response = img_encoded.tostring()
    
    #remove temporary image
    os.remove(image_name)

    try:
        return Response(response=response, status=200, mimetype='image/png')
    except FileNotFoundError:
        abort(404)
if __name__ == '__main__':
    app.run(debug=True, host = '0.0.0.0', port=5000)

I’m pretty sure the reason for the new error is that i copied the new lines of code wrong into the python file.
Could you please tell me how i can make it right?

Hi,

1.
If the configure doesn’t help, you may really meet the out of memory issue.
This can be verified via tegrastats.

$ sudo tegrastats

RAM 1784/3956MB (lfb 91x4MB) SWAP 30/1978MB (cached 1MB) IRAM 0/252kB(lfb 252kB) …

Please monitor the device with tegrastats at the same time and share the output log with us.

2.
Since Nano only has 4G memory, it may not have enough capacity for the YOLOv3 model.
Would you mind to use tiny version instead to see if works?

tiny = True                    # set to True if using a Yolov3 Tiny model
...

Thanks.

Thank you for your help in advance.

I try to test this as soon as possible and share the output with you after that.

Best regards

Hi,

these are the results:

At first the outup after running python3 app.py

(I did not apply this configuration Jetson Nano running out of memory: ResourceExhaustedError: OOM when allocating tensor with shape[3,3,512,1024 - #5 by AastaLLL while running the program because I could not correctly insert this configuration into the Python file as I described above.)

christopher@ccz:~$ source aiguyyolotest1/bin/activate
(aiguyyolotest1) christopher@ccz:~$ cd aiguyyolotest1
(aiguyyolotest1) christopher@ccz:~/aiguyyolotest1$ cd Object-Detection-API/
(aiguyyolotest1) christopher@ccz:~/aiguyyolotest1/Object-Detection-API$ python3 app.py
2020-09-30 09:29:22.856483: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-30 09:29:29.065965: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.6
2020-09-30 09:29:29.108129: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.6
2020-09-30 09:29:43.449528: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-09-30 09:29:43.492119: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-30 09:29:43.492306: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: 
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 23.84GiB/s
2020-09-30 09:29:43.492436: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-30 09:29:43.492607: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-09-30 09:29:43.578593: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-09-30 09:29:43.699116: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-09-30 09:29:43.839232: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-09-30 09:29:43.917010: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-09-30 09:29:43.917644: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-09-30 09:29:43.918622: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-30 09:29:43.919459: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-30 09:29:43.919638: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-09-30 09:29:44.036835: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2020-09-30 09:29:44.037683: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x23980050 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-09-30 09:29:44.037772: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-09-30 09:29:44.151225: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-30 09:29:44.151570: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x238ece70 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-09-30 09:29:44.151630: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
2020-09-30 09:29:44.152456: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-30 09:29:44.152595: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: 
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 23.84GiB/s
2020-09-30 09:29:44.152799: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-30 09:29:44.152893: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-09-30 09:29:44.153031: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-09-30 09:29:44.153133: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-09-30 09:29:44.153226: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-09-30 09:29:44.153314: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-09-30 09:29:44.153378: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-09-30 09:29:44.153723: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-30 09:29:44.154067: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-30 09:29:44.154147: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-09-30 09:29:44.154286: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-30 09:29:55.282967: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-09-30 09:29:55.283152: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102]      0 
2020-09-30 09:29:55.283188: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0:   N 
2020-09-30 09:29:55.300888: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-30 09:29:55.301693: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-30 09:29:55.306043: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 281 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2020-09-30 09:30:30.114452: W tensorflow/core/common_runtime/bfc_allocator.cc:424] Allocator (GPU_0_bfc) ran out of memory trying to allocate 18.00MiB (rounded to 18874368).  Current allocation summary follows.
2020-09-30 09:30:30.114702: I tensorflow/core/common_runtime/bfc_allocator.cc:894] BFCAllocator dump for GPU_0_bfc
2020-09-30 09:30:30.115116: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (256): 	Total Chunks: 25, Chunks in use: 25. 6.2KiB allocated for chunks. 6.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2020-09-30 09:30:30.115509: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (512): 	Total Chunks: 44, Chunks in use: 44. 22.0KiB allocated for chunks. 22.0KiB in use in bin. 22.0KiB client-requested in use in bin.
2020-09-30 09:30:30.115920: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (1024): 	Total Chunks: 69, Chunks in use: 69. 69.2KiB allocated for chunks. 69.2KiB in use in bin. 69.0KiB client-requested in use in bin.
2020-09-30 09:30:30.116322: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (2048): 	Total Chunks: 65, Chunks in use: 65. 133.0KiB allocated for chunks. 133.0KiB in use in bin. 131.4KiB client-requested in use in bin.
2020-09-30 09:30:30.116587: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (4096): 	Total Chunks: 28, Chunks in use: 28. 114.0KiB allocated for chunks. 114.0KiB in use in bin. 112.0KiB client-requested in use in bin.
2020-09-30 09:30:30.116901: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (8192): 	Total Chunks: 1, Chunks in use: 1. 8.0KiB allocated for chunks. 8.0KiB in use in bin. 8.0KiB client-requested in use in bin.
2020-09-30 09:30:30.117183: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (16384): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-30 09:30:30.117473: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (32768): 	Total Chunks: 2, Chunks in use: 2. 64.0KiB allocated for chunks. 64.0KiB in use in bin. 64.0KiB client-requested in use in bin.
2020-09-30 09:30:30.117745: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (65536): 	Total Chunks: 3, Chunks in use: 2. 223.5KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2020-09-30 09:30:30.118033: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (131072): 	Total Chunks: 8, Chunks in use: 8. 1.00MiB allocated for chunks. 1.00MiB in use in bin. 1.00MiB client-requested in use in bin.
2020-09-30 09:30:30.118355: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (262144): 	Total Chunks: 3, Chunks in use: 3. 864.0KiB allocated for chunks. 864.0KiB in use in bin. 864.0KiB client-requested in use in bin.
2020-09-30 09:30:30.118511: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (524288): 	Total Chunks: 10, Chunks in use: 8. 5.28MiB allocated for chunks. 4.28MiB in use in bin. 4.00MiB client-requested in use in bin.
2020-09-30 09:30:30.118649: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (1048576): 	Total Chunks: 9, Chunks in use: 9. 10.88MiB allocated for chunks. 10.88MiB in use in bin. 10.12MiB client-requested in use in bin.
2020-09-30 09:30:30.118797: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (2097152): 	Total Chunks: 9, Chunks in use: 7. 19.50MiB allocated for chunks. 15.00MiB in use in bin. 14.00MiB client-requested in use in bin.
2020-09-30 09:30:30.119035: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (4194304): 	Total Chunks: 9, Chunks in use: 9. 42.38MiB allocated for chunks. 42.38MiB in use in bin. 40.50MiB client-requested in use in bin.
2020-09-30 09:30:30.119226: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (8388608): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-30 09:30:30.119728: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (16777216): 	Total Chunks: 10, Chunks in use: 9. 201.17MiB allocated for chunks. 185.17MiB in use in bin. 162.00MiB client-requested in use in bin.
2020-09-30 09:30:30.119924: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (33554432): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-30 09:30:30.120323: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (67108864): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-30 09:30:30.120658: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (134217728): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-30 09:30:30.120873: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (268435456): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-30 09:30:30.121274: I tensorflow/core/common_runtime/bfc_allocator.cc:917] Bin for 18.00MiB was 16.00MiB, Chunk State: 
2020-09-30 09:30:30.121585: I tensorflow/core/common_runtime/bfc_allocator.cc:923]   Size: 16.00MiB | Requested Size: 4B | in_use: 0 | bin_num: 16, prev:   Size: 2.00MiB | Requested Size: 2.00MiB | in_use: 1 | bin_num: -1, next:   Size: 20.00MiB | Requested Size: 18.00MiB | in_use: 1 | bin_num: -1
2020-09-30 09:30:30.121734: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 1048576
2020-09-30 09:30:30.136911: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0000 of size 1280 next 1
2020-09-30 09:30:30.137084: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0500 of size 256 next 2
2020-09-30 09:30:30.137257: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0600 of size 256 next 5
2020-09-30 09:30:30.137384: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0700 of size 256 next 4
2020-09-30 09:30:30.137516: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0800 of size 256 next 7
2020-09-30 09:30:30.137646: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0900 of size 256 next 8
2020-09-30 09:30:30.137780: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0a00 of size 256 next 9
2020-09-30 09:30:30.137910: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0b00 of size 256 next 11
2020-09-30 09:30:30.138040: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0c00 of size 256 next 13
2020-09-30 09:30:30.138169: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0d00 of size 256 next 14
2020-09-30 09:30:30.138303: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0e00 of size 256 next 15
2020-09-30 09:30:30.138433: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0f00 of size 256 next 17
2020-09-30 09:30:30.138596: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1000 of size 256 next 19
2020-09-30 09:30:30.138746: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1100 of size 256 next 20
2020-09-30 09:30:30.139481: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1200 of size 256 next 22
2020-09-30 09:30:30.139662: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1300 of size 256 next 24
2020-09-30 09:30:30.139831: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1400 of size 256 next 25
2020-09-30 09:30:30.140037: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1500 of size 256 next 26
2020-09-30 09:30:30.140226: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1600 of size 512 next 27
2020-09-30 09:30:30.140818: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1800 of size 512 next 30
2020-09-30 09:30:30.140960: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1a00 of size 512 next 31
2020-09-30 09:30:30.141156: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1c00 of size 512 next 32
2020-09-30 09:30:30.141348: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1e00 of size 256 next 33
2020-09-30 09:30:30.141520: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1f00 of size 256 next 35
2020-09-30 09:30:30.141689: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f2000 of size 256 next 36
2020-09-30 09:30:30.141905: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f2100 of size 256 next 37
2020-09-30 09:30:30.142100: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f2200 of size 512 next 3
2020-09-30 09:30:30.142307: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f2400 of size 3584 next 6
2020-09-30 09:30:30.142520: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3200 of size 512 next 38
2020-09-30 09:30:30.142712: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3400 of size 512 next 40
2020-09-30 09:30:30.143202: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3600 of size 512 next 41
2020-09-30 09:30:30.143304: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3800 of size 256 next 42
2020-09-30 09:30:30.143406: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3900 of size 256 next 44
2020-09-30 09:30:30.143546: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3a00 of size 256 next 45
2020-09-30 09:30:30.143690: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3b00 of size 256 next 46
2020-09-30 09:30:30.143821: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3c00 of size 512 next 47
2020-09-30 09:30:30.143951: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3e00 of size 512 next 49
2020-09-30 09:30:30.144085: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4000 of size 512 next 50
2020-09-30 09:30:30.144217: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4200 of size 512 next 51
2020-09-30 09:30:30.144391: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4400 of size 1024 next 53
2020-09-30 09:30:30.144539: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4800 of size 1024 next 56
2020-09-30 09:30:30.144672: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4c00 of size 1024 next 57
2020-09-30 09:30:30.144803: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5000 of size 1024 next 58
2020-09-30 09:30:30.144930: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5400 of size 512 next 59
2020-09-30 09:30:30.145057: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5600 of size 512 next 61
2020-09-30 09:30:30.145306: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5800 of size 512 next 63
2020-09-30 09:30:30.145399: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5a00 of size 512 next 64
2020-09-30 09:30:30.145471: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5c00 of size 1024 next 65
2020-09-30 09:30:30.145557: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6000 of size 1024 next 66
2020-09-30 09:30:30.145632: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6400 of size 1024 next 67
2020-09-30 09:30:30.145714: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6800 of size 1024 next 68
2020-09-30 09:30:30.145819: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6c00 of size 512 next 69
2020-09-30 09:30:30.145913: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6e00 of size 512 next 71
2020-09-30 09:30:30.146031: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f7000 of size 512 next 18
2020-09-30 09:30:30.146137: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f7200 of size 8192 next 16
2020-09-30 09:30:30.146278: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f9200 of size 32768 next 34
2020-09-30 09:30:30.146386: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00901200 of size 32768 next 43
2020-09-30 09:30:30.146486: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00909200 of size 512 next 72
2020-09-30 09:30:30.146587: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00909400 of size 1024 next 75
2020-09-30 09:30:30.146686: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00909800 of size 1024 next 76
2020-09-30 09:30:30.146783: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00909c00 of size 1024 next 77
2020-09-30 09:30:30.146920: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090a000 of size 1024 next 78
2020-09-30 09:30:30.147023: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090a400 of size 512 next 79
2020-09-30 09:30:30.147121: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090a600 of size 512 next 80
2020-09-30 09:30:30.147218: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090a800 of size 512 next 82
2020-09-30 09:30:30.147316: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090aa00 of size 512 next 83
2020-09-30 09:30:30.147488: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090ac00 of size 1024 next 85
2020-09-30 09:30:30.147549: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090b000 of size 1024 next 86
2020-09-30 09:30:30.147608: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090b400 of size 1024 next 87
2020-09-30 09:30:30.147667: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090b800 of size 1024 next 88
2020-09-30 09:30:30.147725: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090bc00 of size 512 next 90
2020-09-30 09:30:30.147784: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090be00 of size 512 next 91
2020-09-30 09:30:30.147848: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090c000 of size 512 next 92
2020-09-30 09:30:30.147916: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090c200 of size 512 next 93
2020-09-30 09:30:30.147988: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090c400 of size 1024 next 95
2020-09-30 09:30:30.148062: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090c800 of size 1024 next 97
2020-09-30 09:30:30.148141: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090cc00 of size 1024 next 98
2020-09-30 09:30:30.148219: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090d000 of size 1024 next 99
2020-09-30 09:30:30.148302: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090d400 of size 512 next 101
2020-09-30 09:30:30.148385: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090d600 of size 512 next 102
2020-09-30 09:30:30.148468: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090d800 of size 512 next 103
2020-09-30 09:30:30.148659: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090da00 of size 512 next 104
2020-09-30 09:30:30.148711: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090dc00 of size 1024 next 106
2020-09-30 09:30:30.148762: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090e000 of size 1024 next 107
2020-09-30 09:30:30.148813: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090e400 of size 1024 next 108
2020-09-30 09:30:30.148863: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090e800 of size 1024 next 109
2020-09-30 09:30:30.148916: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090ec00 of size 512 next 111
2020-09-30 09:30:30.148972: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090ee00 of size 512 next 112
2020-09-30 09:30:30.149032: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090f000 of size 512 next 113
2020-09-30 09:30:30.149093: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090f200 of size 512 next 114
2020-09-30 09:30:30.149158: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090f400 of size 1024 next 116
2020-09-30 09:30:30.149225: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090f800 of size 1024 next 117
2020-09-30 09:30:30.149294: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090fc00 of size 1024 next 118
2020-09-30 09:30:30.149364: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910000 of size 1024 next 119
2020-09-30 09:30:30.149436: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910400 of size 512 next 121
2020-09-30 09:30:30.149507: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910600 of size 512 next 122
2020-09-30 09:30:30.149581: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910800 of size 512 next 123
2020-09-30 09:30:30.149675: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910a00 of size 512 next 124
2020-09-30 09:30:30.149751: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910c00 of size 1024 next 125
2020-09-30 09:30:30.149825: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911000 of size 1024 next 126
2020-09-30 09:30:30.149897: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911400 of size 1024 next 127
2020-09-30 09:30:30.149991: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911800 of size 1024 next 128
2020-09-30 09:30:30.150068: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911c00 of size 512 next 130
2020-09-30 09:30:30.150153: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911e00 of size 512 next 131
2020-09-30 09:30:30.150226: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912000 of size 512 next 132
2020-09-30 09:30:30.150301: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912200 of size 512 next 133
2020-09-30 09:30:30.150373: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912400 of size 1024 next 136
2020-09-30 09:30:30.150444: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912800 of size 1024 next 137
2020-09-30 09:30:30.150515: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912c00 of size 1024 next 138
2020-09-30 09:30:30.150586: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00913000 of size 1024 next 139
2020-09-30 09:30:30.150660: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00913400 of size 2048 next 140
2020-09-30 09:30:30.150777: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00913c00 of size 2048 next 141
2020-09-30 09:30:30.150823: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00914400 of size 2048 next 142
2020-09-30 09:30:30.150881: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00914c00 of size 2048 next 144
2020-09-30 09:30:30.150925: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00915400 of size 1024 next 145
2020-09-30 09:30:30.150969: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00915800 of size 1024 next 146
2020-09-30 09:30:30.151013: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00915c00 of size 1024 next 148
2020-09-30 09:30:30.151056: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00916000 of size 1024 next 149
2020-09-30 09:30:30.151102: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00916400 of size 3584 next 12
2020-09-30 09:30:30.151150: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00917200 of size 73728 next 10
2020-09-30 09:30:30.151203: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00929200 of size 2048 next 150
2020-09-30 09:30:30.151255: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00929a00 of size 2048 next 153
2020-09-30 09:30:30.151309: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092a200 of size 2048 next 154
2020-09-30 09:30:30.151366: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092aa00 of size 1024 next 156
2020-09-30 09:30:30.151424: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092ae00 of size 1024 next 157
2020-09-30 09:30:30.151482: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092b200 of size 1024 next 158
2020-09-30 09:30:30.151542: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092b600 of size 1024 next 159
2020-09-30 09:30:30.151624: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092ba00 of size 2048 next 160
2020-09-30 09:30:30.151696: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092c200 of size 2048 next 162
2020-09-30 09:30:30.151759: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092ca00 of size 2048 next 163
2020-09-30 09:30:30.151876: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092d200 of size 2048 next 164
2020-09-30 09:30:30.151904: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092da00 of size 1024 next 165
2020-09-30 09:30:30.151927: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092de00 of size 1024 next 166
2020-09-30 09:30:30.151971: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092e200 of size 1024 next 167
2020-09-30 09:30:30.152003: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092e600 of size 1024 next 168
2020-09-30 09:30:30.152028: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092ea00 of size 2048 next 169
2020-09-30 09:30:30.152058: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092f200 of size 2048 next 171
2020-09-30 09:30:30.152084: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092fa00 of size 2048 next 172
2020-09-30 09:30:30.152109: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00930200 of size 2048 next 173
2020-09-30 09:30:30.152135: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00930a00 of size 1024 next 175
2020-09-30 09:30:30.152160: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00930e00 of size 1024 next 176
2020-09-30 09:30:30.152187: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00931200 of size 1024 next 177
2020-09-30 09:30:30.152216: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00931600 of size 1024 next 178
2020-09-30 09:30:30.152245: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00931a00 of size 2048 next 180
2020-09-30 09:30:30.152274: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00932200 of size 2048 next 182
2020-09-30 09:30:30.152305: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00932a00 of size 2048 next 183
2020-09-30 09:30:30.152353: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00933200 of size 2048 next 184
2020-09-30 09:30:30.152388: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00933a00 of size 1024 next 186
2020-09-30 09:30:30.152422: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00933e00 of size 1024 next 187
2020-09-30 09:30:30.152471: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00934200 of size 1024 next 188
2020-09-30 09:30:30.152509: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00934600 of size 1024 next 189
2020-09-30 09:30:30.152545: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00934a00 of size 2048 next 190
2020-09-30 09:30:30.152580: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00935200 of size 2048 next 191
2020-09-30 09:30:30.152618: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00935a00 of size 2048 next 193
2020-09-30 09:30:30.152656: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00936200 of size 2048 next 194
2020-09-30 09:30:30.152693: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00936a00 of size 1024 next 196
2020-09-30 09:30:30.152730: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00936e00 of size 1024 next 197
2020-09-30 09:30:30.152767: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00937200 of size 1024 next 198
2020-09-30 09:30:30.157772: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00937600 of size 1024 next 199
2020-09-30 09:30:30.157860: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00937a00 of size 2048 next 202
2020-09-30 09:30:30.157913: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00938200 of size 2048 next 203
2020-09-30 09:30:30.157963: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00938a00 of size 2048 next 204
2020-09-30 09:30:30.158012: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00939200 of size 2048 next 205
2020-09-30 09:30:30.158062: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00939a00 of size 1024 next 207
2020-09-30 09:30:30.158110: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00939e00 of size 1024 next 208
2020-09-30 09:30:30.158158: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0093a200 of size 1024 next 209
2020-09-30 09:30:30.158207: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0093a600 of size 1024 next 210
2020-09-30 09:30:30.158255: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0093aa00 of size 2048 next 23
2020-09-30 09:30:30.158304: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0093b200 of size 73728 next 21
2020-09-30 09:30:30.158352: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094d200 of size 2048 next 212
2020-09-30 09:30:30.158401: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094da00 of size 2048 next 213
2020-09-30 09:30:30.158449: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094e200 of size 2048 next 214
2020-09-30 09:30:30.158497: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094ea00 of size 1024 next 216
2020-09-30 09:30:30.158546: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094ee00 of size 1024 next 217
2020-09-30 09:30:30.158595: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094f200 of size 1024 next 218
2020-09-30 09:30:30.158644: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094f600 of size 1024 next 219
2020-09-30 09:30:30.158692: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094fa00 of size 2048 next 221
2020-09-30 09:30:30.158740: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00950200 of size 2048 next 222
2020-09-30 09:30:30.158788: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00950a00 of size 2048 next 223
2020-09-30 09:30:30.158837: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00951200 of size 2048 next 224
2020-09-30 09:30:30.159027: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00951a00 of size 4096 next 225
2020-09-30 09:30:30.159096: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00952a00 of size 4096 next 228
2020-09-30 09:30:30.159323: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00953a00 of size 4096 next 229
2020-09-30 09:30:30.159368: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00954a00 of size 4096 next 230
2020-09-30 09:30:30.159408: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00955a00 of size 2048 next 231
2020-09-30 09:30:30.159447: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00956200 of size 2048 next 232
2020-09-30 09:30:30.159486: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00956a00 of size 2048 next 234
2020-09-30 09:30:30.159526: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00957200 of size 2048 next 235
2020-09-30 09:30:30.159565: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00957a00 of size 4096 next 236
2020-09-30 09:30:30.159606: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00958a00 of size 4096 next 238
2020-09-30 09:30:30.159648: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00959a00 of size 4096 next 239
2020-09-30 09:30:30.159692: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095aa00 of size 4096 next 240
2020-09-30 09:30:30.159738: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095ba00 of size 2048 next 242
2020-09-30 09:30:30.159785: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095c200 of size 2048 next 243
2020-09-30 09:30:30.159834: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095ca00 of size 2048 next 244
2020-09-30 09:30:30.159885: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095d200 of size 2048 next 245
2020-09-30 09:30:30.159937: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095da00 of size 4096 next 246
2020-09-30 09:30:30.159989: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095ea00 of size 4096 next 248
2020-09-30 09:30:30.160044: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095fa00 of size 4096 next 249
2020-09-30 09:30:30.160101: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00960a00 of size 4096 next 250
2020-09-30 09:30:30.160157: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00961a00 of size 2048 next 252
2020-09-30 09:30:30.160215: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00962200 of size 2048 next 253
2020-09-30 09:30:30.160275: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00962a00 of size 2048 next 254
2020-09-30 09:30:30.160331: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00963200 of size 2048 next 255
2020-09-30 09:30:30.160388: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00963a00 of size 4096 next 256
2020-09-30 09:30:30.160447: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00964a00 of size 4096 next 258
2020-09-30 09:30:30.160622: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00965a00 of size 4096 next 259
2020-09-30 09:30:30.160657: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00966a00 of size 4096 next 260
2020-09-30 09:30:30.160692: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00967a00 of size 2048 next 262
2020-09-30 09:30:30.160726: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00968200 of size 2048 next 263
2020-09-30 09:30:30.160761: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00968a00 of size 2048 next 264
2020-09-30 09:30:30.160795: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00969200 of size 2048 next 265
2020-09-30 09:30:30.160830: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00969a00 of size 4096 next 266
2020-09-30 09:30:30.160866: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0096aa00 of size 4096 next 268
2020-09-30 09:30:30.160905: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0096ba00 of size 6144 next 81
2020-09-30 09:30:30.160952: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0096d200 of size 131072 next 62
2020-09-30 09:30:30.160996: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0098d200 of size 131072 next 60
2020-09-30 09:30:30.161040: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009ad200 of size 131072 next 70
2020-09-30 09:30:30.161084: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009cd200 of size 4096 next 269
2020-09-30 09:30:30.161128: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009ce200 of size 2048 next 270
2020-09-30 09:30:30.161174: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009cea00 of size 2048 next 271
2020-09-30 09:30:30.161221: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009cf200 of size 2048 next 272
2020-09-30 09:30:30.161269: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009cfa00 of size 2048 next 273
2020-09-30 09:30:30.161317: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d0200 of size 4096 next 274
2020-09-30 09:30:30.161371: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d1200 of size 4096 next 276
2020-09-30 09:30:30.161420: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d2200 of size 4096 next 277
2020-09-30 09:30:30.161462: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d3200 of size 4096 next 278
2020-09-30 09:30:30.161504: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d4200 of size 2048 next 280
2020-09-30 09:30:30.161546: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d4a00 of size 2048 next 281
2020-09-30 09:30:30.161588: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d5200 of size 2048 next 282
2020-09-30 09:30:30.161630: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d5a00 of size 2048 next 283
2020-09-30 09:30:30.161673: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d6200 of size 4096 next 284
2020-09-30 09:30:30.161713: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d7200 of size 4096 next 285
2020-09-30 09:30:30.161754: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d8200 of size 4096 next 287
2020-09-30 09:30:30.161795: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009d9200 of size 4096 next 288
2020-09-30 09:30:30.161836: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009da200 of size 2048 next 290
2020-09-30 09:30:30.161876: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009daa00 of size 2048 next 291
2020-09-30 09:30:30.161915: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009db200 of size 2048 next 292
2020-09-30 09:30:30.161955: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009dba00 of size 2048 next 293
2020-09-30 09:30:30.161996: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at f009dc200 of size 81408 next 18446744073709551615
2020-09-30 09:30:30.162036: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 2097152
2020-09-30 09:30:30.162079: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009f0000 of size 294912 next 29
2020-09-30 09:30:30.162122: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00a38000 of size 294912 next 39
2020-09-30 09:30:30.162162: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00a80000 of size 294912 next 48
2020-09-30 09:30:30.162203: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00ac8000 of size 524288 next 155
2020-09-30 09:30:30.162246: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00b48000 of size 688128 next 18446744073709551615
2020-09-30 09:30:30.162286: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 4194304
2020-09-30 09:30:30.162326: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00bf0000 of size 131072 next 89
2020-09-30 09:30:30.162367: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00c10000 of size 131072 next 100
2020-09-30 09:30:30.162407: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00c30000 of size 131072 next 110
2020-09-30 09:30:30.162447: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00c50000 of size 131072 next 120
2020-09-30 09:30:30.162488: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00c70000 of size 131072 next 129
2020-09-30 09:30:30.162528: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at f00c90000 of size 524288 next 55
2020-09-30 09:30:30.162568: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00d10000 of size 1179648 next 52
2020-09-30 09:30:30.162611: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00e30000 of size 1835008 next 18446744073709551615
2020-09-30 09:30:30.162650: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 8388608
2020-09-30 09:30:30.162690: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00ff0000 of size 1179648 next 74
2020-09-30 09:30:30.162732: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01110000 of size 1179648 next 84
2020-09-30 09:30:30.162772: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at f01230000 of size 524288 next 147
2020-09-30 09:30:30.162813: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f012b0000 of size 655360 next 96
2020-09-30 09:30:30.163042: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01350000 of size 1179648 next 94
2020-09-30 09:30:30.163072: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01470000 of size 1179648 next 105
2020-09-30 09:30:30.163090: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01590000 of size 1179648 next 115
2020-09-30 09:30:30.163109: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f016b0000 of size 1310720 next 18446744073709551615
2020-09-30 09:30:30.163126: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 16777216
2020-09-30 09:30:30.163144: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f017f0000 of size 1179648 next 135
2020-09-30 09:30:30.163162: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01910000 of size 524288 next 174
2020-09-30 09:30:30.163179: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01990000 of size 2097152 next 241
2020-09-30 09:30:30.163197: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01b90000 of size 2097152 next 251
2020-09-30 09:30:30.163214: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01d90000 of size 2097152 next 261
2020-09-30 09:30:30.163231: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at f01f90000 of size 2621440 next 143
2020-09-30 09:30:30.163249: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f02210000 of size 6160384 next 18446744073709551615
2020-09-30 09:30:30.163266: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 33554432
2020-09-30 09:30:30.163285: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f027f0000 of size 4718592 next 152
2020-09-30 09:30:30.163302: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f02c70000 of size 4718592 next 161
2020-09-30 09:30:30.163319: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f030f0000 of size 4718592 next 170
2020-09-30 09:30:30.163336: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f03570000 of size 524288 next 185
2020-09-30 09:30:30.163353: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f035f0000 of size 524288 next 195
2020-09-30 09:30:30.163371: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f03670000 of size 524288 next 206
2020-09-30 09:30:30.163388: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f036f0000 of size 524288 next 215
2020-09-30 09:30:30.163406: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f03770000 of size 2621440 next 181
2020-09-30 09:30:30.163425: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f039f0000 of size 4718592 next 179
2020-09-30 09:30:30.163444: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at f03e70000 of size 2097152 next 233
2020-09-30 09:30:30.163462: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f04070000 of size 2621440 next 192
2020-09-30 09:30:30.163481: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f042f0000 of size 5242880 next 18446744073709551615
2020-09-30 09:30:30.163498: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 67108864
2020-09-30 09:30:30.163517: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f047f0000 of size 4718592 next 201
2020-09-30 09:30:30.163536: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f04c70000 of size 4718592 next 211
2020-09-30 09:30:30.163554: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f050f0000 of size 4718592 next 220
2020-09-30 09:30:30.163574: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f05570000 of size 18874368 next 275
2020-09-30 09:30:30.163594: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f06770000 of size 2097152 next 279
2020-09-30 09:30:30.163615: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f06970000 of size 31981568 next 18446744073709551615
2020-09-30 09:30:30.163634: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 134217728
2020-09-30 09:30:30.163654: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f087f0000 of size 18874368 next 227
2020-09-30 09:30:30.163676: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f099f0000 of size 18874368 next 237
2020-09-30 09:30:30.163695: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0abf0000 of size 18874368 next 247
2020-09-30 09:30:30.163716: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0bdf0000 of size 18874368 next 257
2020-09-30 09:30:30.163737: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0cff0000 of size 18874368 next 267
2020-09-30 09:30:30.163758: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0e1f0000 of size 2097152 next 289
2020-09-30 09:30:30.163779: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at f0e3f0000 of size 16777216 next 286
2020-09-30 09:30:30.163817: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0f3f0000 of size 20971520 next 18446744073709551615
2020-09-30 09:30:30.163840: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 27963392
2020-09-30 09:30:30.163862: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f107f0000 of size 27963392 next 18446744073709551615
2020-09-30 09:30:30.163996: I tensorflow/core/common_runtime/bfc_allocator.cc:955]      Summary of in-use Chunks by size: 
2020-09-30 09:30:30.164033: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 25 Chunks of size 256 totalling 6.2KiB
2020-09-30 09:30:30.164060: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 44 Chunks of size 512 totalling 22.0KiB
2020-09-30 09:30:30.164083: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 68 Chunks of size 1024 totalling 68.0KiB
2020-09-30 09:30:30.164113: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 1280 totalling 1.2KiB
2020-09-30 09:30:30.164155: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 63 Chunks of size 2048 totalling 126.0KiB
2020-09-30 09:30:30.164191: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 3584 totalling 7.0KiB
2020-09-30 09:30:30.164218: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 27 Chunks of size 4096 totalling 108.0KiB
2020-09-30 09:30:30.164251: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 6144 totalling 6.0KiB
2020-09-30 09:30:30.164278: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 8192 totalling 8.0KiB
2020-09-30 09:30:30.164371: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 32768 totalling 64.0KiB
2020-09-30 09:30:30.164419: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 73728 totalling 144.0KiB
2020-09-30 09:30:30.164445: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 8 Chunks of size 131072 totalling 1.00MiB
2020-09-30 09:30:30.164471: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 3 Chunks of size 294912 totalling 864.0KiB
2020-09-30 09:30:30.164494: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 6 Chunks of size 524288 totalling 3.00MiB
2020-09-30 09:30:30.164525: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 655360 totalling 640.0KiB
2020-09-30 09:30:30.164565: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 688128 totalling 672.0KiB
2020-09-30 09:30:30.164600: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 7 Chunks of size 1179648 totalling 7.88MiB
2020-09-30 09:30:30.164625: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 1310720 totalling 1.25MiB
2020-09-30 09:30:30.164648: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 1835008 totalling 1.75MiB
2020-09-30 09:30:30.164672: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 5 Chunks of size 2097152 totalling 10.00MiB
2020-09-30 09:30:30.164695: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 2621440 totalling 5.00MiB
2020-09-30 09:30:30.164720: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 7 Chunks of size 4718592 totalling 31.50MiB
2020-09-30 09:30:30.164744: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 5242880 totalling 5.00MiB
2020-09-30 09:30:30.164768: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 6160384 totalling 5.88MiB
2020-09-30 09:30:30.164793: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 6 Chunks of size 18874368 totalling 108.00MiB
2020-09-30 09:30:30.164828: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 20971520 totalling 20.00MiB
2020-09-30 09:30:30.164854: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 27963392 totalling 26.67MiB
2020-09-30 09:30:30.164880: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 31981568 totalling 30.50MiB
2020-09-30 09:30:30.164924: I tensorflow/core/common_runtime/bfc_allocator.cc:962] Sum Total of in-use chunks: 260.09MiB
2020-09-30 09:30:30.164961: I tensorflow/core/common_runtime/bfc_allocator.cc:964] total_region_allocated_bytes_: 295350272 memory_limit_: 295350272 available bytes: 0 curr_region_allocation_bytes_: 536870912
2020-09-30 09:30:30.182837: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Stats: 
Limit:                   295350272
InUse:                   272724480
MaxInUse:                272724480
NumAllocs:                    1432
MaxAllocSize:             34078720

2020-09-30 09:30:30.182998: W tensorflow/core/common_runtime/bfc_allocator.cc:429] *****************************************xxxx*********************************_____**************xxx
2020-09-30 09:30:30.215689: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at cwise_ops_common.cc:82 : Resource exhausted: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
  File "app.py", line 43, in <module>
    yolo = YoloV3(classes=num_classes)
  File "/home/christopher/aiguyyolotest1/Object-Detection-API/yolov3_tf2/models.py", line 217, in YoloV3
    output_0 = YoloOutput(512, len(masks[0]), classes, name='yolo_output_0')(x)
  File "/home/christopher/aiguyyolotest1/Object-Detection-API/yolov3_tf2/models.py", line 146, in yolo_output
    x = DarknetConv(x, filters * 2, 3)
  File "/home/christopher/aiguyyolotest1/Object-Detection-API/yolov3_tf2/models.py", line 52, in DarknetConv
    use_bias=not batch_norm, kernel_regularizer=l2(0.0005))(x)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 748, in __call__
    self._maybe_build(inputs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 2116, in _maybe_build
    self.build(input_shapes)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/layers/convolutional.py", line 161, in build
    dtype=self.dtype)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 446, in add_weight
    caching_device=caching_device)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/tracking/base.py", line 744, in _add_variable_with_custom_getter
    **kwargs_for_getter)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer_utils.py", line 142, in make_variable
    shape=variable_shape if variable_shape else None)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py", line 258, in __call__
    return cls._variable_v1_call(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py", line 219, in _variable_v1_call
    shape=shape)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py", line 197, in <lambda>
    previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variable_scope.py", line 2596, in default_variable_creator
    shape=shape)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py", line 262, in __call__
    return super(VariableMetaclass, cls).__call__(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py", line 1411, in __init__
    distribute_strategy=distribute_strategy)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py", line 1542, in _init_from_args
    initial_value() if init_from_fn else initial_value,
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer_utils.py", line 122, in <lambda>
    init_val = lambda: initializer(shape, dtype=dtype)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops_v2.py", line 425, in __call__
    return self._random_generator.random_uniform(shape, -limit, limit, dtype)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops_v2.py", line 788, in random_uniform
    shape=shape, minval=minval, maxval=maxval, dtype=dtype, seed=self.seed)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/random_ops.py", line 273, in random_uniform
    result = math_ops.add(rnd * (maxval - minval), minval, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_math_ops.py", line 343, in add
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 6606, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Add] name: conv2d_57/kernel/Initializer/random_uniform/

Because I was not really sure if I have to run sudo tegrastats within the virtual environment or not, it did both.

I could not enter more text here. You will find the rest of my answer in the next post.

This is the continuation of my previous post.

Output of sudo tegrastats within the virtual environment

christopher@ccz:~/aiguyyolotest1/Object-Detection-API$ cd
christopher@ccz:~$ source aiguyyolotest1/bin/activate
(aiguyyolotest1) christopher@ccz:~$ cd aiguyyolotest1/Object-Detection-API/
(aiguyyolotest1) christopher@ccz:~/aiguyyolotest1/Object-Detection-API$ sudo tegrastats
RAM 1874/3963MB (lfb 108x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [11%@307,15%@307,12%@307,11%@307] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@33C PMIC@100C GPU@32.5C AO@42.5C thermal@33.5C POM_5V_IN 2228/2228 POM_5V_GPU 80/80 POM_5V_CPU 283/283
RAM 1874/3963MB (lfb 108x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [10%@204,10%@204,4%@204,4%@204] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@33C PMIC@100C GPU@32.5C AO@42C thermal@32.75C POM_5V_IN 2032/2130 POM_5V_GPU 40/60 POM_5V_CPU 121/202
RAM 1883/3963MB (lfb 108x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [38%@1479,14%@1479,18%@1479,24%@1479] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@33.5C CPU@34.5C PMIC@100C GPU@33C AO@43C thermal@33C POM_5V_IN 4238/2832 POM_5V_GPU 79/66 POM_5V_CPU 2135/846
RAM 1896/3963MB (lfb 108x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [35%@204,24%@204,33%@204,32%@204] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@33C PMIC@100C GPU@33C AO@42.5C thermal@33.75C POM_5V_IN 2467/2741 POM_5V_GPU 40/59 POM_5V_CPU 242/695
RAM 1909/3963MB (lfb 108x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [17%@1326,13%@1326,24%@1326,19%@1326] EMC_FREQ 2%@1600 GR3D_FREQ 1%@153 APE 25 PLL@33C CPU@33.5C PMIC@100C GPU@33C AO@42.5C thermal@33.25C POM_5V_IN 2974/2787 POM_5V_GPU 120/71 POM_5V_CPU 762/708
RAM 1919/3963MB (lfb 108x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [48%@1224,30%@1224,4%@1224,3%@1224] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@33C CPU@33.5C PMIC@100C GPU@32.5C AO@42.5C thermal@33.25C POM_5V_IN 2898/2806 POM_5V_GPU 40/66 POM_5V_CPU 845/731
RAM 1934/3963MB (lfb 108x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [15%@1479,33%@1479,42%@710,3%@710] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@33C CPU@34C PMIC@100C GPU@35C AO@42C thermal@33.25C POM_5V_IN 2898/2819 POM_5V_GPU 0/57 POM_5V_CPU 885/753
RAM 1945/3963MB (lfb 108x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [42%@710,4%@710,36%@710,8%@710] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@33C CPU@34C PMIC@100C GPU@33C AO@42.5C thermal@34.5C POM_5V_IN 2938/2834 POM_5V_GPU 40/54 POM_5V_CPU 805/759
RAM 1961/3963MB (lfb 108x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [42%@204,6%@204,6%@204,25%@204] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@33.5C PMIC@100C GPU@32.5C AO@42.5C thermal@33.5C POM_5V_IN 2624/2810 POM_5V_GPU 40/53 POM_5V_CPU 403/720
RAM 1974/3963MB (lfb 108x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [43%@1479,26%@1479,20%@1479,2%@1479] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@33C CPU@34C PMIC@100C GPU@33C AO@42.5C thermal@33C POM_5V_IN 3054/2835 POM_5V_GPU 40/51 POM_5V_CPU 922/740
RAM 1989/3963MB (lfb 108x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [41%@1479,8%@1479,23%@1479,18%@1479] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@33.5C PMIC@100C GPU@33C AO@42.5C thermal@33.75C POM_5V_IN 2978/2848 POM_5V_GPU 40/50 POM_5V_CPU 964/760
RAM 2003/3963MB (lfb 108x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [7%@1479,18%@1479,57%@1479,12%@1479] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@33C CPU@33.5C PMIC@100C GPU@33C AO@43C thermal@33.75C POM_5V_IN 3014/2861 POM_5V_GPU 40/49 POM_5V_CPU 924/774
RAM 2015/3963MB (lfb 108x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [20%@1479,16%@1479,41%@1479,2%@1479] EMC_FREQ 2%@1600 GR3D_FREQ 9%@153 APE 25 PLL@33C CPU@34C PMIC@100C GPU@33C AO@43C thermal@33.25C POM_5V_IN 3054/2876 POM_5V_GPU 80/52 POM_5V_CPU 922/785
RAM 2023/3963MB (lfb 107x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [3%@1479,2%@1479,3%@1479,74%@1479] EMC_FREQ 2%@1600 GR3D_FREQ 0%@76 APE 25 PLL@33C CPU@33.5C PMIC@100C GPU@35C AO@42.5C thermal@34.25C POM_5V_IN 2858/2875 POM_5V_GPU 0/48 POM_5V_CPU 925/795
RAM 2036/3963MB (lfb 107x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [10%@1479,1%@1479,1%@1479,77%@1479] EMC_FREQ 2%@1600 GR3D_FREQ 0%@76 APE 25 PLL@33C CPU@33.5C PMIC@100C GPU@35C AO@42.5C thermal@34.5C POM_5V_IN 2858/2874 POM_5V_GPU 0/45 POM_5V_CPU 1046/812
RAM 2053/3963MB (lfb 107x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [39%@1479,34%@1479,21%@1479,4%@1479] EMC_FREQ 2%@1600 GR3D_FREQ 0%@76 APE 25 PLL@33C CPU@34C PMIC@100C GPU@33C AO@43C thermal@33.5C POM_5V_IN 2898/2875 POM_5V_GPU 40/44 POM_5V_CPU 845/814
RAM 2066/3963MB (lfb 107x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [8%@1479,2%@1479,68%@1479,14%@1479] EMC_FREQ 2%@1600 GR3D_FREQ 0%@76 APE 25 PLL@33C CPU@34.5C PMIC@100C GPU@35C AO@42.5C thermal@33.5C POM_5V_IN 2938/2879 POM_5V_GPU 0/42 POM_5V_CPU 924/820
RAM 2080/3963MB (lfb 107x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [70%@1479,14%@1479,5%@1479,2%@1479] EMC_FREQ 2%@1600 GR3D_FREQ 0%@76 APE 25 PLL@33C CPU@34.5C PMIC@100C GPU@35C AO@43C thermal@34.75C POM_5V_IN 2898/2880 POM_5V_GPU 0/39 POM_5V_CPU 1046/833
RAM 2136/3963MB (lfb 107x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [18%@1479,14%@1479,30%@1479,23%@1479] EMC_FREQ 2%@1600 GR3D_FREQ 10%@76 APE 25 PLL@32.5C CPU@34C PMIC@100C GPU@33C AO@43C thermal@33.5C POM_5V_IN 2974/2885 POM_5V_GPU 120/44 POM_5V_CPU 723/827
RAM 2262/3963MB (lfb 107x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [20%@825,12%@825,31%@825,19%@825] EMC_FREQ 3%@1600 GR3D_FREQ 12%@76 APE 25 PLL@33C CPU@34C PMIC@100C GPU@33C AO@42.5C thermal@33.5C POM_5V_IN 3019/2892 POM_5V_GPU 120/47 POM_5V_CPU 683/820
RAM 2345/3963MB (lfb 107x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [17%@1479,14%@1479,17%@1479,48%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 30%@76 APE 25 PLL@33.5C CPU@34C PMIC@100C GPU@33C AO@42.5C thermal@33.5C POM_5V_IN 3210/2907 POM_5V_GPU 80/49 POM_5V_CPU 1083/832
RAM 2426/3963MB (lfb 86x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [36%@1479,21%@1479,15%@1479,25%@1479] EMC_FREQ 4%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@34C PMIC@100C GPU@33C AO@43C thermal@33.25C POM_5V_IN 2978/2910 POM_5V_GPU 80/50 POM_5V_CPU 924/836
RAM 2480/3963MB (lfb 69x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [18%@1479,19%@1479,12%@1479,45%@1479] EMC_FREQ 4%@1600 GR3D_FREQ 0%@153 APE 25 PLL@33.5C CPU@34C PMIC@100C GPU@33C AO@43C thermal@33.5C POM_5V_IN 2978/2913 POM_5V_GPU 40/50 POM_5V_CPU 1085/847
RAM 2480/3963MB (lfb 68x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [15%@1479,20%@1479,24%@1479,46%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@33C CPU@34.5C PMIC@100C GPU@33C AO@43.5C thermal@33.75C POM_5V_IN 3054/2919 POM_5V_GPU 40/49 POM_5V_CPU 1085/857
RAM 2483/3963MB (lfb 68x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [46%@1479,5%@1479,31%@1479,25%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@33C CPU@34C PMIC@100C GPU@33C AO@43C thermal@33.5C POM_5V_IN 2898/2918 POM_5V_GPU 40/49 POM_5V_CPU 966/861
RAM 2487/3963MB (lfb 68x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [21%@1479,41%@1479,36%@1479,8%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 7%@153 APE 25 PLL@33C CPU@34.5C PMIC@100C GPU@33.5C AO@43.5C thermal@33.75C POM_5V_IN 2978/2920 POM_5V_GPU 80/50 POM_5V_CPU 924/864
RAM 2489/3963MB (lfb 67x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [4%@1479,8%@1479,24%@1479,69%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 3%@153 APE 25 PLL@33C CPU@34.5C PMIC@100C GPU@33C AO@43C thermal@33.75C POM_5V_IN 2898/2919 POM_5V_GPU 40/50 POM_5V_CPU 966/868
RAM 2493/3963MB (lfb 66x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [25%@1479,21%@1479,35%@1479,24%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@33C CPU@34.5C PMIC@100C GPU@33C AO@43C thermal@34C POM_5V_IN 2898/2919 POM_5V_GPU 40/49 POM_5V_CPU 966/871
RAM 2497/3963MB (lfb 65x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [27%@1479,7%@1479,32%@1479,42%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 4%@153 APE 25 PLL@33C CPU@34.5C PMIC@100C GPU@33C AO@42.5C thermal@33.75C POM_5V_IN 2858/2916 POM_5V_GPU 80/51 POM_5V_CPU 885/872
RAM 2502/3963MB (lfb 64x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [19%@1479,6%@1479,21%@1479,60%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@33C CPU@35C PMIC@100C GPU@33C AO@42.5C thermal@33.75C POM_5V_IN 2898/2916 POM_5V_GPU 40/50 POM_5V_CPU 966/875
RAM 2505/3963MB (lfb 63x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [25%@1479,24%@1479,31%@1479,25%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@33.5C CPU@34C PMIC@100C GPU@33.5C AO@43C thermal@34C POM_5V_IN 2938/2916 POM_5V_GPU 80/51 POM_5V_CPU 1004/879
RAM 2510/3963MB (lfb 62x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [23%@1479,21%@1479,50%@1479,9%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@33C CPU@35C PMIC@100C GPU@33C AO@43C thermal@34.25C POM_5V_IN 2858/2915 POM_5V_GPU 40/51 POM_5V_CPU 925/880
RAM 2514/3963MB (lfb 61x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [56%@1479,6%@1479,13%@1479,26%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 56%@153 APE 25 PLL@33C CPU@34C PMIC@100C GPU@33C AO@43C thermal@33.75C POM_5V_IN 2701/2908 POM_5V_GPU 161/54 POM_5V_CPU 604/872
RAM 2514/3963MB (lfb 61x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [5%@102,4%@102,3%@102,4%@102] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@33.5C PMIC@100C GPU@34.5C AO@42.5C thermal@33.75C POM_5V_IN 1950/2880 POM_5V_GPU 0/52 POM_5V_CPU 162/851
RAM 2514/3963MB (lfb 61x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [7%@102,12%@102,3%@102,1%@102] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@33.5C PMIC@100C GPU@34.5C AO@42.5C thermal@34C POM_5V_IN 1950/2853 POM_5V_GPU 0/51 POM_5V_CPU 121/830
RAM 2514/3963MB (lfb 61x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [6%@102,9%@102,3%@102,1%@102] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@34C PMIC@100C GPU@34.5C AO@43C thermal@33.75C POM_5V_IN 1950/2828 POM_5V_GPU 0/50 POM_5V_CPU 121/810
RAM 2514/3963MB (lfb 61x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [4%@102,8%@102,3%@102,6%@102] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@33.5C PMIC@100C GPU@34.5C AO@42.5C thermal@34C POM_5V_IN 1950/2805 POM_5V_GPU 0/48 POM_5V_CPU 121/792
RAM 2514/3963MB (lfb 61x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [7%@102,8%@102,4%@102,1%@102] EMC_FREQ 1%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@34C PMIC@100C GPU@34.5C AO@43C thermal@34C POM_5V_IN 1950/2782 POM_5V_GPU 0/47 POM_5V_CPU 121/774
RAM 2514/3963MB (lfb 61x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [7%@102,9%@204,2%@204,2%@204] EMC_FREQ 1%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@33.5C PMIC@100C GPU@34.5C AO@42.5C thermal@33.25C POM_5V_IN 1950/2761 POM_5V_GPU 0/46 POM_5V_CPU 121/757
RAM 2515/3963MB (lfb 61x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [11%@102,9%@102,6%@204,5%@204] EMC_FREQ 1%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@34C PMIC@100C GPU@34.5C AO@42.5C thermal@33.25C POM_5V_IN 1991/2741 POM_5V_GPU 0/45 POM_5V_CPU 162/742
RAM 2515/3963MB (lfb 61x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [6%@102,8%@102,3%@102,2%@102] EMC_FREQ 1%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@33.5C PMIC@100C GPU@34.5C AO@42.5C thermal@33.25C POM_5V_IN 1950/2722 POM_5V_GPU 0/43 POM_5V_CPU 121/727
RAM 2515/3963MB (lfb 61x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [9%@102,5%@102,5%@102,3%@102] EMC_FREQ 1%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@33.5C PMIC@100C GPU@34.5C AO@42.5C thermal@33.25C POM_5V_IN 1950/2704 POM_5V_GPU 0/42 POM_5V_CPU 121/713
RAM 2516/3963MB (lfb 60x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [10%@1479,10%@1479,7%@1479,6%@1479] EMC_FREQ 1%@1600 GR3D_FREQ 0%@153 APE 25 PLL@33C CPU@34C PMIC@100C GPU@33C AO@43C thermal@33.5C POM_5V_IN 2893/2708 POM_5V_GPU 80/43 POM_5V_CPU 803/715
RAM 2516/3963MB (lfb 59x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [99%@1479,5%@1479,2%@1479,1%@1479] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@33C CPU@34C PMIC@100C GPU@35C AO@42.5C thermal@33.5C POM_5V_IN 2822/2711 POM_5V_GPU 40/43 POM_5V_CPU 805/717
RAM 2502/3963MB (lfb 59x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [85%@614,2%@614,1%@614,5%@614] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@33C CPU@34C PMIC@100C GPU@33C AO@43C thermal@33.5C POM_5V_IN 2665/2710 POM_5V_GPU 40/43 POM_5V_CPU 444/711
RAM 1803/3963MB (lfb 73x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [4%@102,19%@102,12%@102,6%@204] EMC_FREQ 4%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@33.5C PMIC@100C GPU@33C AO@43C thermal@33.25C POM_5V_IN 2032/2695 POM_5V_GPU 40/43 POM_5V_CPU 121/698
RAM 1803/3963MB (lfb 73x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [8%@102,7%@102,3%@204,3%@204] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@34C PMIC@100C GPU@33C AO@43C thermal@33.25C POM_5V_IN 2032/2681 POM_5V_GPU 40/43 POM_5V_CPU 162/687
RAM 1803/3963MB (lfb 73x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [7%@102,9%@204,5%@204,4%@204] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@33.5C PMIC@100C GPU@33C AO@43C thermal@33.25C POM_5V_IN 2032/2667 POM_5V_GPU 40/43 POM_5V_CPU 121/675
RAM 1803/3963MB (lfb 73x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [16%@614,13%@614,11%@614,18%@614] EMC_FREQ 2%@1600 GR3D_FREQ 9%@153 APE 25 PLL@32.5C CPU@34C PMIC@100C GPU@33C AO@43C thermal@33.25C POM_5V_IN 2224/2658 POM_5V_GPU 80/44 POM_5V_CPU 323/668
RAM 1804/3963MB (lfb 73x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [26%@204,11%@204,14%@204,13%@204] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@33.5C PMIC@100C GPU@33C AO@42.5C thermal@33.5C POM_5V_IN 2028/2646 POM_5V_GPU 40/44 POM_5V_CPU 162/658
RAM 1804/3963MB (lfb 73x4MB) SWAP 89/1982MB (cached 1MB) IRAM 0/252kB(lfb 252kB) CPU [15%@102,8%@102,9%@102,12%@204] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@33.5C PMIC@100C GPU@33C AO@42.5C thermal@33.25C POM_5V_IN 1988/2633 POM_5V_GPU 40/43 POM_5V_CPU 121/647
^C
(aiguyyolotest1) christopher@ccz:~/aiguyyolotest1/Object-Detection-API$ 

Output of sudo tegrastats NOT within the virtual environment

christopher@ccz:~/aiguyyolotest1/Object-Detection-API$ sudo tegrastats
[sudo] password for christopher: 
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [6%@102,7%@204,2%@204,3%@204] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@31C CPU@32C PMIC@100C GPU@31.5C AO@41.5C thermal@32C POM_5V_IN 1991/1991 POM_5V_GPU 40/40 POM_5V_CPU 121/121
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [9%@102,11%@102,4%@102,3%@102] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@31C CPU@32C PMIC@100C GPU@31.5C AO@41C thermal@31.75C POM_5V_IN 1991/1991 POM_5V_GPU 40/40 POM_5V_CPU 121/121
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [9%@204,6%@204,4%@204,1%@204] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@31C CPU@32C PMIC@100C GPU@31C AO@41C thermal@31.75C POM_5V_IN 2109/2030 POM_5V_GPU 81/53 POM_5V_CPU 162/134
RAM 1501/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [9%@204,9%@204,2%@204,5%@204] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@31C CPU@32C PMIC@100C GPU@31.5C AO@41C thermal@31.5C POM_5V_IN 1991/2020 POM_5V_GPU 40/50 POM_5V_CPU 121/131
RAM 1501/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [14%@102,2%@102,3%@102,5%@102] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@31C CPU@32C PMIC@100C GPU@31.5C AO@40.5C thermal@32C POM_5V_IN 1991/2014 POM_5V_GPU 40/48 POM_5V_CPU 121/129
RAM 1501/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [12%@102,7%@102,5%@102,4%@102] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@31C CPU@32C PMIC@100C GPU@31.5C AO@41C thermal@31.75C POM_5V_IN 2069/2023 POM_5V_GPU 81/53 POM_5V_CPU 162/134
RAM 1501/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [14%@102,4%@102,6%@102,4%@204] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@31C CPU@32C PMIC@100C GPU@31C AO@41C thermal@31.5C POM_5V_IN 1991/2019 POM_5V_GPU 40/51 POM_5V_CPU 121/132
RAM 1501/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [12%@102,7%@102,4%@102,3%@102] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@31C CPU@32C PMIC@100C GPU@31.5C AO@41C thermal@31.75C POM_5V_IN 1991/2015 POM_5V_GPU 40/50 POM_5V_CPU 121/131
RAM 1501/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [15%@102,4%@102,3%@102,3%@102] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@31C CPU@32C PMIC@100C GPU@31.5C AO@41C thermal@31.5C POM_5V_IN 2032/2017 POM_5V_GPU 40/49 POM_5V_CPU 162/134
RAM 1501/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [13%@204,6%@204,3%@204,3%@102] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@31C CPU@31.5C PMIC@100C GPU@33C AO@41.5C thermal@31.75C POM_5V_IN 1991/2014 POM_5V_GPU 0/44 POM_5V_CPU 162/137
RAM 1501/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [29%@403,25%@403,21%@403,18%@403] EMC_FREQ 1%@1600 GR3D_FREQ 18%@76 APE 25 PLL@31C CPU@32C PMIC@100C GPU@31.5C AO@41.5C thermal@32C POM_5V_IN 2423/2051 POM_5V_GPU 121/51 POM_5V_CPU 403/161
RAM 1501/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [25%@102,12%@102,16%@102,14%@102] EMC_FREQ 2%@1600 GR3D_FREQ 0%@76 APE 25 PLL@31C CPU@32C PMIC@100C GPU@31.5C AO@41C thermal@32.25C POM_5V_IN 2069/2053 POM_5V_GPU 40/50 POM_5V_CPU 162/161
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [30%@825,13%@825,14%@825,13%@825] EMC_FREQ 2%@1600 GR3D_FREQ 47%@76 APE 25 PLL@31.5C CPU@32.5C PMIC@100C GPU@31.5C AO@41C thermal@32C POM_5V_IN 2544/2091 POM_5V_GPU 120/55 POM_5V_CPU 483/186
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [23%@307,18%@307,12%@307,17%@307] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31.5C CPU@32C PMIC@100C GPU@32C AO@41C thermal@32C POM_5V_IN 2224/2100 POM_5V_GPU 40/54 POM_5V_CPU 323/196
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [28%@307,19%@403,12%@403,13%@403] EMC_FREQ 2%@1600 GR3D_FREQ 2%@153 APE 25 PLL@31C CPU@32.5C PMIC@100C GPU@32C AO@41C thermal@31.75C POM_5V_IN 2503/2127 POM_5V_GPU 80/56 POM_5V_CPU 322/204
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [21%@102,11%@102,10%@102,8%@204] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31.5C CPU@31.5C PMIC@100C GPU@31.5C AO@41C thermal@31.75C POM_5V_IN 2028/2121 POM_5V_GPU 40/55 POM_5V_CPU 162/201
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [20%@204,18%@204,18%@204,14%@204] EMC_FREQ 2%@1600 GR3D_FREQ 2%@153 APE 25 PLL@31C CPU@32C PMIC@100C GPU@31.5C AO@41.5C thermal@32.25C POM_5V_IN 2228/2127 POM_5V_GPU 81/56 POM_5V_CPU 283/206
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [11%@204,9%@204,4%@204,3%@204] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31C CPU@32C PMIC@100C GPU@31.5C AO@41C thermal@32C POM_5V_IN 2032/2122 POM_5V_GPU 40/55 POM_5V_CPU 121/201
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [24%@710,12%@710,10%@825,9%@825] EMC_FREQ 2%@1600 GR3D_FREQ 16%@153 APE 25 PLL@31.5C CPU@33C PMIC@100C GPU@31.5C AO@41.5C thermal@31.75C POM_5V_IN 2265/2129 POM_5V_GPU 80/57 POM_5V_CPU 323/208
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [11%@102,2%@102,5%@102,6%@102] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31.5C CPU@32C PMIC@100C GPU@31.5C AO@41.5C thermal@31.75C POM_5V_IN 1988/2122 POM_5V_GPU 40/56 POM_5V_CPU 121/203
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [15%@102,6%@102,5%@102,4%@102] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31C CPU@32C PMIC@100C GPU@31.5C AO@41C thermal@31.75C POM_5V_IN 2032/2118 POM_5V_GPU 40/55 POM_5V_CPU 121/199
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [19%@102,9%@102,8%@102,6%@102] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31C CPU@32C PMIC@100C GPU@31.5C AO@41.5C thermal@31.75C POM_5V_IN 2028/2114 POM_5V_GPU 40/54 POM_5V_CPU 162/198
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [14%@102,5%@204,4%@204,4%@204] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31C CPU@32C PMIC@100C GPU@31.5C AO@41C thermal@31.75C POM_5V_IN 2150/2115 POM_5V_GPU 40/54 POM_5V_CPU 162/196
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [9%@102,9%@102,4%@102,2%@102] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31.5C CPU@32C PMIC@100C GPU@31.5C AO@41.5C thermal@31.75C POM_5V_IN 2032/2112 POM_5V_GPU 40/53 POM_5V_CPU 121/193
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [7%@204,10%@204,8%@204,4%@204] EMC_FREQ 1%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31C CPU@32C PMIC@100C GPU@31.5C AO@41C thermal@31.75C POM_5V_IN 1991/2107 POM_5V_GPU 40/52 POM_5V_CPU 202/193
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [6%@102,5%@204,3%@204,4%@204] EMC_FREQ 1%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31C CPU@32C PMIC@100C GPU@31.5C AO@41C thermal@32C POM_5V_IN 1991/2102 POM_5V_GPU 40/52 POM_5V_CPU 121/191
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [8%@102,10%@102,4%@102,4%@102] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@31C CPU@32C PMIC@100C GPU@31.5C AO@41C thermal@31.75C POM_5V_IN 1991/2098 POM_5V_GPU 40/52 POM_5V_CPU 121/188
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [8%@102,8%@102,3%@102,3%@102] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@31C CPU@32C PMIC@100C GPU@33C AO@40.5C thermal@31.75C POM_5V_IN 1991/2094 POM_5V_GPU 0/50 POM_5V_CPU 121/186
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [14%@403,13%@403,15%@403,9%@403] EMC_FREQ 1%@1600 GR3D_FREQ 13%@76 APE 25 PLL@31.5C CPU@32C PMIC@100C GPU@31.5C AO@41C thermal@31.75C POM_5V_IN 2463/2107 POM_5V_GPU 121/52 POM_5V_CPU 403/193
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [14%@204,11%@204,8%@204,10%@204] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@31C CPU@32C PMIC@100C GPU@33C AO@41.5C thermal@31.75C POM_5V_IN 1947/2102 POM_5V_GPU 0/50 POM_5V_CPU 121/191
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [10%@403,12%@403,9%@403,6%@403] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@31C CPU@32C PMIC@100C GPU@31.5C AO@41C thermal@31.75C POM_5V_IN 2032/2099 POM_5V_GPU 40/50 POM_5V_CPU 162/190
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [14%@204,13%@204,6%@204,6%@204] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@31C CPU@32C PMIC@100C GPU@31.5C AO@41C thermal@31.75C POM_5V_IN 1991/2096 POM_5V_GPU 40/50 POM_5V_CPU 121/187
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [10%@204,13%@204,9%@204,8%@204] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@31C CPU@32.5C PMIC@100C GPU@31.5C AO@41C thermal@31.75C POM_5V_IN 2032/2094 POM_5V_GPU 40/49 POM_5V_CPU 162/187
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [7%@204,10%@204,5%@204,4%@204] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@31C CPU@32C PMIC@100C GPU@32C AO@41C thermal@31.75C POM_5V_IN 2032/2092 POM_5V_GPU 40/49 POM_5V_CPU 121/185
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [12%@204,15%@204,12%@204,9%@204] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@31C CPU@32.5C PMIC@100C GPU@32C AO@41.5C thermal@32C POM_5V_IN 2032/2091 POM_5V_GPU 40/49 POM_5V_CPU 162/184
RAM 1500/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [9%@403,11%@403,11%@403,5%@403] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@31C CPU@32C PMIC@100C GPU@31.5C AO@41.5C thermal@32C POM_5V_IN 2150/2092 POM_5V_GPU 81/50 POM_5V_CPU 243/186
RAM 1507/3963MB (lfb 282x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [20%@307,15%@307,21%@307,10%@307] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@31C CPU@32C PMIC@100C GPU@32C AO@41C thermal@31.75C POM_5V_IN 2548/2104 POM_5V_GPU 40/49 POM_5V_CPU 282/188
RAM 1516/3963MB (lfb 281x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [33%@403,34%@403,40%@403,29%@403] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@31.5C CPU@32.5C PMIC@100C GPU@31.5C AO@41.5C thermal@33C POM_5V_IN 2544/2116 POM_5V_GPU 40/49 POM_5V_CPU 363/193
RAM 1526/3963MB (lfb 281x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [39%@1479,6%@1479,9%@1479,5%@1479] EMC_FREQ 1%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31.5C CPU@32.5C PMIC@100C GPU@31.5C AO@41.5C thermal@32.25C POM_5V_IN 2938/2137 POM_5V_GPU 80/50 POM_5V_CPU 602/203
RAM 1527/3963MB (lfb 274x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [15%@204,23%@307,9%@307,5%@307] EMC_FREQ 1%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31C CPU@32C PMIC@100C GPU@31.5C AO@41.5C thermal@32C POM_5V_IN 2705/2151 POM_5V_GPU 80/51 POM_5V_CPU 322/206
RAM 1541/3963MB (lfb 257x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [9%@403,11%@403,19%@403,16%@403] EMC_FREQ 1%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31.5C CPU@32.5C PMIC@100C GPU@31.5C AO@41.5C thermal@32C POM_5V_IN 2507/2160 POM_5V_GPU 40/50 POM_5V_CPU 282/208
RAM 1549/3963MB (lfb 254x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [18%@1479,15%@1479,46%@1479,13%@1479] EMC_FREQ 1%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31.5C CPU@33C PMIC@100C GPU@32C AO@42C thermal@32.5C POM_5V_IN 3094/2182 POM_5V_GPU 120/52 POM_5V_CPU 922/225
RAM 1554/3963MB (lfb 253x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [14%@1036,11%@1036,14%@1036,17%@1036] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31.5C CPU@32.5C PMIC@100C GPU@32C AO@41.5C thermal@32C POM_5V_IN 2858/2198 POM_5V_GPU 40/52 POM_5V_CPU 684/236
RAM 1564/3963MB (lfb 249x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [24%@1479,5%@1479,32%@1479,16%@1479] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31.5C CPU@33C PMIC@100C GPU@32.5C AO@41.5C thermal@32.75C POM_5V_IN 2978/2216 POM_5V_GPU 40/51 POM_5V_CPU 924/251
RAM 1577/3963MB (lfb 244x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [34%@1479,7%@1479,2%@1479,50%@1479] EMC_FREQ 2%@1600 GR3D_FREQ 1%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32C AO@41.5C thermal@32.25C POM_5V_IN 2978/2233 POM_5V_GPU 80/52 POM_5V_CPU 844/265
RAM 1591/3963MB (lfb 233x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [9%@307,3%@307,5%@307,53%@307] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31.5C CPU@32.5C PMIC@100C GPU@32C AO@41.5C thermal@32.25C POM_5V_IN 2584/2240 POM_5V_GPU 40/52 POM_5V_CPU 282/265
RAM 1601/3963MB (lfb 223x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [28%@1479,22%@1479,14%@1479,9%@1479] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31.5C CPU@33C PMIC@100C GPU@32C AO@41.5C thermal@32.25C POM_5V_IN 2898/2254 POM_5V_GPU 40/52 POM_5V_CPU 885/278
RAM 1609/3963MB (lfb 214x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [46%@204,7%@204,4%@307,16%@307] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31.5C CPU@32.5C PMIC@100C GPU@33.5C AO@41.5C thermal@32.5C POM_5V_IN 2426/2258 POM_5V_GPU 0/50 POM_5V_CPU 323/279
RAM 1618/3963MB (lfb 201x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [26%@1479,19%@1479,25%@1479,7%@1479] EMC_FREQ 2%@1600 GR3D_FREQ 13%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32C AO@41.5C thermal@32.25C POM_5V_IN 3285/2279 POM_5V_GPU 80/51 POM_5V_CPU 1080/295
RAM 1629/3963MB (lfb 194x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [30%@1479,26%@1479,22%@1479,12%@1479] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32C AO@41.5C thermal@32.25C POM_5V_IN 2934/2292 POM_5V_GPU 40/51 POM_5V_CPU 844/306
RAM 1640/3963MB (lfb 190x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [65%@1479,20%@1479,11%@1479,27%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 19%@153 APE 25 PLL@32.5C CPU@33C PMIC@100C GPU@32.5C AO@41.5C thermal@32.25C POM_5V_IN 3781/2321 POM_5V_GPU 199/54 POM_5V_CPU 1353/327
RAM 1651/3963MB (lfb 183x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [32%@1036,21%@1036,34%@1036,22%@1036] EMC_FREQ 4%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32.5C AO@41.5C thermal@32.75C POM_5V_IN 2817/2330 POM_5V_GPU 120/55 POM_5V_CPU 602/332
RAM 1652/3963MB (lfb 183x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [21%@204,11%@204,14%@204,11%@204] EMC_FREQ 4%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31.5C CPU@32.5C PMIC@100C GPU@32C AO@41.5C thermal@32.25C POM_5V_IN 2265/2329 POM_5V_GPU 40/55 POM_5V_CPU 161/329
RAM 1652/3963MB (lfb 183x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [19%@518,15%@518,14%@518,11%@518] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31.5C CPU@32.5C PMIC@100C GPU@32C AO@41.5C thermal@32.25C POM_5V_IN 2305/2329 POM_5V_GPU 40/54 POM_5V_CPU 242/327
RAM 1658/3963MB (lfb 180x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [12%@1479,5%@1479,25%@1479,48%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33.5C PMIC@100C GPU@32C AO@41.5C thermal@33C POM_5V_IN 2858/2338 POM_5V_GPU 80/55 POM_5V_CPU 763/335
RAM 1667/3963MB (lfb 171x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [12%@1428,9%@1428,42%@1428,8%@1428] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32C AO@41.5C thermal@33C POM_5V_IN 2817/2347 POM_5V_GPU 40/55 POM_5V_CPU 683/341
RAM 1682/3963MB (lfb 164x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [28%@1479,5%@1479,44%@1479,10%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33.5C PMIC@100C GPU@32.5C AO@42C thermal@32.75C POM_5V_IN 2858/2356 POM_5V_GPU 40/54 POM_5V_CPU 724/348
RAM 1692/3963MB (lfb 154x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [14%@1479,3%@1479,47%@1479,10%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33.5C PMIC@100C GPU@32C AO@41.5C thermal@32.75C POM_5V_IN 2978/2367 POM_5V_GPU 80/55 POM_5V_CPU 844/357
RAM 1701/3963MB (lfb 151x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [36%@1479,4%@1479,34%@1479,2%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33.5C PMIC@100C GPU@32C AO@42C thermal@32.5C POM_5V_IN 2938/2376 POM_5V_GPU 40/55 POM_5V_CPU 844/365
RAM 1713/3963MB (lfb 146x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [43%@1479,5%@1479,24%@1479,21%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32.5C AO@42C thermal@32.5C POM_5V_IN 2938/2386 POM_5V_GPU 40/54 POM_5V_CPU 764/372
RAM 1728/3963MB (lfb 133x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [23%@921,21%@921,30%@921,16%@921] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32.5C AO@41.5C thermal@32.5C POM_5V_IN 3014/2396 POM_5V_GPU 160/56 POM_5V_CPU 763/378
RAM 1754/3963MB (lfb 116x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [17%@1479,8%@1479,12%@1479,32%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32C AO@41.5C thermal@32.25C POM_5V_IN 3560/2415 POM_5V_GPU 39/56 POM_5V_CPU 1238/392
RAM 1786/3963MB (lfb 115x4MB) SWAP 2/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [25%@204,13%@204,20%@204,26%@204] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31.5C CPU@33C PMIC@100C GPU@32C AO@41.5C thermal@32.5C POM_5V_IN 2507/2416 POM_5V_GPU 40/55 POM_5V_CPU 282/390
RAM 1836/3963MB (lfb 113x4MB) SWAP 2/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [23%@1479,10%@1479,17%@1479,10%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 47%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@31.5C AO@41.5C thermal@32.25C POM_5V_IN 2898/2424 POM_5V_GPU 80/56 POM_5V_CPU 805/397
RAM 1872/3963MB (lfb 113x4MB) SWAP 2/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [15%@204,12%@204,18%@204,16%@204] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31.5C CPU@33C PMIC@100C GPU@32C AO@41.5C thermal@32.25C POM_5V_IN 2503/2425 POM_5V_GPU 40/56 POM_5V_CPU 322/395
RAM 1944/3963MB (lfb 112x4MB) SWAP 2/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [16%@1224,9%@1224,23%@1224,18%@1224] EMC_FREQ 3%@1600 GR3D_FREQ 6%@153 APE 25 PLL@31.5C CPU@33C PMIC@100C GPU@32C AO@41.5C thermal@32.25C POM_5V_IN 2741/2430 POM_5V_GPU 40/55 POM_5V_CPU 524/397
RAM 1990/3963MB (lfb 110x4MB) SWAP 8/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [11%@307,13%@307,21%@614,23%@614] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32C AO@42C thermal@32.5C POM_5V_IN 2741/2434 POM_5V_GPU 40/55 POM_5V_CPU 483/399
RAM 2043/3963MB (lfb 108x4MB) SWAP 11/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [22%@307,13%@307,17%@307,21%@307] EMC_FREQ 3%@1600 GR3D_FREQ 14%@153 APE 25 PLL@31.5C CPU@32.5C PMIC@100C GPU@32.5C AO@42C thermal@32.75C POM_5V_IN 2503/2435 POM_5V_GPU 40/55 POM_5V_CPU 322/397
RAM 2083/3963MB (lfb 107x4MB) SWAP 17/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [11%@1479,11%@1479,18%@1479,18%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@32.5C PMIC@100C GPU@32C AO@41.5C thermal@32.5C POM_5V_IN 2624/2438 POM_5V_GPU 40/55 POM_5V_CPU 443/398
RAM 2126/3963MB (lfb 107x4MB) SWAP 18/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [18%@1036,17%@1036,17%@1036,15%@1036] EMC_FREQ 3%@1600 GR3D_FREQ 21%@153 APE 25 PLL@32C CPU@33.5C PMIC@100C GPU@32C AO@41.5C thermal@32.75C POM_5V_IN 2782/2443 POM_5V_GPU 80/55 POM_5V_CPU 604/401
RAM 2210/3963MB (lfb 107x4MB) SWAP 20/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [31%@825,12%@825,15%@825,24%@825] EMC_FREQ 3%@1600 GR3D_FREQ 9%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32.5C AO@42C thermal@32.75C POM_5V_IN 2938/2450 POM_5V_GPU 80/55 POM_5V_CPU 724/406
RAM 2284/3963MB (lfb 107x4MB) SWAP 25/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [28%@825,9%@825,33%@825,19%@825] EMC_FREQ 3%@1600 GR3D_FREQ 2%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32C AO@41.5C thermal@32.75C POM_5V_IN 2898/2456 POM_5V_GPU 40/55 POM_5V_CPU 684/409
RAM 2294/3963MB (lfb 107x4MB) SWAP 27/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [25%@204,12%@204,12%@204,21%@204] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31.5C CPU@33C PMIC@100C GPU@32C AO@42C thermal@32.5C POM_5V_IN 2349/2455 POM_5V_GPU 40/55 POM_5V_CPU 202/407
RAM 2298/3963MB (lfb 107x4MB) SWAP 27/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [47%@1479,37%@1479,14%@1479,7%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32.5C AO@41.5C thermal@32.75C POM_5V_IN 2898/2461 POM_5V_GPU 80/55 POM_5V_CPU 964/414
RAM 2302/3963MB (lfb 107x4MB) SWAP 27/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [21%@1479,44%@1479,19%@1479,21%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33.5C PMIC@100C GPU@32.5C AO@42C thermal@33C POM_5V_IN 2858/2466 POM_5V_GPU 40/55 POM_5V_CPU 925/421
RAM 2309/3963MB (lfb 107x4MB) SWAP 27/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [9%@1479,46%@1479,20%@1479,29%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@33C PMIC@100C GPU@32C AO@42C thermal@33C POM_5V_IN 2858/2471 POM_5V_GPU 40/55 POM_5V_CPU 925/428
RAM 2319/3963MB (lfb 107x4MB) SWAP 28/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [32%@1479,7%@1479,53%@1479,16%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33.5C PMIC@100C GPU@32.5C AO@42C thermal@32.5C POM_5V_IN 2898/2477 POM_5V_GPU 40/55 POM_5V_CPU 966/435
RAM 2322/3963MB (lfb 107x4MB) SWAP 28/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [34%@1428,20%@1428,23%@1428,29%@1428] EMC_FREQ 3%@1600 GR3D_FREQ 13%@153 APE 25 PLL@32C CPU@33.5C PMIC@100C GPU@32.5C AO@42C thermal@33C POM_5V_IN 2782/2481 POM_5V_GPU 80/55 POM_5V_CPU 845/440
RAM 2326/3963MB (lfb 107x4MB) SWAP 29/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [40%@1479,40%@1479,5%@1479,26%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 6%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32C AO@42C thermal@33C POM_5V_IN 2938/2486 POM_5V_GPU 40/55 POM_5V_CPU 1004/447
RAM 2379/3963MB (lfb 107x4MB) SWAP 33/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [54%@1479,14%@1479,24%@1479,28%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 11%@153 APE 25 PLL@32.5C CPU@34C PMIC@100C GPU@32.5C AO@42C thermal@33C POM_5V_IN 3480/2499 POM_5V_GPU 80/55 POM_5V_CPU 1400/459
RAM 2382/3963MB (lfb 107x4MB) SWAP 33/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [37%@1479,43%@1479,6%@1479,19%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@33C PMIC@100C GPU@32.5C AO@42C thermal@33C POM_5V_IN 2858/2503 POM_5V_GPU 40/55 POM_5V_CPU 966/465
RAM 2385/3963MB (lfb 107x4MB) SWAP 33/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [9%@1479,46%@1479,40%@1479,11%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 17%@153 APE 25 PLL@32C CPU@33.5C PMIC@100C GPU@32.5C AO@42C thermal@33.25C POM_5V_IN 3014/2509 POM_5V_GPU 80/55 POM_5V_CPU 964/471
RAM 2448/3963MB (lfb 107x4MB) SWAP 46/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [36%@1479,52%@1479,28%@1479,38%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@33.5C PMIC@100C GPU@32.5C AO@42C thermal@32.75C POM_5V_IN 2898/2514 POM_5V_GPU 40/55 POM_5V_CPU 925/477
RAM 2570/3963MB (lfb 107x4MB) SWAP 58/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [8%@1036,53%@1036,37%@1036,25%@1132] EMC_FREQ 3%@1600 GR3D_FREQ 3%@153 APE 25 PLL@32.5C CPU@33.5C PMIC@100C GPU@32.5C AO@42C thermal@32.75C POM_5V_IN 2822/2518 POM_5V_GPU 80/55 POM_5V_CPU 885/481
RAM 2573/3963MB (lfb 107x4MB) SWAP 58/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [20%@1479,19%@1479,58%@1479,5%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@33.5C PMIC@100C GPU@32.5C AO@42C thermal@32.75C POM_5V_IN 2974/2523 POM_5V_GPU 40/55 POM_5V_CPU 963/487
RAM 2576/3963MB (lfb 107x4MB) SWAP 59/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [17%@1479,12%@1479,10%@1479,66%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@33.5C PMIC@100C GPU@32.5C AO@42.5C thermal@32.75C POM_5V_IN 2898/2527 POM_5V_GPU 40/55 POM_5V_CPU 925/492
RAM 2580/3963MB (lfb 107x4MB) SWAP 59/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [100%@1479,1%@1479,1%@1479,1%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32.5C AO@42.5C thermal@33C POM_5V_IN 2822/2531 POM_5V_GPU 0/54 POM_5V_CPU 885/497
RAM 2582/3963MB (lfb 107x4MB) SWAP 59/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [100%@1479,2%@1479,3%@1479,1%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@33.5C PMIC@100C GPU@32.5C AO@42C thermal@33C POM_5V_IN 2938/2535 POM_5V_GPU 40/54 POM_5V_CPU 924/502
RAM 2586/3963MB (lfb 107x4MB) SWAP 60/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [100%@1479,4%@1479,2%@1479,2%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@34C PMIC@100C GPU@32.5C AO@42.5C thermal@33.25C POM_5V_IN 2858/2539 POM_5V_GPU 40/54 POM_5V_CPU 925/506
RAM 2590/3963MB (lfb 107x4MB) SWAP 60/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [100%@1479,2%@1479,1%@1479,1%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33.5C PMIC@100C GPU@32.5C AO@42.5C thermal@33C POM_5V_IN 2978/2544 POM_5V_GPU 40/54 POM_5V_CPU 1004/512
RAM 2592/3963MB (lfb 107x4MB) SWAP 63/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [100%@1479,12%@1479,2%@1479,1%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@34C PMIC@100C GPU@32.5C AO@42.5C thermal@33C POM_5V_IN 2858/2547 POM_5V_GPU 40/54 POM_5V_CPU 925/516
RAM 2595/3963MB (lfb 107x4MB) SWAP 63/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [100%@1479,3%@1479,2%@1479,1%@1479] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@34C PMIC@100C GPU@33C AO@42.5C thermal@33.25C POM_5V_IN 2858/2551 POM_5V_GPU 40/53 POM_5V_CPU 925/521
RAM 2599/3963MB (lfb 107x4MB) SWAP 63/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [100%@1479,2%@1479,2%@1479,1%@1479] EMC_FREQ 2%@1600 GR3D_FREQ 9%@153 APE 25 PLL@32.5C CPU@34C PMIC@100C GPU@32.5C AO@42.5C thermal@33.5C POM_5V_IN 2938/2555 POM_5V_GPU 80/54 POM_5V_CPU 964/526
RAM 2603/3963MB (lfb 107x4MB) SWAP 63/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [33%@1479,70%@1479,2%@1479,1%@1479] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@33C PMIC@100C GPU@32.5C AO@42C thermal@32.75C POM_5V_IN 2817/2558 POM_5V_GPU 40/54 POM_5V_CPU 925/530
RAM 2607/3963MB (lfb 107x4MB) SWAP 63/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [5%@1479,73%@1479,6%@1479,21%@1479] EMC_FREQ 2%@1600 GR3D_FREQ 2%@153 APE 25 PLL@32.5C CPU@33.5C PMIC@100C GPU@32.5C AO@42C thermal@33C POM_5V_IN 2978/2562 POM_5V_GPU 161/55 POM_5V_CPU 844/533
RAM 2610/3963MB (lfb 107x4MB) SWAP 65/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [20%@1479,20%@1479,58%@1479,13%@1479] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32.5C CPU@34C PMIC@100C GPU@33C AO@42.5C thermal@33C POM_5V_IN 2858/2565 POM_5V_GPU 40/55 POM_5V_CPU 925/537
RAM 2637/3963MB (lfb 107x4MB) SWAP 67/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [8%@204,2%@204,62%@204,10%@204] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@33C AO@42.5C thermal@33.5C POM_5V_IN 2150/2561 POM_5V_GPU 40/54 POM_5V_CPU 243/534
RAM 2638/3963MB (lfb 107x4MB) SWAP 67/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [9%@204,12%@204,5%@204,2%@204] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31.5C CPU@32.5C PMIC@100C GPU@32.5C AO@42C thermal@32.75C POM_5V_IN 1988/2555 POM_5V_GPU 40/54 POM_5V_CPU 162/530
RAM 2638/3963MB (lfb 107x4MB) SWAP 67/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [7%@102,6%@102,4%@102,3%@102] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32.5C AO@42C thermal@32.75C POM_5V_IN 1991/2549 POM_5V_GPU 40/54 POM_5V_CPU 121/526
RAM 2638/3963MB (lfb 107x4MB) SWAP 67/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [9%@102,10%@102,4%@102,5%@102] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31.5C CPU@33C PMIC@100C GPU@32.5C AO@42C thermal@32.75C POM_5V_IN 1991/2544 POM_5V_GPU 40/54 POM_5V_CPU 121/522
RAM 2638/3963MB (lfb 107x4MB) SWAP 67/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [10%@204,6%@204,4%@204,3%@204] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32.5C AO@42C thermal@32.5C POM_5V_IN 2150/2540 POM_5V_GPU 81/54 POM_5V_CPU 202/519
RAM 2638/3963MB (lfb 107x4MB) SWAP 67/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [5%@102,12%@102,5%@102,2%@102] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31.5C CPU@33C PMIC@100C GPU@32C AO@42C thermal@32.75C POM_5V_IN 2032/2535 POM_5V_GPU 40/54 POM_5V_CPU 121/515
RAM 2639/3963MB (lfb 107x4MB) SWAP 67/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [21%@102,16%@102,10%@102,8%@102] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31.5C CPU@33C PMIC@100C GPU@32.5C AO@42C thermal@33.5C POM_5V_IN 2032/2530 POM_5V_GPU 40/54 POM_5V_CPU 162/512
RAM 2639/3963MB (lfb 107x4MB) SWAP 67/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [9%@102,10%@102,5%@102,5%@102] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32.5C AO@42C thermal@32.5C POM_5V_IN 2109/2526 POM_5V_GPU 121/55 POM_5V_CPU 162/508
RAM 2639/3963MB (lfb 107x4MB) SWAP 67/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [5%@102,6%@102,4%@204,5%@204] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@31.5C CPU@33C PMIC@100C GPU@32.5C AO@42C thermal@32.75C POM_5V_IN 2028/2521 POM_5V_GPU 40/54 POM_5V_CPU 121/505
RAM 2639/3963MB (lfb 107x4MB) SWAP 67/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [11%@102,11%@102,6%@102,3%@102] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@31.5C CPU@33C PMIC@100C GPU@32C AO@42C thermal@32.5C POM_5V_IN 1988/2516 POM_5V_GPU 40/54 POM_5V_CPU 121/501
RAM 2637/3963MB (lfb 107x4MB) SWAP 71/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [15%@710,15%@710,13%@710,10%@710] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@32C CPU@33.5C PMIC@100C GPU@32.5C AO@42C thermal@32.5C POM_5V_IN 2624/2517 POM_5V_GPU 40/54 POM_5V_CPU 363/500
RAM 2635/3963MB (lfb 107x4MB) SWAP 75/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [51%@1479,17%@1479,11%@1479,5%@1479] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33.5C PMIC@100C GPU@32.5C AO@42C thermal@32.75C POM_5V_IN 2898/2521 POM_5V_GPU 40/54 POM_5V_CPU 805/503
RAM 2635/3963MB (lfb 107x4MB) SWAP 75/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [89%@921,4%@921,3%@921,1%@921] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33.5C PMIC@100C GPU@34C AO@42C thermal@33C POM_5V_IN 2624/2522 POM_5V_GPU 0/54 POM_5V_CPU 444/502
RAM 2636/3963MB (lfb 107x4MB) SWAP 75/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [75%@1479,11%@1479,5%@1479,2%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32.5C AO@42C thermal@33C POM_5V_IN 2782/2524 POM_5V_GPU 40/53 POM_5V_CPU 725/504
RAM 2602/3963MB (lfb 107x4MB) SWAP 75/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [29%@307,11%@307,26%@307,13%@307] EMC_FREQ 4%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33.5C PMIC@100C GPU@32.5C AO@42C thermal@33C POM_5V_IN 2463/2523 POM_5V_GPU 40/53 POM_5V_CPU 242/502
RAM 2601/3963MB (lfb 107x4MB) SWAP 76/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [11%@1479,9%@1479,17%@1479,25%@1479] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32.5C AO@42C thermal@33C POM_5V_IN 2544/2524 POM_5V_GPU 40/53 POM_5V_CPU 362/500
RAM 2595/3963MB (lfb 107x4MB) SWAP 81/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [14%@307,8%@307,17%@307,21%@307] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32.5C AO@42C thermal@32.75C POM_5V_IN 2386/2522 POM_5V_GPU 40/53 POM_5V_CPU 242/498
RAM 2588/3963MB (lfb 107x4MB) SWAP 89/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [23%@307,17%@307,15%@307,6%@307] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32.5C AO@42C thermal@33.5C POM_5V_IN 2507/2522 POM_5V_GPU 40/53 POM_5V_CPU 242/496
RAM 1854/3963MB (lfb 110x4MB) SWAP 90/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [12%@518,23%@518,28%@518,7%@518] EMC_FREQ 3%@1600 GR3D_FREQ 7%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32.5C AO@42.5C thermal@33.25C POM_5V_IN 2309/2520 POM_5V_GPU 81/53 POM_5V_CPU 324/494
RAM 1854/3963MB (lfb 110x4MB) SWAP 90/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [5%@102,11%@102,2%@102,3%@102] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32.5C AO@42C thermal@32.75C POM_5V_IN 1991/2516 POM_5V_GPU 40/53 POM_5V_CPU 121/491
RAM 1854/3963MB (lfb 110x4MB) SWAP 90/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [11%@102,10%@102,8%@102,5%@102] EMC_FREQ 3%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32.5C AO@42C thermal@32.75C POM_5V_IN 2028/2512 POM_5V_GPU 40/53 POM_5V_CPU 121/488
RAM 1854/3963MB (lfb 110x4MB) SWAP 90/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [8%@102,9%@102,5%@204,5%@204] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@31.5C CPU@32.5C PMIC@100C GPU@32C AO@42C thermal@32.75C POM_5V_IN 1988/2507 POM_5V_GPU 40/53 POM_5V_CPU 121/485
RAM 1854/3963MB (lfb 110x4MB) SWAP 90/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [12%@102,15%@102,4%@102,4%@102] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32.5C AO@42C thermal@32.75C POM_5V_IN 2028/2503 POM_5V_GPU 40/53 POM_5V_CPU 162/482
RAM 1854/3963MB (lfb 110x4MB) SWAP 90/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [10%@102,12%@102,14%@102,10%@204] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32.5C AO@42C thermal@32.25C POM_5V_IN 2066/2500 POM_5V_GPU 81/53 POM_5V_CPU 202/480
RAM 1854/3963MB (lfb 110x4MB) SWAP 90/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [23%@204,16%@204,13%@204,10%@204] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32.5C AO@42.5C thermal@32.75C POM_5V_IN 2028/2496 POM_5V_GPU 40/53 POM_5V_CPU 121/477
RAM 1854/3963MB (lfb 110x4MB) SWAP 90/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [22%@204,19%@204,13%@204,13%@204] EMC_FREQ 2%@1600 GR3D_FREQ 0%@153 APE 25 PLL@32C CPU@33C PMIC@100C GPU@32.5C AO@42C thermal@32.75C POM_5V_IN 2106/2492 POM_5V_GPU 40/53 POM_5V_CPU 243/475
^C
christopher@ccz:~/aiguyyolotest1/Object-Detection-API$ 

Unfortunatelly using the tiny version did not work either.

I think it is me you is making a mistake at some point because I can’t image that it is not running on the nano at all.

I’m very sorry about the whole mess!!! I just realizied that there is an open error regarding the tiny version.

Saving the yolov3 weights as tensorflow model did not work.

Input:

python load_weights.py --weights ./weights/yolov3-tiny.weights --output ./weights/yolov3-tiny.tf --tiny

Output:

(aiguyyolotest1) christopher@ccz:~/aiguyyolotest1/Object-Detection-API$ python load_weights.py --weights ./weights/yolov3-tiny.weights --output ./weights/yolov3-tiny.tf --tiny
2020-09-30 12:00:32.386392: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-30 12:00:35.241788: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.6
2020-09-30 12:00:35.243542: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.6
2020-09-30 12:00:42.639616: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-09-30 12:00:42.646019: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-30 12:00:42.646179: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: 
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 23.84GiB/s
2020-09-30 12:00:42.646246: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-30 12:00:42.646369: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-09-30 12:00:42.726111: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-09-30 12:00:42.836383: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-09-30 12:00:42.963807: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-09-30 12:00:43.032119: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-09-30 12:00:43.032526: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-09-30 12:00:43.033197: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-30 12:00:43.033881: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-30 12:00:43.034062: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-09-30 12:00:43.145020: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2020-09-30 12:00:43.146691: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x10c50320 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-09-30 12:00:43.146748: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-09-30 12:00:43.223097: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-30 12:00:43.223487: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x10bb5760 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-09-30 12:00:43.223545: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
2020-09-30 12:00:43.224270: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-30 12:00:43.224409: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: 
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 23.84GiB/s
2020-09-30 12:00:43.224503: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-30 12:00:43.224558: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-09-30 12:00:43.224763: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-09-30 12:00:43.224885: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-09-30 12:00:43.224984: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-09-30 12:00:43.225074: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-09-30 12:00:43.225118: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-09-30 12:00:43.225475: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-30 12:00:43.225841: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-30 12:00:43.225934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-09-30 12:00:43.226041: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-30 12:00:53.335147: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-09-30 12:00:53.335276: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102]      0 
2020-09-30 12:00:53.335315: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0:   N 
2020-09-30 12:00:53.353703: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-30 12:00:53.355045: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-30 12:00:53.355514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 164 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
Model: "yolov3_tiny"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input (InputLayer)              [(None, None, None,  0                                            
__________________________________________________________________________________________________
yolo_darknet (Model)            ((None, None, None,  6298480     input[0][0]                      
__________________________________________________________________________________________________
yolo_conv_0 (Model)             (None, None, None, 2 263168      yolo_darknet[1][1]               
__________________________________________________________________________________________________
yolo_conv_1 (Model)             (None, None, None, 3 33280       yolo_conv_0[1][0]                
                                                                 yolo_darknet[1][0]               
__________________________________________________________________________________________________
yolo_output_0 (Model)           (None, None, None, 3 1312511     yolo_conv_0[1][0]                
__________________________________________________________________________________________________
yolo_output_1 (Model)           (None, None, None, 3 951295      yolo_conv_1[1][0]                
__________________________________________________________________________________________________
yolo_boxes_0 (Lambda)           ((None, None, None,  0           yolo_output_0[1][0]              
__________________________________________________________________________________________________
yolo_boxes_1 (Lambda)           ((None, None, None,  0           yolo_output_1[1][0]              
__________________________________________________________________________________________________
yolo_nms (Lambda)               ((None, 100, 4), (No 0           yolo_boxes_0[0][0]               
                                                                 yolo_boxes_0[0][1]               
                                                                 yolo_boxes_0[0][2]               
                                                                 yolo_boxes_1[0][0]               
                                                                 yolo_boxes_1[0][1]               
                                                                 yolo_boxes_1[0][2]               
==================================================================================================
Total params: 8,858,734
Trainable params: 8,852,366
Non-trainable params: 6,368
__________________________________________________________________________________________________
I0930 12:01:00.512497 548077543440 load_weights.py:25] model created
I0930 12:01:00.517812 548077543440 utils.py:47] yolo_darknet/conv2d bn
I0930 12:01:00.530101 548077543440 utils.py:47] yolo_darknet/conv2d_1 bn
I0930 12:01:00.547734 548077543440 utils.py:47] yolo_darknet/conv2d_2 bn
I0930 12:01:00.566265 548077543440 utils.py:47] yolo_darknet/conv2d_3 bn
I0930 12:01:00.576798 548077543440 utils.py:47] yolo_darknet/conv2d_4 bn
I0930 12:01:00.596955 548077543440 utils.py:47] yolo_darknet/conv2d_5 bn
Traceback (most recent call last):
  File "load_weights.py", line 40, in <module>
    app.run(main)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "load_weights.py", line 27, in main
    load_darknet_weights(yolo, FLAGS.weights, FLAGS.tiny)
  File "/home/christopher/aiguyyolotest1/Object-Detection-API/yolov3_tf2/utils.py", line 68, in load_darknet_weights
    conv_shape).transpose([2, 3, 1, 0])
ValueError: cannot reshape array of size 324670 into shape (512,256,3,3)

Maybe it is better to solve this problem first?

I really hope I’m not driving you crazy ;)

Hi,

Thanks for sharing the detail.

Based on the log, it seems that the ./weights/yolov3-tiny.weights file is not compatible to the default YOLOv3_tiny configure within the TensorFlow object detection API.
Is this a customized model?
If yes, would you mind to tell us how many class are you training on?

Thanks.

Hi,

no this is not a customized model. I did not change anything.

Don’t ask me why but now there are some files within the weights folder that should be the correct ones:

(aiguyyolotest1) christopher@ccz:~/aiguyyolotest1/Object-Detection-API/weights$ ls
checkpoint                          yolov3-tiny.tf.index
yolov3-tiny.tf.data-00000-of-00002  yolov3-tiny.weights

Nevertheless I’m absolutely confused why I can’t reproduce MY OWN failure???
When I do the same as before:

 `(aiguyyolotest1) christopher@ccz:~/aiguyyolotest1/Object-Detection-API$ python load_weights.py --weights ./weights/yolov3-tiny.weights --output ./weights/yolov3-tiny.tf --tiny


**Now again the OOM problem occurs:**

2020-10-08 10:28:43.695412: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-10-08 10:28:47.694721: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.6
2020-10-08 10:28:47.699409: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.6
2020-10-08 10:28:58.469681: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-10-08 10:28:58.476625: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-10-08 10:28:58.476813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: 
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 23.84GiB/s
2020-10-08 10:28:58.476894: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-10-08 10:28:58.477030: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-10-08 10:28:58.554407: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-10-08 10:28:58.660263: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-10-08 10:28:58.769775: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-10-08 10:28:58.833535: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-10-08 10:28:58.833831: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-10-08 10:28:58.834373: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-10-08 10:28:58.835141: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-10-08 10:28:58.835263: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-10-08 10:28:58.965511: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2020-10-08 10:28:58.966144: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xe8db2d0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-10-08 10:28:58.966197: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-10-08 10:28:59.051739: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-10-08 10:28:59.052051: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xe8408e0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-10-08 10:28:59.052104: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
2020-10-08 10:28:59.052783: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-10-08 10:28:59.052910: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: 
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 23.84GiB/s
2020-10-08 10:28:59.052995: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-10-08 10:28:59.053048: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-10-08 10:28:59.053257: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-10-08 10:28:59.053380: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-10-08 10:28:59.053475: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-10-08 10:28:59.053566: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-10-08 10:28:59.053610: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-10-08 10:28:59.053945: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-10-08 10:28:59.054298: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-10-08 10:28:59.054378: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-10-08 10:28:59.054460: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-10-08 10:29:01.431027: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-10-08 10:29:01.431252: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102]      0 
2020-10-08 10:29:01.431299: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0:   N 
2020-10-08 10:29:01.432170: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-10-08 10:29:01.432830: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-10-08 10:29:01.433172: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 61 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2020-10-08 10:29:14.135452: W tensorflow/core/common_runtime/bfc_allocator.cc:424] Allocator (GPU_0_bfc) ran out of memory trying to allocate 18.00MiB (rounded to 18874368).  Current allocation summary follows.
2020-10-08 10:29:14.135782: I tensorflow/core/common_runtime/bfc_allocator.cc:894] BFCAllocator dump for GPU_0_bfc
2020-10-08 10:29:14.135959: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (256): 	Total Chunks: 15, Chunks in use: 13. 3.8KiB allocated for chunks. 3.2KiB in use in bin. 1.8KiB client-requested in use in bin.
2020-10-08 10:29:14.136011: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (512): 	Total Chunks: 4, Chunks in use: 4. 2.0KiB allocated for chunks. 2.0KiB in use in bin. 2.0KiB client-requested in use in bin.
2020-10-08 10:29:14.136048: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (1024): 	Total Chunks: 6, Chunks in use: 6. 7.0KiB allocated for chunks. 7.0KiB in use in bin. 6.7KiB client-requested in use in bin.
2020-10-08 10:29:14.136081: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (2048): 	Total Chunks: 4, Chunks in use: 4. 8.0KiB allocated for chunks. 8.0KiB in use in bin. 8.0KiB client-requested in use in bin.
2020-10-08 10:29:14.136115: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (4096): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-08 10:29:14.136145: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (8192): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-08 10:29:14.136179: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (16384): 	Total Chunks: 2, Chunks in use: 1. 40.5KiB allocated for chunks. 18.0KiB in use in bin. 18.0KiB client-requested in use in bin.
2020-10-08 10:29:14.136210: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (32768): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-08 10:29:14.136240: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (65536): 	Total Chunks: 1, Chunks in use: 1. 72.0KiB allocated for chunks. 72.0KiB in use in bin. 72.0KiB client-requested in use in bin.
2020-10-08 10:29:14.136273: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (131072): 	Total Chunks: 1, Chunks in use: 0. 144.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-08 10:29:14.136306: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (262144): 	Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2020-10-08 10:29:14.136336: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (524288): 	Total Chunks: 1, Chunks in use: 0. 746.8KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-08 10:29:14.136753: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (1048576): 	Total Chunks: 4, Chunks in use: 1. 5.72MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2020-10-08 10:29:14.136935: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (2097152): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-08 10:29:14.136988: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (4194304): 	Total Chunks: 3, Chunks in use: 1. 16.00MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2020-10-08 10:29:14.137023: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (8388608): 	Total Chunks: 1, Chunks in use: 0. 8.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-08 10:29:14.137058: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (16777216): 	Total Chunks: 1, Chunks in use: 1. 30.51MiB allocated for chunks. 30.51MiB in use in bin. 18.00MiB client-requested in use in bin.
2020-10-08 10:29:14.137088: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (33554432): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-08 10:29:14.137115: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (67108864): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-08 10:29:14.137142: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (134217728): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-08 10:29:14.137169: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (268435456): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-10-08 10:29:14.137197: I tensorflow/core/common_runtime/bfc_allocator.cc:917] Bin for 18.00MiB was 16.00MiB, Chunk State: 
2020-10-08 10:29:14.137220: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 1048576
2020-10-08 10:29:14.137263: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0000 of size 1280 next 1
2020-10-08 10:29:14.137293: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at f008f0500 of size 256 next 2
2020-10-08 10:29:14.137318: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0600 of size 256 next 5
2020-10-08 10:29:14.137343: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0700 of size 256 next 4
2020-10-08 10:29:14.137368: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0800 of size 256 next 7
2020-10-08 10:29:14.137393: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0900 of size 256 next 8
2020-10-08 10:29:14.137418: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0a00 of size 256 next 9
2020-10-08 10:29:14.137442: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0b00 of size 256 next 11
2020-10-08 10:29:14.137465: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0c00 of size 256 next 13
2020-10-08 10:29:14.137489: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0d00 of size 256 next 14
2020-10-08 10:29:14.137512: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0e00 of size 256 next 15
2020-10-08 10:29:14.137536: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0f00 of size 256 next 17
2020-10-08 10:29:14.137559: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1000 of size 256 next 19
2020-10-08 10:29:14.137582: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1100 of size 256 next 20
2020-10-08 10:29:14.137607: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1200 of size 512 next 21
2020-10-08 10:29:14.137631: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1400 of size 512 next 3
2020-10-08 10:29:14.137656: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1600 of size 1792 next 6
2020-10-08 10:29:14.137681: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1d00 of size 512 next 24
2020-10-08 10:29:14.137705: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1f00 of size 512 next 25
2020-10-08 10:29:14.137730: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f2100 of size 1024 next 27
2020-10-08 10:29:14.137755: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f2500 of size 1024 next 30
2020-10-08 10:29:14.137779: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f2900 of size 1024 next 31
2020-10-08 10:29:14.137804: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f2d00 of size 1024 next 32
2020-10-08 10:29:14.137828: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3100 of size 2048 next 35
2020-10-08 10:29:14.137852: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3900 of size 2048 next 38
2020-10-08 10:29:14.137875: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4100 of size 2048 next 39
2020-10-08 10:29:14.137899: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4900 of size 2048 next 40
2020-10-08 10:29:14.137925: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at f008f5100 of size 256 next 42
2020-10-08 10:29:14.137952: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5200 of size 256 next 43
2020-10-08 10:29:14.138004: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at f008f5300 of size 23040 next 12
2020-10-08 10:29:14.138057: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008fad00 of size 18432 next 10
2020-10-08 10:29:14.138090: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at f008ff500 of size 147456 next 18
2020-10-08 10:29:14.138120: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00923500 of size 73728 next 16
2020-10-08 10:29:14.138148: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at f00935500 of size 764672 next 18446744073709551615
2020-10-08 10:29:14.138197: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 2097152
2020-10-08 10:29:14.138233: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009f0000 of size 294912 next 23
2020-10-08 10:29:14.138261: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at f00a38000 of size 1802240 next 18446744073709551615
2020-10-08 10:29:14.138285: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 4194304
2020-10-08 10:29:14.138307: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at f00bf0000 of size 1179648 next 29
2020-10-08 10:29:14.138331: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00d10000 of size 1179648 next 26
2020-10-08 10:29:14.138356: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at f00e30000 of size 1835008 next 18446744073709551615
2020-10-08 10:29:14.138378: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 8388608
2020-10-08 10:29:14.138400: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at f00ff0000 of size 8388608 next 18446744073709551615
2020-10-08 10:29:14.138421: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 16777216
2020-10-08 10:29:14.138445: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at f017f0000 of size 4718592 next 37
2020-10-08 10:29:14.138625: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01c70000 of size 4718592 next 34
2020-10-08 10:29:14.138677: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free  at f020f0000 of size 7340032 next 18446744073709551615
2020-10-08 10:29:14.138707: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 31989760
2020-10-08 10:29:14.138732: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f027f0000 of size 31989760 next 18446744073709551615
2020-10-08 10:29:14.138753: I tensorflow/core/common_runtime/bfc_allocator.cc:955]      Summary of in-use Chunks by size: 
2020-10-08 10:29:14.138783: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 13 Chunks of size 256 totalling 3.2KiB
2020-10-08 10:29:14.138810: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 4 Chunks of size 512 totalling 2.0KiB
2020-10-08 10:29:14.138834: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 4 Chunks of size 1024 totalling 4.0KiB
2020-10-08 10:29:14.138858: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 1280 totalling 1.2KiB
2020-10-08 10:29:14.138882: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 1792 totalling 1.8KiB
2020-10-08 10:29:14.138908: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 4 Chunks of size 2048 totalling 8.0KiB
2020-10-08 10:29:14.138934: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 18432 totalling 18.0KiB
2020-10-08 10:29:14.138961: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 73728 totalling 72.0KiB
2020-10-08 10:29:14.138987: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 294912 totalling 288.0KiB
2020-10-08 10:29:14.139013: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 1179648 totalling 1.12MiB
2020-10-08 10:29:14.139038: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 4718592 totalling 4.50MiB
2020-10-08 10:29:14.139065: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 31989760 totalling 30.51MiB
2020-10-08 10:29:14.139090: I tensorflow/core/common_runtime/bfc_allocator.cc:962] Sum Total of in-use chunks: 36.52MiB
2020-10-08 10:29:14.139113: I tensorflow/core/common_runtime/bfc_allocator.cc:964] total_region_allocated_bytes_: 64495616 memory_limit_: 64495616 available bytes: 0 curr_region_allocation_bytes_: 67108864
2020-10-08 10:29:14.139150: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Stats: 
Limit:                    64495616
InUse:                    38295808
MaxInUse:                 38296320
NumAllocs:                     155
MaxAllocSize:             31989760

2020-10-08 10:29:14.139196: W tensorflow/core/common_runtime/bfc_allocator.cc:429] ***___***______________________*********__________******************************xxxxxxxxxxxxxxxxxxxx
2020-10-08 10:29:14.139289: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at cwise_ops_common.cc:82 : Resource exhausted: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
  File "load_weights.py", line 34, in <module>
    app.run(main)
  File "/home/christopher/aiguyyolotest1/lib/python3.6/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/home/christopher/aiguyyolotest1/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "load_weights.py", line 15, in main
    yolo = YoloV3Tiny(classes=FLAGS.num_classes)
  File "/home/christopher/aiguyyolotest1/Object-Detection-API/yolov3_tf2/models.py", line 248, in YoloV3Tiny
    x_8, x = DarknetTiny(name='yolo_darknet')(x)
  File "/home/christopher/aiguyyolotest1/Object-Detection-API/yolov3_tf2/models.py", line 99, in DarknetTiny
    x = DarknetConv(x, 1024, 3)
  File "/home/christopher/aiguyyolotest1/Object-Detection-API/yolov3_tf2/models.py", line 52, in DarknetConv
    use_bias=not batch_norm, kernel_regularizer=l2(0.0005))(x)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 748, in __call__
    self._maybe_build(inputs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 2116, in _maybe_build
    self.build(input_shapes)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/layers/convolutional.py", line 161, in build
    dtype=self.dtype)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 446, in add_weight
    caching_device=caching_device)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/tracking/base.py", line 744, in _add_variable_with_custom_getter
    **kwargs_for_getter)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer_utils.py", line 142, in make_variable
    shape=variable_shape if variable_shape else None)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py", line 258, in __call__
    return cls._variable_v1_call(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py", line 219, in _variable_v1_call
    shape=shape)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py", line 197, in <lambda>
    previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variable_scope.py", line 2596, in default_variable_creator
    shape=shape)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py", line 262, in __call__
    return super(VariableMetaclass, cls).__call__(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py", line 1411, in __init__
    distribute_strategy=distribute_strategy)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py", line 1542, in _init_from_args
    initial_value() if init_from_fn else initial_value,
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer_utils.py", line 122, in <lambda>
    init_val = lambda: initializer(shape, dtype=dtype)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops_v2.py", line 425, in __call__
    return self._random_generator.random_uniform(shape, -limit, limit, dtype)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops_v2.py", line 788, in random_uniform
    shape=shape, minval=minval, maxval=maxval, dtype=dtype, seed=self.seed)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/random_ops.py", line 273, in random_uniform
    result = math_ops.add(rnd * (maxval - minval), minval, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/math_ops.py", line 902, in binary_op_wrapper
    return func(x, y, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/math_ops.py", line 1201, in _mul_dispatch
    return gen_math_ops.mul(x, y, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_math_ops.py", line 6122, in mul
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 6606, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Mul] name: conv2d_6/kernel/Initializer/random_uniform/mul/

Meanwhile I am a little bit desperate because of all the problems.

Maybe this is an important hint:

When I try to run the python file for the first time I always get this error.

Cannot allocate memory in static TLS block

This also happens with other python files.
I solve this problem with

export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libgomp.so.1

But on the next day this problem occurs again.

Hi,

The issue is OOM. Please note that Nano only have 4G memory.

2020-10-08 10:29:14.139113: I tensorflow/core/common_runtime/bfc_allocator.cc:964] total_region_allocated_bytes_: 64495616 memory_limit_: 64495616 available bytes: 0 curr_region_allocation_bytes_: 67108864
2020-10-08 10:29:14.139150: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Stats: 
Limit:                    64495616
InUse:                    38295808
MaxInUse:                 38296320
NumAllocs:                     155
MaxAllocSize:             31989760

It’s known that TensorFlow use much more memory for inference on Jetson.
If TensorFlow is not essential, you can try our sample and library for YOLOv4 instead:

Thanks

Thanks for the link. I will have a look at it.

I read something about creating a swap partition to avoid the OOM issue.
But I do not know whether this is a long-term solution.
Or is this only a workaround solution and rather not recommended?

Best regards.

Hi,

Sorry that this is a hardware limitation since Nano only has 4G memory.
Swap memory can be used by CPU but not GPU.

It’s known that TensorFlow uses lots of memory on Jetson.
So the better solution is to use TensorRT rather than TensorFlow.

Thanks.

Hi,

thanks for this information. I will try to find a solution with TensorRT or as another solution I will look for a script to convert TensorFlow to TensorRT.

Best regards
chris