I tried running it again on the nano and got another error, I put you everything this time :
2021-07-28 12:05:40.697152: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
Looking for /home/quadient/.keras-ocr/craft_mlt_25k.h5
2021-07-28 12:05:50.206055: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-07-28 12:05:50.212491: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero
2021-07-28 12:05:50.212652: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1742] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.86GiB deviceMemoryBandwidth: 194.55MiB/s
2021-07-28 12:05:50.212758: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
2021-07-28 12:05:50.217804: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
2021-07-28 12:05:50.221326: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-07-28 12:05:50.222332: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-07-28 12:05:50.227968: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-07-28 12:05:50.232081: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10
2021-07-28 12:05:50.232790: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-07-28 12:05:50.233274: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero
2021-07-28 12:05:50.233668: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero
2021-07-28 12:05:50.233782: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1884] Adding visible gpu devices: 0
2021-07-28 12:05:50.263544: W tensorflow/core/platform/profile_utils/cpu_utils.cc:108] Failed to find bogomips or clock in /proc/cpuinfo; cannot determine CPU frequency
2021-07-28 12:05:50.264127: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x151f2610 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-07-28 12:05:50.264186: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2021-07-28 12:05:50.337820: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero
2021-07-28 12:05:50.338132: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x15234fa0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-07-28 12:05:50.338187: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
2021-07-28 12:05:50.338725: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero
2021-07-28 12:05:50.338838: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1742] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.86GiB deviceMemoryBandwidth: 194.55MiB/s
2021-07-28 12:05:50.338945: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
2021-07-28 12:05:50.339072: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
2021-07-28 12:05:50.339161: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-07-28 12:05:50.339237: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-07-28 12:05:50.339309: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-07-28 12:05:50.339380: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10
2021-07-28 12:05:50.339450: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-07-28 12:05:50.339707: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero
2021-07-28 12:05:50.339973: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero
2021-07-28 12:05:50.340041: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1884] Adding visible gpu devices: 0
2021-07-28 12:05:50.340124: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.2
2021-07-28 12:05:52.510201: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1283] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-07-28 12:05:52.510361: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1289] 0
2021-07-28 12:05:52.510398: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1302] 0: N
2021-07-28 12:05:52.510956: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero
2021-07-28 12:05:52.511342: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1046] ARM64 does not support NUMA - returning NUMA node zero
2021-07-28 12:05:52.511501: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1428] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 88 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2021-07-28 12:06:04.705958: W tensorflow/core/common_runtime/bfc_allocator.cc:431] Allocator (GPU_0_bfc) ran out of memory trying to allocate 18.00MiB (rounded to 18874368)requested by op RandomUniform
Current allocation summary follows.
2021-07-28 12:06:04.706038: I tensorflow/core/common_runtime/bfc_allocator.cc:970] BFCAllocator dump for GPU_0_bfc
2021-07-28 12:06:04.706080: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (256): Total Chunks: 29, Chunks in use: 29. 7.2KiB allocated for chunks. 7.2KiB in use in bin. 2.6KiB client-requested in use in bin.
2021-07-28 12:06:04.706118: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (512): Total Chunks: 10, Chunks in use: 10. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2021-07-28 12:06:04.706156: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (1024): Total Chunks: 16, Chunks in use: 16. 16.8KiB allocated for chunks. 16.8KiB in use in bin. 16.0KiB client-requested in use in bin.
2021-07-28 12:06:04.706191: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (2048): Total Chunks: 30, Chunks in use: 30. 60.0KiB allocated for chunks. 60.0KiB in use in bin. 60.0KiB client-requested in use in bin.
2021-07-28 12:06:04.706223: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (4096): Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2021-07-28 12:06:04.706253: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-07-28 12:06:04.706282: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-07-28 12:06:04.706311: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-07-28 12:06:04.706339: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-07-28 12:06:04.706372: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (131072): Total Chunks: 2, Chunks in use: 1. 358.2KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2021-07-28 12:06:04.706404: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (262144): Total Chunks: 2, Chunks in use: 1. 576.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2021-07-28 12:06:04.706438: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (524288): Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2021-07-28 12:06:04.706469: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (1048576): Total Chunks: 3, Chunks in use: 1. 3.38MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2021-07-28 12:06:04.706499: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (2097152): Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2021-07-28 12:06:04.706530: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (4194304): Total Chunks: 3, Chunks in use: 1. 13.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2021-07-28 12:06:04.706563: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (8388608): Total Chunks: 7, Chunks in use: 5. 65.70MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2021-07-28 12:06:04.706592: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-07-28 12:06:04.706621: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-07-28 12:06:04.706649: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-07-28 12:06:04.706677: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-07-28 12:06:04.706705: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-07-28 12:06:04.706736: I tensorflow/core/common_runtime/bfc_allocator.cc:993] Bin for 18.00MiB was 16.00MiB, Chunk State:
2021-07-28 12:06:04.706760: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 92950528
2021-07-28 12:06:04.706794: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b30000 of size 1280 next 1
2021-07-28 12:06:04.706821: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b30500 of size 256 next 5
2021-07-28 12:06:04.706847: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b30600 of size 256 next 8
2021-07-28 12:06:04.706872: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b30700 of size 256 next 9
2021-07-28 12:06:04.706898: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b30800 of size 256 next 10
2021-07-28 12:06:04.706925: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b30900 of size 256 next 11
2021-07-28 12:06:04.706952: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b30a00 of size 256 next 12
2021-07-28 12:06:04.706982: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b30b00 of size 256 next 13
2021-07-28 12:06:04.707008: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b30c00 of size 256 next 15
2021-07-28 12:06:04.707033: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b30d00 of size 256 next 16
2021-07-28 12:06:04.707058: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b30e00 of size 256 next 14
2021-07-28 12:06:04.707083: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b30f00 of size 256 next 19
2021-07-28 12:06:04.707107: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b31000 of size 256 next 20
2021-07-28 12:06:04.707132: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b31100 of size 256 next 21
2021-07-28 12:06:04.707156: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b31200 of size 256 next 22
2021-07-28 12:06:04.707181: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b31300 of size 256 next 23
2021-07-28 12:06:04.707206: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b31400 of size 256 next 24
2021-07-28 12:06:04.707232: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b31500 of size 512 next 27
2021-07-28 12:06:04.707256: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b31700 of size 512 next 28
2021-07-28 12:06:04.707281: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b31900 of size 512 next 29
2021-07-28 12:06:04.707306: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b31b00 of size 512 next 30
2021-07-28 12:06:04.707333: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b31d00 of size 768 next 2
2021-07-28 12:06:04.707358: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b32000 of size 256 next 3
2021-07-28 12:06:04.707383: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b32100 of size 256 next 4
2021-07-28 12:06:04.707407: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b32200 of size 256 next 32
2021-07-28 12:06:04.707431: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b32300 of size 256 next 33
2021-07-28 12:06:04.707455: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b32400 of size 512 next 31
2021-07-28 12:06:04.707479: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b32600 of size 512 next 36
2021-07-28 12:06:04.707505: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b32800 of size 512 next 37
2021-07-28 12:06:04.707530: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b32a00 of size 512 next 38
2021-07-28 12:06:04.707555: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b32c00 of size 512 next 39
2021-07-28 12:06:04.707579: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b32e00 of size 256 next 40
2021-07-28 12:06:04.707603: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b32f00 of size 256 next 41
2021-07-28 12:06:04.707628: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b33000 of size 1024 next 44
2021-07-28 12:06:04.707653: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b33400 of size 1024 next 45
2021-07-28 12:06:04.707678: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b33800 of size 1536 next 6
2021-07-28 12:06:04.707705: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b33e00 of size 6912 next 7
2021-07-28 12:06:04.707731: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b35900 of size 1024 next 46
2021-07-28 12:06:04.707756: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b35d00 of size 1024 next 47
2021-07-28 12:06:04.707781: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b36100 of size 256 next 49
2021-07-28 12:06:04.707805: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b36200 of size 256 next 50
2021-07-28 12:06:04.707830: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b36300 of size 1024 next 48
2021-07-28 12:06:04.707855: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b36700 of size 1024 next 53
2021-07-28 12:06:04.707880: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b36b00 of size 1024 next 54
2021-07-28 12:06:04.707904: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b36f00 of size 1024 next 55
2021-07-28 12:06:04.707929: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b37300 of size 1024 next 56
2021-07-28 12:06:04.707953: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b37700 of size 1024 next 57
2021-07-28 12:06:04.707977: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b37b00 of size 1024 next 59
2021-07-28 12:06:04.708003: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b37f00 of size 1024 next 60
2021-07-28 12:06:04.708028: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b38300 of size 1024 next 61
2021-07-28 12:06:04.708053: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b38700 of size 1024 next 62
2021-07-28 12:06:04.708078: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b38b00 of size 256 next 63
2021-07-28 12:06:04.708103: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b38c00 of size 256 next 64
2021-07-28 12:06:04.708129: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b38d00 of size 2048 next 67
2021-07-28 12:06:04.708154: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b39500 of size 2048 next 68
2021-07-28 12:06:04.708178: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b39d00 of size 2048 next 69
2021-07-28 12:06:04.708203: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b3a500 of size 2048 next 70
2021-07-28 12:06:04.708227: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b3ad00 of size 2048 next 71
2021-07-28 12:06:04.708251: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b3b500 of size 256 next 73
2021-07-28 12:06:04.708276: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b3b600 of size 256 next 74
2021-07-28 12:06:04.708302: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b3b700 of size 2048 next 72
2021-07-28 12:06:04.708326: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b3bf00 of size 2048 next 77
2021-07-28 12:06:04.708350: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b3c700 of size 2048 next 78
2021-07-28 12:06:04.708374: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b3cf00 of size 2048 next 79
2021-07-28 12:06:04.708398: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b3d700 of size 2048 next 80
2021-07-28 12:06:04.708423: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b3df00 of size 2048 next 81
2021-07-28 12:06:04.708448: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b3e700 of size 2048 next 83
2021-07-28 12:06:04.708473: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b3ef00 of size 2048 next 84
2021-07-28 12:06:04.708499: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b3f700 of size 2048 next 85
2021-07-28 12:06:04.708524: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b3ff00 of size 2048 next 86
2021-07-28 12:06:04.708550: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b40700 of size 2048 next 87
2021-07-28 12:06:04.708574: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b40f00 of size 2048 next 89
2021-07-28 12:06:04.708598: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b41700 of size 2048 next 90
2021-07-28 12:06:04.708622: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b41f00 of size 2048 next 91
2021-07-28 12:06:04.708647: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b42700 of size 2048 next 92
2021-07-28 12:06:04.708672: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b42f00 of size 2048 next 93
2021-07-28 12:06:04.708697: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b43700 of size 2048 next 95
2021-07-28 12:06:04.708721: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b43f00 of size 2048 next 96
2021-07-28 12:06:04.708746: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b44700 of size 2048 next 97
2021-07-28 12:06:04.708770: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b44f00 of size 2048 next 98
2021-07-28 12:06:04.708795: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b45700 of size 2048 next 99
2021-07-28 12:06:04.708820: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b45f00 of size 2048 next 101
2021-07-28 12:06:04.708845: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b46700 of size 2048 next 102
2021-07-28 12:06:04.708870: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b46f00 of size 2048 next 103
2021-07-28 12:06:04.708894: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b47700 of size 2048 next 104
2021-07-28 12:06:04.708919: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b47f00 of size 256 next 105
2021-07-28 12:06:04.708945: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at f00b48000 of size 219392 next 18
2021-07-28 12:06:04.709026: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00b7d900 of size 147456 next 17
2021-07-28 12:06:04.709056: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at f00ba1900 of size 294912 next 26
2021-07-28 12:06:04.709081: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00be9900 of size 294912 next 25
2021-07-28 12:06:04.709107: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at f00c31900 of size 1179648 next 35
2021-07-28 12:06:04.709132: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00d51900 of size 589824 next 34
2021-07-28 12:06:04.709157: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at f00de1900 of size 1179648 next 43
2021-07-28 12:06:04.709182: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f00f01900 of size 1179648 next 42
2021-07-28 12:06:04.709208: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at f01021900 of size 4718592 next 52
2021-07-28 12:06:04.709233: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f014a1900 of size 2359296 next 51
2021-07-28 12:06:04.709259: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f016e1900 of size 2359296 next 58
2021-07-28 12:06:04.709285: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at f01921900 of size 4718592 next 66
2021-07-28 12:06:04.709310: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f01da1900 of size 4718592 next 65
2021-07-28 12:06:04.709336: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at f02221900 of size 9437184 next 100
2021-07-28 12:06:04.709362: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f02b21900 of size 9437184 next 76
2021-07-28 12:06:04.709387: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f03421900 of size 9437184 next 75
2021-07-28 12:06:04.709411: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f03d21900 of size 9437184 next 82
2021-07-28 12:06:04.709436: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f04621900 of size 9437184 next 88
2021-07-28 12:06:04.709460: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at f04f21900 of size 9437184 next 94
2021-07-28 12:06:04.709485: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at f05821900 of size 12269312 next 18446744073709551615
2021-07-28 12:06:04.709509: I tensorflow/core/common_runtime/bfc_allocator.cc:1031] Summary of in-use Chunks by size:
2021-07-28 12:06:04.709540: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 29 Chunks of size 256 totalling 7.2KiB
2021-07-28 12:06:04.709568: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 9 Chunks of size 512 totalling 4.5KiB
2021-07-28 12:06:04.709595: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 768 totalling 768B
2021-07-28 12:06:04.709622: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 14 Chunks of size 1024 totalling 14.0KiB
2021-07-28 12:06:04.709650: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 1280 totalling 1.2KiB
2021-07-28 12:06:04.709676: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 1536 totalling 1.5KiB
2021-07-28 12:06:04.709705: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 30 Chunks of size 2048 totalling 60.0KiB
2021-07-28 12:06:04.709733: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 6912 totalling 6.8KiB
2021-07-28 12:06:04.709762: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 147456 totalling 144.0KiB
2021-07-28 12:06:04.709790: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 294912 totalling 288.0KiB
2021-07-28 12:06:04.709817: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 589824 totalling 576.0KiB
2021-07-28 12:06:04.709844: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 1179648 totalling 1.12MiB
2021-07-28 12:06:04.709871: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 2 Chunks of size 2359296 totalling 4.50MiB
2021-07-28 12:06:04.709897: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 4718592 totalling 4.50MiB
2021-07-28 12:06:04.709925: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 5 Chunks of size 9437184 totalling 45.00MiB
2021-07-28 12:06:04.709952: I tensorflow/core/common_runtime/bfc_allocator.cc:1038] Sum Total of in-use chunks: 56.20MiB
2021-07-28 12:06:04.709977: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] total_region_allocated_bytes_: 92950528 memory_limit_: 92950528 available bytes: 0 curr_region_allocation_bytes_: 185901056
2021-07-28 12:06:04.710011: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Stats:
Limit: 92950528
InUse: 58933248
MaxInUse: 80629248
NumAllocs: 305
MaxAllocSize: 12269312
Reserved: 0
PeakReserved: 0
LargestFreeBlock: 0
2021-07-28 12:06:04.710082: W tensorflow/core/common_runtime/bfc_allocator.cc:439] ******____******____******__________***************************************************_____________
2021-07-28 12:06:04.710166: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at random_op.cc:77 : Resource exhausted: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "keras_test.py", line 8, in <module>
pipeline = keras_ocr.pipeline.Pipeline()
File "/home/quadient/Documents/autolabelocr/venvocr/lib/python3.6/site-packages/keras_ocr/pipeline.py", line 19, in __init__
detector = detection.Detector()
File "/home/quadient/Documents/autolabelocr/venvocr/lib/python3.6/site-packages/keras_ocr/detection.py", line 610, in __init__
self.model = build_keras_model(weights_path=weights_path, backbone_name=backbone_name)
File "/home/quadient/Documents/autolabelocr/venvocr/lib/python3.6/site-packages/keras_ocr/detection.py", line 318, in build_keras_model
name='basenet.slice5.1')(s5)
File "/home/quadient/Documents/autolabelocr/venvocr/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 926, in __call__
input_list)
File "/home/quadient/Documents/autolabelocr/venvocr/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1098, in _functional_construction_call
self._maybe_build(inputs)
File "/home/quadient/Documents/autolabelocr/venvocr/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 2643, in _maybe_build
self.build(input_shapes) # pylint:disable=not-callable
File "/home/quadient/Documents/autolabelocr/venvocr/lib/python3.6/site-packages/tensorflow/python/keras/layers/convolutional.py", line 204, in build
dtype=self.dtype)
File "/home/quadient/Documents/autolabelocr/venvocr/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 614, in add_weight
caching_device=caching_device)
File "/home/quadient/Documents/autolabelocr/venvocr/lib/python3.6/site-packages/tensorflow/python/training/tracking/base.py", line 750, in _add_variable_with_custom_getter
**kwargs_for_getter)
File "/home/quadient/Documents/autolabelocr/venvocr/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer_utils.py", line 145, in make_variable
shape=variable_shape if variable_shape else None)
File "/home/quadient/Documents/autolabelocr/venvocr/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 260, in __call__
return cls._variable_v1_call(*args, **kwargs)
File "/home/quadient/Documents/autolabelocr/venvocr/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 221, in _variable_v1_call
shape=shape)
File "/home/quadient/Documents/autolabelocr/venvocr/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 199, in <lambda>
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File "/home/quadient/Documents/autolabelocr/venvocr/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 2597, in default_variable_creator
shape=shape)
File "/home/quadient/Documents/autolabelocr/venvocr/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 264, in __call__
return super(VariableMetaclass, cls).__call__(*args, **kwargs)
File "/home/quadient/Documents/autolabelocr/venvocr/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 1518, in __init__
distribute_strategy=distribute_strategy)
File "/home/quadient/Documents/autolabelocr/venvocr/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 1651, in _init_from_args
initial_value() if init_from_fn else initial_value,
File "/home/quadient/Documents/autolabelocr/venvocr/lib/python3.6/site-packages/tensorflow/python/keras/initializers/initializers_v2.py", line 397, in __call__
return super(VarianceScaling, self).__call__(shape, dtype=_get_dtype(dtype))
File "/home/quadient/Documents/autolabelocr/venvocr/lib/python3.6/site-packages/tensorflow/python/ops/init_ops_v2.py", line 561, in __call__
return self._random_generator.random_uniform(shape, -limit, limit, dtype)
File "/home/quadient/Documents/autolabelocr/venvocr/lib/python3.6/site-packages/tensorflow/python/ops/init_ops_v2.py", line 1044, in random_uniform
shape=shape, minval=minval, maxval=maxval, dtype=dtype, seed=self.seed)
File "/home/quadient/Documents/autolabelocr/venvocr/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py", line 201, in wrapper
return target(*args, **kwargs)
File "/home/quadient/Documents/autolabelocr/venvocr/lib/python3.6/site-packages/tensorflow/python/ops/random_ops.py", line 302, in random_uniform
shape, dtype, seed=seed1, seed2=seed2)
File "/home/quadient/Documents/autolabelocr/venvocr/lib/python3.6/site-packages/tensorflow/python/ops/gen_random_ops.py", line 726, in random_uniform
_ops.raise_from_not_ok_status(e, name)
File "/home/quadient/Documents/autolabelocr/venvocr/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 6843, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:RandomUniform]