Hi,
I just upgraded my Xavier NX from Jetpack 4.5 to Jetpack 4.6, and installed tensorflow 2.6.2+nv21.12. When I build my program to exe with Pyinstaller 4.2, the exe cannot run. I found that sensorflow does not work. This software (exe was built with Pyinstall 4.2) runs well on Jetpack 4.5. Please help. Thanks.
Hi,
We want to reproduce this issue internally.
Could you please share a simple TensorFlow script and detailed steps to reproduce with us?
Thanks.
Hi, Aasta:
For sharing a simple Tensorflow script, I need to take it out from my main.py, this may take some time. Let me try how to do it.
Below is the error log:
2022-09-28 09:50:17.762678: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:828] layout failed: Invalid argument: Subshape must have computed start >= end since stride is negative, but is 0 and 2 (computed from start 0 and end 9223372036854775807 over shape with rank 2 and stride-1)
2022-09-28 09:50:21.688914: I tensorflow/stream_executor/cuda/cuda_dnn.cc:381] Loaded cuDNN version 8201
2022-09-28 09:50:46.413678: W tensorflow/core/common_runtime/bfc_allocator.cc:463] Allocator (GPU_0_bfc) ran out of memory trying to allocate 96.00MiB (rounded to 100663296)requested by op concatenate/concat
If the cause is memory fragmentation maybe the environment variable ‘TF_GPU_ALLOCATOR=cuda_malloc_async’ will improve the situation.
Current allocation summary follows.
Current allocation summary follows.
2022-09-28 09:50:46.414058: I tensorflow/core/common_runtime/bfc_allocator.cc:1013] BFCAllocator dump for GPU_0_bfc
2022-09-28 09:50:46.414181: I tensorflow/core/common_runtime/bfc_allocator.cc:1020] Bin (256): Total Chunks: 116, Chunks in use: 116. 29.0KiB allocated for chunks. 29.0KiB in use in bin. 17.5KiB client-requested in use in bin.
2022-09-28 09:50:46.414780: I tensorflow/core/common_runtime/bfc_allocator.cc:1020] Bin (512): Total Chunks: 83, Chunks in use: 83. 42.8KiB allocated for chunks. 42.8KiB in use in bin. 41.6KiB client-requested in use in bin.
2022-09-28 09:50:46.415254: I tensorflow/core/common_runtime/bfc_allocator.cc:1020] Bin (1024): Total Chunks: 26, Chunks in use: 26. 29.0KiB allocated for chunks. 29.0KiB in use in bin. 28.1KiB client-requested in use in bin.
2022-09-28 09:50:46.415446: I tensorflow/core/common_runtime/bfc_allocator.cc:1020] Bin (2048): Total Chunks: 7, Chunks in use: 7. 16.8KiB allocated for chunks. 16.8KiB in use in bin. 16.8KiB client-requested in use in bin.
2022-09-28 09:50:46.415633: I tensorflow/core/common_runtime/bfc_allocator.cc:1020] Bin (4096): Total Chunks: 7, Chunks in use: 7. 39.2KiB allocated for chunks. 39.2KiB in use in bin. 39.2KiB client-requested in use in bin.
2022-09-28 09:50:46.415855: I tensorflow/core/common_runtime/bfc_allocator.cc:1020] Bin (8192): Total Chunks: 5, Chunks in use: 5. 49.8KiB allocated for chunks. 49.8KiB in use in bin. 49.6KiB client-requested in use in bin.
2022-09-28 09:50:46.416129: I tensorflow/core/common_runtime/bfc_allocator.cc:1020] Bin (16384): Total Chunks: 2, Chunks in use: 2. 50.0KiB allocated for chunks. 50.0KiB in use in bin. 50.0KiB client-requested in use in bin.
2022-09-28 09:50:46.416344: I tensorflow/core/common_runtime/bfc_allocator.cc:1020] Bin (32768): Total Chunks: 26, Chunks in use: 26. 857.0KiB allocated for chunks. 857.0KiB in use in bin. 857.0KiB client-requested in use in bin.
2022-09-28 09:50:46.416528: I tensorflow/core/common_runtime/bfc_allocator.cc:1020] Bin (65536): Total Chunks: 2, Chunks in use: 2. 136.0KiB allocated for chunks. 136.0KiB in use in bin. 136.0KiB client-requested in use in bin.
2022-09-28 09:50:46.416824: I tensorflow/core/common_runtime/bfc_allocator.cc:1020] Bin (131072): Total Chunks: 10, Chunks in use: 10. 1.27MiB allocated for chunks. 1.27MiB in use in bin. 1.27MiB client-requested in use in bin.
2022-09-28 09:50:46.417003: I tensorflow/core/common_runtime/bfc_allocator.cc:1020] Bin (262144): Total Chunks: 23, Chunks in use: 23. 6.47MiB allocated for chunks. 6.47MiB in use in bin. 6.47MiB client-requested in use in bin.
2022-09-28 09:50:46.417187: I tensorflow/core/common_runtime/bfc_allocator.cc:1020] Bin (524288): Total Chunks: 3, Chunks in use: 3. 1.69MiB allocated for chunks. 1.69MiB in use in bin. 1.69MiB client-requested in use in bin.
2022-09-28 09:50:46.417371: I tensorflow/core/common_runtime/bfc_allocator.cc:1020] Bin (1048576): Total Chunks: 9, Chunks in use: 9. 10.12MiB allocated for chunks. 10.12MiB in use in bin. 10.12MiB client-requested in use in bin.
2022-09-28 09:50:46.417548: I tensorflow/core/common_runtime/bfc_allocator.cc:1020] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-09-28 09:50:46.417726: I tensorflow/core/common_runtime/bfc_allocator.cc:1020] Bin (4194304): Total Chunks: 1, Chunks in use: 0. 6.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-09-28 09:50:46.417905: I tensorflow/core/common_runtime/bfc_allocator.cc:1020] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-09-28 09:50:46.418167: I tensorflow/core/common_runtime/bfc_allocator.cc:1020] Bin (16777216): Total Chunks: 4, Chunks in use: 3. 72.00MiB allocated for chunks. 56.00MiB in use in bin. 56.00MiB client-requested in use in bin.
2022-09-28 09:50:46.418293: I tensorflow/core/common_runtime/bfc_allocator.cc:1020] Bin (33554432): Total Chunks: 2, Chunks in use: 2. 64.00MiB allocated for chunks. 64.00MiB in use in bin. 64.00MiB client-requested in use in bin.
2022-09-28 09:50:46.418595: I tensorflow/core/common_runtime/bfc_allocator.cc:1020] Bin (67108864): Total Chunks: 2, Chunks in use: 1. 158.80MiB allocated for chunks. 72.00MiB in use in bin. 72.00MiB client-requested in use in bin.
2022-09-28 09:50:46.418670: I tensorflow/core/common_runtime/bfc_allocator.cc:1020] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-09-28 09:50:46.418846: I tensorflow/core/common_runtime/bfc_allocator.cc:1020] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-09-28 09:50:46.418966: I tensorflow/core/common_runtime/bfc_allocator.cc:1036] Bin for 96.00MiB was 64.00MiB, Chunk State:
2022-09-28 09:50:46.419036: I tensorflow/core/common_runtime/bfc_allocator.cc:1042] Size: 86.80MiB | Requested Size: 32.00MiB | in_use: 0 | bin_num: 18, prev: Size: 16.00MiB | Requested Size: 16.00MiB | in_use: 1 | bin_num: -1
2022-09-28 09:50:46.419085: I tensorflow/core/common_runtime/bfc_allocator.cc:1049] Next region of size 337186816
2022-09-28 09:50:46.419155: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20a72a000 of size 1179648 next 1
2022-09-28 09:50:46.419204: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20a84a000 of size 33792 next 2
2022-09-28 09:50:46.419249: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20a852400 of size 256 next 3
2022-09-28 09:50:46.419302: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20a852500 of size 1179648 next 4
2022-09-28 09:50:46.419350: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20a972500 of size 33792 next 5
2022-09-28 09:50:46.419393: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20a97a900 of size 256 next 6
2022-09-28 09:50:46.419436: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20a97aa00 of size 589824 next 7
2022-09-28 09:50:46.419481: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20aa0aa00 of size 32768 next 8
2022-09-28 09:50:46.419532: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20aa12a00 of size 294912 next 9
2022-09-28 09:50:46.419575: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20aa5aa00 of size 32768 next 10
2022-09-28 09:50:46.419651: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20aa62a00 of size 294912 next 11
2022-09-28 09:50:46.419706: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20aaaaa00 of size 32768 next 12
2022-09-28 09:50:46.419750: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20aab2a00 of size 294912 next 13
2022-09-28 09:50:46.419798: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20aafaa00 of size 32768 next 14
2022-09-28 09:50:46.419881: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20ab02a00 of size 294912 next 15
2022-09-28 09:50:46.419927: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20ab4aa00 of size 65536 next 16
2022-09-28 09:50:46.419983: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20ab5aa00 of size 1179648 next 17
2022-09-28 09:50:46.420064: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20ac7aa00 of size 131072 next 18
2022-09-28 09:50:46.420104: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20ac9aa00 of size 1179648 next 19
2022-09-28 09:50:46.420140: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20adbaa00 of size 131072 next 20
2022-09-28 09:50:46.420177: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20addaa00 of size 589824 next 21
2022-09-28 09:50:46.420214: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20ae6aa00 of size 32768 next 22
2022-09-28 09:50:46.420250: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20ae72a00 of size 294912 next 23
2022-09-28 09:50:46.420286: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20aebaa00 of size 32768 next 24
2022-09-28 09:50:46.420322: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20aec2a00 of size 294912 next 25
2022-09-28 09:50:46.420374: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20af0aa00 of size 32768 next 26
2022-09-28 09:50:46.420412: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20af12a00 of size 294912 next 27
2022-09-28 09:50:46.420448: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20af5aa00 of size 32768 next 28
2022-09-28 09:50:46.420484: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20af62a00 of size 294912 next 29
2022-09-28 09:50:46.420520: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20afaaa00 of size 32768 next 30
2022-09-28 09:50:46.420555: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20afb2a00 of size 294912 next 31
2022-09-28 09:50:46.420591: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20affaa00 of size 32768 next 32
2022-09-28 09:50:46.420626: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b002a00 of size 294912 next 33
2022-09-28 09:50:46.420662: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b04aa00 of size 32768 next 34
2022-09-28 09:50:46.420699: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b052a00 of size 294912 next 35
2022-09-28 09:50:46.420734: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b09aa00 of size 32768 next 36
2022-09-28 09:50:46.420769: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b0a2a00 of size 294912 next 37
2022-09-28 09:50:46.420804: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b0eaa00 of size 131072 next 38
2022-09-28 09:50:46.420841: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b10aa00 of size 1179648 next 39
2022-09-28 09:50:46.420877: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b22aa00 of size 131072 next 40
2022-09-28 09:50:46.420912: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b24aa00 of size 1179648 next 41
2022-09-28 09:50:46.420947: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b36aa00 of size 131072 next 42
2022-09-28 09:50:46.420986: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b38aa00 of size 6912 next 43
2022-09-28 09:50:46.421022: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b38c500 of size 147456 next 44
2022-09-28 09:50:46.421080: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b3b0500 of size 8192 next 45
2022-09-28 09:50:46.421117: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b3b2500 of size 73728 next 46
2022-09-28 09:50:46.421152: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b3c4500 of size 294912 next 47
2022-09-28 09:50:46.421184: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b40c500 of size 32768 next 48
2022-09-28 09:50:46.421241: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b414500 of size 294912 next 49
2022-09-28 09:50:46.421275: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b45c500 of size 32768 next 50
2022-09-28 09:50:46.421311: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b464500 of size 294912 next 51
2022-09-28 09:50:46.421344: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b4ac500 of size 589824 next 52
2022-09-28 09:50:46.421376: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b53c500 of size 32768 next 53
2022-09-28 09:50:46.421447: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b544500 of size 294912 next 54
2022-09-28 09:50:46.421481: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b58c500 of size 32768 next 55
2022-09-28 09:50:46.421513: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b594500 of size 294912 next 56
2022-09-28 09:50:46.421544: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b5dc500 of size 32768 next 57
2022-09-28 09:50:46.421605: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b5e4500 of size 294912 next 58
2022-09-28 09:50:46.421639: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b62c500 of size 32768 next 59
2022-09-28 09:50:46.421671: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b634500 of size 294912 next 60
2022-09-28 09:50:46.421702: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b67c500 of size 32768 next 61
2022-09-28 09:50:46.421734: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b684500 of size 294912 next 62
2022-09-28 09:50:46.421770: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b6cc500 of size 32768 next 63
2022-09-28 09:50:46.421801: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b6d4500 of size 294912 next 64
2022-09-28 09:50:46.421833: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b71c500 of size 32768 next 65
2022-09-28 09:50:46.421865: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b724500 of size 294912 next 66
2022-09-28 09:50:46.421898: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b76c500 of size 32768 next 67
2022-09-28 09:50:46.421930: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b774500 of size 294912 next 68
2022-09-28 09:50:46.421961: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b7bc500 of size 131072 next 69
2022-09-28 09:50:46.422033: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b7dc500 of size 1179648 next 70
2022-09-28 09:50:46.422067: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b8fc500 of size 131072 next 71
2022-09-28 09:50:46.422101: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20b91c500 of size 1179648 next 72
2022-09-28 09:50:46.422136: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20ba3c500 of size 131072 next 73
2022-09-28 09:50:46.422169: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20ba5c500 of size 1179648 next 74
2022-09-28 09:50:46.422223: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb7c500 of size 33792 next 75
2022-09-28 09:50:46.422250: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb84900 of size 256 next 76
2022-09-28 09:50:46.422273: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb84a00 of size 256 next 77
2022-09-28 09:50:46.422295: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb84b00 of size 256 next 78
2022-09-28 09:50:46.422317: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb84c00 of size 256 next 79
2022-09-28 09:50:46.422339: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb84d00 of size 256 next 80
2022-09-28 09:50:46.422361: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb84e00 of size 256 next 81
2022-09-28 09:50:46.422383: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb84f00 of size 256 next 82
2022-09-28 09:50:46.422405: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb85000 of size 256 next 83
2022-09-28 09:50:46.422427: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb85100 of size 256 next 84
2022-09-28 09:50:46.422448: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb85200 of size 256 next 85
2022-09-28 09:50:46.422475: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb85300 of size 512 next 86
2022-09-28 09:50:46.422499: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb85500 of size 512 next 87
2022-09-28 09:50:46.422521: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb85700 of size 256 next 88
2022-09-28 09:50:46.422542: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb85800 of size 256 next 89
2022-09-28 09:50:46.422564: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb85900 of size 512 next 90
2022-09-28 09:50:46.422586: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb85b00 of size 512 next 91
2022-09-28 09:50:46.422632: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb85d00 of size 256 next 92
2022-09-28 09:50:46.422655: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb85e00 of size 256 next 93
2022-09-28 09:50:46.422677: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb85f00 of size 512 next 94
2022-09-28 09:50:46.422699: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb86100 of size 512 next 95
2022-09-28 09:50:46.422721: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb86300 of size 256 next 96
2022-09-28 09:50:46.422775: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb86400 of size 256 next 97
2022-09-28 09:50:46.422800: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb86500 of size 512 next 98
2022-09-28 09:50:46.422822: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb86700 of size 512 next 99
2022-09-28 09:50:46.422843: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb86900 of size 256 next 100
2022-09-28 09:50:46.422865: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb86a00 of size 256 next 101
2022-09-28 09:50:46.422934: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb86b00 of size 512 next 102
2022-09-28 09:50:46.422958: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb86d00 of size 512 next 103
2022-09-28 09:50:46.422980: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb86f00 of size 256 next 104
2022-09-28 09:50:46.423002: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb87000 of size 256 next 105
2022-09-28 09:50:46.423024: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb87100 of size 256 next 106
2022-09-28 09:50:46.423059: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb87200 of size 256 next 107
2022-09-28 09:50:46.423088: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb87300 of size 512 next 108
2022-09-28 09:50:46.423116: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb87500 of size 512 next 109
2022-09-28 09:50:46.423148: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb87700 of size 256 next 110
2022-09-28 09:50:46.423178: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb87800 of size 256 next 111
2022-09-28 09:50:46.423206: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb87900 of size 512 next 112
2022-09-28 09:50:46.423235: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb87b00 of size 512 next 113
2022-09-28 09:50:46.423274: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb87d00 of size 256 next 114
2022-09-28 09:50:46.423304: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb87e00 of size 256 next 115
2022-09-28 09:50:46.423326: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb87f00 of size 512 next 116
2022-09-28 09:50:46.423346: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb88100 of size 512 next 117
2022-09-28 09:50:46.423367: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb88300 of size 256 next 118
2022-09-28 09:50:46.423388: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb88400 of size 256 next 119
2022-09-28 09:50:46.423408: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb88500 of size 512 next 120
2022-09-28 09:50:46.423428: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb88700 of size 512 next 121
2022-09-28 09:50:46.423448: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb88900 of size 512 next 122
2022-09-28 09:50:46.423469: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb88b00 of size 512 next 123
2022-09-28 09:50:46.423489: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb88d00 of size 256 next 124
2022-09-28 09:50:46.423518: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb88e00 of size 256 next 125
2022-09-28 09:50:46.423541: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb88f00 of size 512 next 126
2022-09-28 09:50:46.423561: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb89100 of size 512 next 127
2022-09-28 09:50:46.423591: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb89300 of size 256 next 128
2022-09-28 09:50:46.423612: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb89400 of size 256 next 129
2022-09-28 09:50:46.423632: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb89500 of size 256 next 130
2022-09-28 09:50:46.423653: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb89600 of size 256 next 131
2022-09-28 09:50:46.423674: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb89700 of size 512 next 132
2022-09-28 09:50:46.423694: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb89900 of size 512 next 133
2022-09-28 09:50:46.423714: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb89b00 of size 256 next 134
2022-09-28 09:50:46.423734: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb89c00 of size 256 next 135
2022-09-28 09:50:46.423755: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb89d00 of size 512 next 136
2022-09-28 09:50:46.423775: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb89f00 of size 512 next 137
2022-09-28 09:50:46.423818: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8a100 of size 256 next 138
2022-09-28 09:50:46.423841: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8a200 of size 256 next 139
2022-09-28 09:50:46.423861: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8a300 of size 512 next 140
2022-09-28 09:50:46.423881: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8a500 of size 512 next 141
2022-09-28 09:50:46.423901: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8a700 of size 256 next 142
2022-09-28 09:50:46.423942: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8a800 of size 256 next 143
2022-09-28 09:50:46.423964: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8a900 of size 512 next 144
2022-09-28 09:50:46.423985: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8ab00 of size 512 next 145
2022-09-28 09:50:46.424005: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8ad00 of size 256 next 146
2022-09-28 09:50:46.424027: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8ae00 of size 256 next 147
2022-09-28 09:50:46.424075: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8af00 of size 512 next 148
2022-09-28 09:50:46.424101: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8b100 of size 512 next 149
2022-09-28 09:50:46.424122: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8b300 of size 256 next 150
2022-09-28 09:50:46.430590: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8b400 of size 256 next 151
2022-09-28 09:50:46.430672: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8b500 of size 256 next 152
2022-09-28 09:50:46.430738: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8b600 of size 256 next 153
2022-09-28 09:50:46.430771: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8b700 of size 512 next 154
2022-09-28 09:50:46.430795: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8b900 of size 512 next 155
2022-09-28 09:50:46.430857: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8bb00 of size 256 next 156
2022-09-28 09:50:46.430896: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8bc00 of size 256 next 157
2022-09-28 09:50:46.430928: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8bd00 of size 512 next 158
2022-09-28 09:50:46.430955: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8bf00 of size 512 next 159
2022-09-28 09:50:46.431024: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8c100 of size 512 next 160
2022-09-28 09:50:46.431074: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8c300 of size 512 next 161
2022-09-28 09:50:46.431103: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8c500 of size 256 next 162
2022-09-28 09:50:46.431126: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8c600 of size 256 next 163
2022-09-28 09:50:46.431150: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8c700 of size 512 next 164
2022-09-28 09:50:46.431213: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8c900 of size 512 next 165
2022-09-28 09:50:46.431248: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8cb00 of size 256 next 166
2022-09-28 09:50:46.431296: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8cc00 of size 256 next 167
2022-09-28 09:50:46.431322: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8cd00 of size 512 next 168
2022-09-28 09:50:46.431346: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8cf00 of size 512 next 169
2022-09-28 09:50:46.431374: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8d100 of size 256 next 170
2022-09-28 09:50:46.431394: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8d200 of size 256 next 171
2022-09-28 09:50:46.431424: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8d300 of size 512 next 172
2022-09-28 09:50:46.431449: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8d500 of size 512 next 173
2022-09-28 09:50:46.431475: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8d700 of size 512 next 174
2022-09-28 09:50:46.431501: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8d900 of size 512 next 175
2022-09-28 09:50:46.431545: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8db00 of size 256 next 176
2022-09-28 09:50:46.431571: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8dc00 of size 256 next 177
2022-09-28 09:50:46.431595: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8dd00 of size 512 next 178
2022-09-28 09:50:46.431623: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8df00 of size 512 next 179
2022-09-28 09:50:46.431664: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8e100 of size 512 next 180
2022-09-28 09:50:46.431699: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8e300 of size 512 next 181
2022-09-28 09:50:46.431733: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8e500 of size 1024 next 182
2022-09-28 09:50:46.431781: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8e900 of size 1024 next 183
2022-09-28 09:50:46.431814: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8ed00 of size 512 next 184
2022-09-28 09:50:46.431839: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8ef00 of size 512 next 185
2022-09-28 09:50:46.431859: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8f100 of size 1024 next 186
2022-09-28 09:50:46.431879: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8f500 of size 1024 next 187
2022-09-28 09:50:46.431951: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8f900 of size 512 next 188
2022-09-28 09:50:46.431980: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8fb00 of size 512 next 189
2022-09-28 09:50:46.432027: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb8fd00 of size 1024 next 190
2022-09-28 09:50:46.432157: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb90100 of size 1024 next 191
2022-09-28 09:50:46.432197: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb90500 of size 512 next 192
2022-09-28 09:50:46.432229: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb90700 of size 512 next 193
2022-09-28 09:50:46.432288: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb90900 of size 256 next 194
2022-09-28 09:50:46.432318: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb90a00 of size 256 next 195
2022-09-28 09:50:46.432347: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb90b00 of size 1024 next 196
2022-09-28 09:50:46.432375: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb90f00 of size 1024 next 197
2022-09-28 09:50:46.432403: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb91300 of size 512 next 198
2022-09-28 09:50:46.432454: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb91500 of size 512 next 199
2022-09-28 09:50:46.432508: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb91700 of size 1024 next 200
2022-09-28 09:50:46.432533: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb91b00 of size 1024 next 201
2022-09-28 09:50:46.432560: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb91f00 of size 512 next 202
2022-09-28 09:50:46.432619: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb92100 of size 512 next 203
2022-09-28 09:50:46.432650: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb92300 of size 1024 next 204
2022-09-28 09:50:46.432679: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb92700 of size 1024 next 205
2022-09-28 09:50:46.432723: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb92b00 of size 512 next 206
2022-09-28 09:50:46.432754: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb92d00 of size 512 next 207
2022-09-28 09:50:46.432801: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb92f00 of size 1024 next 208
2022-09-28 09:50:46.432832: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb93300 of size 1024 next 209
2022-09-28 09:50:46.432860: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb93700 of size 512 next 210
2022-09-28 09:50:46.432889: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb93900 of size 512 next 211
2022-09-28 09:50:46.432918: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb93b00 of size 1024 next 212
2022-09-28 09:50:46.432947: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb93f00 of size 1024 next 213
2022-09-28 09:50:46.432976: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb94300 of size 512 next 214
2022-09-28 09:50:46.433018: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb94500 of size 512 next 215
2022-09-28 09:50:46.433042: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb94700 of size 512 next 216
2022-09-28 09:50:46.433088: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb94900 of size 512 next 217
2022-09-28 09:50:46.433110: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb94b00 of size 1024 next 218
2022-09-28 09:50:46.433132: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb94f00 of size 1024 next 219
2022-09-28 09:50:46.433153: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb95300 of size 256 next 220
2022-09-28 09:50:46.433201: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb95400 of size 256 next 221
2022-09-28 09:50:46.433223: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb95500 of size 512 next 222
2022-09-28 09:50:46.433244: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb95700 of size 512 next 223
2022-09-28 09:50:46.433292: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb95900 of size 1280 next 224
2022-09-28 09:50:46.433318: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb95e00 of size 4096 next 225
2022-09-28 09:50:46.433340: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb96e00 of size 256 next 248
2022-09-28 09:50:46.433363: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb96f00 of size 2048 next 239
2022-09-28 09:50:46.433384: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb97700 of size 256 next 237
2022-09-28 09:50:46.433412: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb97800 of size 9216 next 245
2022-09-28 09:50:46.433436: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb99c00 of size 256 next 238
2022-09-28 09:50:46.433458: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb99d00 of size 2048 next 230
2022-09-28 09:50:46.433530: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb9a500 of size 256 next 243
2022-09-28 09:50:46.433575: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bb9a600 of size 25600 next 241
2022-09-28 09:50:46.433613: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bba0a00 of size 256 next 226
2022-09-28 09:50:46.433637: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bba0b00 of size 55296 next 249
2022-09-28 09:50:46.433659: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbae300 of size 1024 next 236
2022-09-28 09:50:46.433749: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbae700 of size 256 next 252
2022-09-28 09:50:46.433774: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbae800 of size 512 next 246
2022-09-28 09:50:46.433796: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbaea00 of size 256 next 244
2022-09-28 09:50:46.433817: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbaeb00 of size 2304 next 240
2022-09-28 09:50:46.433838: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbaf400 of size 256 next 251
2022-09-28 09:50:46.433859: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbaf500 of size 512 next 233
2022-09-28 09:50:46.433881: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbaf700 of size 256 next 235
2022-09-28 09:50:46.434018: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbaf800 of size 6400 next 250
2022-09-28 09:50:46.434136: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb1100 of size 256 next 242
2022-09-28 09:50:46.434164: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb1200 of size 13824 next 228
2022-09-28 09:50:46.434222: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb4800 of size 256 next 247
2022-09-28 09:50:46.434266: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb4900 of size 256 next 227
2022-09-28 09:50:46.434311: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb4a00 of size 256 next 231
2022-09-28 09:50:46.434349: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb4b00 of size 256 next 229
2022-09-28 09:50:46.434392: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb4c00 of size 768 next 234
2022-09-28 09:50:46.434423: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb4f00 of size 256 next 232
2022-09-28 09:50:46.434453: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb5000 of size 256 next 253
2022-09-28 09:50:46.434484: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb5100 of size 256 next 254
2022-09-28 09:50:46.434514: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb5200 of size 1792 next 255
2022-09-28 09:50:46.434544: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb5900 of size 256 next 256
2022-09-28 09:50:46.434574: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb5a00 of size 2304 next 257
2022-09-28 09:50:46.434634: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb6300 of size 1536 next 258
2022-09-28 09:50:46.434672: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb6900 of size 256 next 259
2022-09-28 09:50:46.434707: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb6a00 of size 768 next 260
2022-09-28 09:50:46.434773: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb6d00 of size 256 next 261
2022-09-28 09:50:46.434811: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb6e00 of size 768 next 262
2022-09-28 09:50:46.434844: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb7100 of size 256 next 263
2022-09-28 09:50:46.434896: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb7200 of size 768 next 264
2022-09-28 09:50:46.434930: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb7500 of size 256 next 265
2022-09-28 09:50:46.434960: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb7600 of size 1792 next 266
2022-09-28 09:50:46.434989: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb7d00 of size 256 next 267
2022-09-28 09:50:46.435042: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb7e00 of size 6144 next 268
2022-09-28 09:50:46.435080: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb9600 of size 256 next 269
2022-09-28 09:50:46.435123: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbb9700 of size 3072 next 270
2022-09-28 09:50:46.435154: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbba300 of size 256 next 271
2022-09-28 09:50:46.435190: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbba400 of size 2304 next 272
2022-09-28 09:50:46.435219: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbbad00 of size 256 next 273
2022-09-28 09:50:46.435504: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbbae00 of size 3072 next 274
2022-09-28 09:50:46.435542: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbbba00 of size 256 next 275
2022-09-28 09:50:46.435905: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbbbb00 of size 6400 next 276
2022-09-28 09:50:46.435942: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbbd400 of size 256 next 277
2022-09-28 09:50:46.435964: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbbd500 of size 512 next 278
2022-09-28 09:50:46.435998: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbbd700 of size 256 next 279
2022-09-28 09:50:46.436026: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbbd800 of size 256 next 280
2022-09-28 09:50:46.436045: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbbd900 of size 256 next 281
2022-09-28 09:50:46.436063: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbbda00 of size 9216 next 282
2022-09-28 09:50:46.436092: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbbfe00 of size 256 next 283
2022-09-28 09:50:46.436114: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbbff00 of size 256 next 284
2022-09-28 09:50:46.436134: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbc0000 of size 256 next 285
2022-09-28 09:50:46.436155: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbc0100 of size 25600 next 286
2022-09-28 09:50:46.436184: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbc6500 of size 256 next 287
2022-09-28 09:50:46.436202: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbc6600 of size 512 next 288
2022-09-28 09:50:46.436222: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbc6800 of size 256 next 289
2022-09-28 09:50:46.436284: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbc6900 of size 6144 next 290
2022-09-28 09:50:46.436306: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbc8100 of size 256 next 291
2022-09-28 09:50:46.436327: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbc8200 of size 1536 next 292
2022-09-28 09:50:46.436347: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbc8800 of size 256 next 293
2022-09-28 09:50:46.436391: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 20bbc8900 of size 75497472 next 294
2022-09-28 09:50:46.436415: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 2103c8900 of size 768 next 295
2022-09-28 09:50:46.436434: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 2103c8c00 of size 25165824 next 296
2022-09-28 09:50:46.436455: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211bc8c00 of size 131072 next 297
2022-09-28 09:50:46.436516: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211be8c00 of size 256 next 298
2022-09-28 09:50:46.436539: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211be8d00 of size 256 next 299
2022-09-28 09:50:46.436556: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211be8e00 of size 256 next 300
2022-09-28 09:50:46.436575: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211be8f00 of size 256 next 301
2022-09-28 09:50:46.436594: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211be9000 of size 1024 next 302
2022-09-28 09:50:46.436614: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211be9400 of size 256 next 303
2022-09-28 09:50:46.436634: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211be9500 of size 256 next 304
2022-09-28 09:50:46.436652: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211be9600 of size 4096 next 305
2022-09-28 09:50:46.436672: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211bea600 of size 256 next 306
2022-09-28 09:50:46.436693: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211bea700 of size 10496 next 307
2022-09-28 09:50:46.436715: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211bed000 of size 256 next 308
2022-09-28 09:50:46.436736: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211bed100 of size 512 next 309
2022-09-28 09:50:46.436756: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211bed300 of size 256 next 310
2022-09-28 09:50:46.436776: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211bed400 of size 256 next 311
2022-09-28 09:50:46.436796: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211bed500 of size 256 next 312
2022-09-28 09:50:46.436816: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211bed600 of size 256 next 313
2022-09-28 09:50:46.436835: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211bed700 of size 512 next 314
2022-09-28 09:50:46.436856: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211bed900 of size 512 next 315
2022-09-28 09:50:46.436876: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211bedb00 of size 256 next 316
2022-09-28 09:50:46.436897: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211bedc00 of size 256 next 317
2022-09-28 09:50:46.436922: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211bedd00 of size 256 next 318
2022-09-28 09:50:46.437016: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211bede00 of size 256 next 319
2022-09-28 09:50:46.437058: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211bedf00 of size 512 next 320
2022-09-28 09:50:46.437088: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 211bee100 of size 1280 next 321
2022-09-28 09:50:46.437126: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] Free at 211bee600 of size 6291456 next 323
2022-09-28 09:50:46.437156: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 2121ee600 of size 33554432 next 324
2022-09-28 09:50:46.437187: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 2141ee600 of size 16777216 next 325
2022-09-28 09:50:46.437229: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] Free at 2151ee600 of size 16777216 next 326
2022-09-28 09:50:46.437271: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 2161ee600 of size 33554432 next 327
2022-09-28 09:50:46.437319: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] InUse at 2181ee600 of size 16777216 next 322
2022-09-28 09:50:46.437341: I tensorflow/core/common_runtime/bfc_allocator.cc:1069] Free at 2191ee600 of size 91015680 next 18446744073709551615
2022-09-28 09:50:46.437361: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] Summary of in-use Chunks by size:
2022-09-28 09:50:46.437411: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 116 Chunks of size 256 totalling 29.0KiB
2022-09-28 09:50:46.437465: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 78 Chunks of size 512 totalling 39.0KiB
2022-09-28 09:50:46.437490: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 5 Chunks of size 768 totalling 3.8KiB
2022-09-28 09:50:46.437512: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 20 Chunks of size 1024 totalling 20.0KiB
2022-09-28 09:50:46.437571: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 2 Chunks of size 1280 totalling 2.5KiB
2022-09-28 09:50:46.437618: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 2 Chunks of size 1536 totalling 3.0KiB
2022-09-28 09:50:46.437643: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 2 Chunks of size 1792 totalling 3.5KiB
2022-09-28 09:50:46.437665: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 2 Chunks of size 2048 totalling 4.0KiB
2022-09-28 09:50:46.437693: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 3 Chunks of size 2304 totalling 6.8KiB
2022-09-28 09:50:46.437716: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 2 Chunks of size 3072 totalling 6.0KiB
2022-09-28 09:50:46.437738: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 2 Chunks of size 4096 totalling 8.0KiB
2022-09-28 09:50:46.437760: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 2 Chunks of size 6144 totalling 12.0KiB
2022-09-28 09:50:46.437785: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 2 Chunks of size 6400 totalling 12.5KiB
2022-09-28 09:50:46.437807: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 1 Chunks of size 6912 totalling 6.8KiB
2022-09-28 09:50:46.437829: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 1 Chunks of size 8192 totalling 8.0KiB
2022-09-28 09:50:46.437854: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 2 Chunks of size 9216 totalling 18.0KiB
2022-09-28 09:50:46.437877: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 1 Chunks of size 10496 totalling 10.2KiB
2022-09-28 09:50:46.437899: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 1 Chunks of size 13824 totalling 13.5KiB
2022-09-28 09:50:46.437921: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 2 Chunks of size 25600 totalling 50.0KiB
2022-09-28 09:50:46.437945: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 22 Chunks of size 32768 totalling 704.0KiB
2022-09-28 09:50:46.437993: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 3 Chunks of size 33792 totalling 99.0KiB
2022-09-28 09:50:46.438017: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 1 Chunks of size 55296 totalling 54.0KiB
2022-09-28 09:50:46.438040: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 1 Chunks of size 65536 totalling 64.0KiB
2022-09-28 09:50:46.438068: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 1 Chunks of size 73728 totalling 72.0KiB
2022-09-28 09:50:46.438093: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 9 Chunks of size 131072 totalling 1.12MiB
2022-09-28 09:50:46.438116: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 1 Chunks of size 147456 totalling 144.0KiB
2022-09-28 09:50:46.438140: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 23 Chunks of size 294912 totalling 6.47MiB
2022-09-28 09:50:46.438162: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 3 Chunks of size 589824 totalling 1.69MiB
2022-09-28 09:50:46.438185: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 9 Chunks of size 1179648 totalling 10.12MiB
2022-09-28 09:50:46.438208: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 2 Chunks of size 16777216 totalling 32.00MiB
2022-09-28 09:50:46.438254: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 1 Chunks of size 25165824 totalling 24.00MiB
2022-09-28 09:50:46.439144: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 2 Chunks of size 33554432 totalling 64.00MiB
2022-09-28 09:50:46.439183: I tensorflow/core/common_runtime/bfc_allocator.cc:1077] 1 Chunks of size 75497472 totalling 72.00MiB
2022-09-28 09:50:46.439256: I tensorflow/core/common_runtime/bfc_allocator.cc:1081] Sum Total of in-use chunks: 212.77MiB
2022-09-28 09:50:46.439282: I tensorflow/core/common_runtime/bfc_allocator.cc:1083] total_region_allocated_bytes_: 337186816 memory_limit_: 337186816 available bytes: 0 curr_region_allocation_bytes_: 674373632
2022-09-28 09:50:46.439331: I tensorflow/core/common_runtime/bfc_allocator.cc:1089] Stats:
Limit: 337186816
InUse: 223102464
MaxInUse: 291845888
NumAllocs: 795
MaxAllocSize: 75497472
Reserved: 0
PeakReserved: 0
LargestFreeBlock: 0
2022-09-28 09:50:46.439426: W tensorflow/core/common_runtime/bfc_allocator.cc:474] *****_______________________________
2022-09-28 09:50:46.439577: W tensorflow/core/framework/op_kernel.cc:1692] OP_REQUIRES failed at concat_op.cc:158 : Resource exhausted: OOM when allocating tensor with shape[1,96,512,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Hi, Aasta:
I extracted a small of tensorflow script from my main code. Please refer to the attachment for the code.
test_tensorflow.py (1.3 KB)
Sorry I cannot upload the exe to you. It is too big (>100MB) to upload.
As the same as my main.py, this program can only run in Pycharm, but cannot run in exe which is built by Pyinstaller 4.2.
Hi,
Thanks for the sample.
Will share more information with you later.
Thanks.
Hi,
Sorry for the late update.
Does this issue only occur with pyinstaller?
We tried your sampe with TensorRT python directly and it can work well.
4# python3 test_tensorflow.py
2022-10-13 05:55:43.423324: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:977] could not open file to read NUMA node: /sys/bus/pci/devices/0000:00:00.0/numa_node
Your kernel may have been built without NUMA support.
2022-10-13 05:55:43.732036: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:977] could not open file to read NUMA node: /sys/bus/pci/devices/0000:00:00.0/numa_node
Your kernel may have been built without NUMA support.
2022-10-13 05:55:43.734079: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:977] could not open file to read NUMA node: /sys/bus/pci/devices/0000:00:00.0/numa_node
Your kernel may have been built without NUMA support.
2022-10-13 05:55:43.739333: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:977] could not open file to read NUMA node: /sys/bus/pci/devices/0000:00:00.0/numa_node
Your kernel may have been built without NUMA support.
2022-10-13 05:55:43.740167: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:977] could not open file to read NUMA node: /sys/bus/pci/devices/0000:00:00.0/numa_node
Your kernel may have been built without NUMA support.
2022-10-13 05:55:43.740921: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:977] could not open file to read NUMA node: /sys/bus/pci/devices/0000:00:00.0/numa_node
Your kernel may have been built without NUMA support.
2022-10-13 05:55:45.732145: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:977] could not open file to read NUMA node: /sys/bus/pci/devices/0000:00:00.0/numa_node
Your kernel may have been built without NUMA support.
2022-10-13 05:55:45.733162: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:977] could not open file to read NUMA node: /sys/bus/pci/devices/0000:00:00.0/numa_node
Your kernel may have been built without NUMA support.
2022-10-13 05:55:45.733362: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Could not identify NUMA node of platform GPU id 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2022-10-13 05:55:45.733839: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:977] could not open file to read NUMA node: /sys/bus/pci/devices/0000:00:00.0/numa_node
Your kernel may have been built without NUMA support.
2022-10-13 05:55:45.734068: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3547 MB memory: -> device: 0, name: Xavier, pci bus id: 0000:00:00.0, compute capability: 7.2
2022-10-13 05:55:46.124684: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled
2022-10-13 05:56:12.741518: I tensorflow/stream_executor/cuda/cuda_dnn.cc:384] Loaded cuDNN version 8401
2022-10-13 05:56:40.293763: W tensorflow/core/common_runtime/bfc_allocator.cc:290] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.62GiB with freed_by_count=0. The caller indicates that this is not a failure, but this may mean that there could be performance gains if more memory were available.
pass
Thanks.
Hi, Aasta:
Sorry for late response.
So far, I just used pyinstaller. The problem is still there. I also found a funny thing: even though the exe cannot run for the first few times, but if I try it more times, it can run finally. But not guaranteed. Sometime work sometime does not work. This makes me even more confused.
Can you please send your built exe to me for a trial? Do you have any advice?
Hi, Aasta:
Did you build the test_tensorflow.py to an executable with Pyinstaller then run it?
You said you tried the sample with TensorRT python directly. For test, you cannot run it directly from python. You need to build it to exe with Pyinstaller, then run the exe for test.
Hi,
Sorry that the comment is confusing.
We mean the testing with TensorRT python works well.
So it looks like this issue is specified to the pyInstaller, is that correct?
We are going to try this again, and share more information with you later.
More, do you mind sharing the executable through some online drive and posting the link here?
Thanks.
Hi, Aasta:
Please refer to the link below for the exe:
Hi,
Thanks for the sharing.
Since JetPack 5 also support Xavier NX, have you tried this on JetPack 5?
Thanks.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.