OOM - ResourceExhaustedError when running simple TF/BERT examples on Xavier

I previously tried to build Q&A system using ktrain but stumbeled on this OOM problem so I tried to drill down where it originated.I found this simple example to reproduce it:

source: transformers · PyPI

from transformers import AutoTokenizer, TFAutoModel

tokenizer = AutoTokenizer.from_pretrained(“bert-base-uncased”)
model = TFAutoModel.from_pretrained(“bert-base-uncased”)

result:

model=TFAutoModel.from_pretrained(“bert-base-uncased”)
2020-12-13 13:54:59.327929: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 93763584 exceeds 10% of free system memory.
2020-12-13 13:55:09.471465: W tensorflow/core/common_runtime/bfc_allocator.cc:431] Allocator (GPU_0_bfc) ran out of memory trying to allocate 89.42MiB (rounded to 93763584)requested by op AssignVariableOp
Current allocation summary follows.
2020-12-13 13:55:09.471697: I tensorflow/core/common_runtime/bfc_allocator.cc:970] BFCAllocator dump for GPU_0_bfc
2020-12-13 13:55:09.471802: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (256): Total Chunks: 12, Chunks in use: 11. 3.0KiB allocated for chunks. 2.8KiB in use in bin. 48B client-requested in use in bin.
2020-12-13 13:55:09.471903: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (512): Total Chunks: 1, Chunks in use: 0. 768B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-12-13 13:55:09.472025: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2020-12-13 13:55:09.472115: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (2048): Total Chunks: 109, Chunks in use: 109. 327.0KiB allocated for chunks. 327.0KiB in use in bin. 327.0KiB client-requested in use in bin.
2020-12-13 13:55:09.472238: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (4096): Total Chunks: 2, Chunks in use: 2. 10.5KiB allocated for chunks. 10.5KiB in use in bin. 6.0KiB client-requested in use in bin.
2020-12-13 13:55:09.472363: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (8192): Total Chunks: 14, Chunks in use: 12. 165.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2020-12-13 13:55:09.472454: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-12-13 13:55:09.472541: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (32768): Total Chunks: 1, Chunks in use: 0. 45.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-12-13 13:55:09.472651: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (65536): Total Chunks: 1, Chunks in use: 0. 78.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-12-13 13:55:09.472750: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-12-13 13:55:09.472865: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-12-13 13:55:09.473023: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (524288): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-12-13 13:55:09.473126: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-12-13 13:55:09.473272: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (2097152): Total Chunks: 52, Chunks in use: 49. 119.73MiB allocated for chunks. 113.22MiB in use in bin. 110.25MiB client-requested in use in bin.
2020-12-13 13:55:09.473367: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (4194304): Total Chunks: 1, Chunks in use: 0. 4.50MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-12-13 13:55:09.473467: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (8388608): Total Chunks: 25, Chunks in use: 24. 225.00MiB allocated for chunks. 216.00MiB in use in bin. 216.00MiB client-requested in use in bin.
2020-12-13 13:55:09.473525: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (16777216): Total Chunks: 1, Chunks in use: 0. 17.42MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-12-13 13:55:09.473572: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-12-13 13:55:09.473637: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (67108864): Total Chunks: 2, Chunks in use: 2. 192.53MiB allocated for chunks. 192.53MiB in use in bin. 178.84MiB client-requested in use in bin.
2020-12-13 13:55:09.473689: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-12-13 13:55:09.473735: I tensorflow/core/common_runtime/bfc_allocator.cc:977] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-12-13 13:55:09.473784: I tensorflow/core/common_runtime/bfc_allocator.cc:993] Bin for 89.42MiB was 64.00MiB, Chunk State:
2020-12-13 13:55:09.473825: I tensorflow/core/common_runtime/bfc_allocator.cc:1006] Next region of size 586981376
2020-12-13 13:55:09.473878: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db2a000 of size 1280 next 1
2020-12-13 13:55:09.473921: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db2a500 of size 256 next 2
2020-12-13 13:55:09.473960: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 21db2a600 of size 256 next 8
2020-12-13 13:55:09.474000: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db2a700 of size 256 next 9
2020-12-13 13:55:09.474046: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db2a800 of size 256 next 12
2020-12-13 13:55:09.474120: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 21db2a900 of size 768 next 64
2020-12-13 13:55:09.474162: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db2ac00 of size 3072 next 215
2020-12-13 13:55:09.474206: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db2b800 of size 3072 next 24
2020-12-13 13:55:09.474243: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db2c400 of size 3072 next 29
2020-12-13 13:55:09.474280: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db2d000 of size 3072 next 22
2020-12-13 13:55:09.474367: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db2dc00 of size 3072 next 32
2020-12-13 13:55:09.474409: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db2e800 of size 3072 next 27
2020-12-13 13:55:09.474480: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db2f400 of size 5632 next 34
2020-12-13 13:55:09.474522: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db30a00 of size 256 next 40
2020-12-13 13:55:09.474563: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db30b00 of size 12288 next 38
2020-12-13 13:55:09.474602: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db33b00 of size 3072 next 13
2020-12-13 13:55:09.474641: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db34700 of size 3072 next 46
2020-12-13 13:55:09.474678: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db35300 of size 3072 next 66
2020-12-13 13:55:09.474744: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db35f00 of size 3072 next 79
2020-12-13 13:55:09.474808: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db36b00 of size 3072 next 52
2020-12-13 13:55:09.474868: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db37700 of size 3072 next 50
2020-12-13 13:55:09.474926: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db38300 of size 3072 next 81
2020-12-13 13:55:09.474992: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db38f00 of size 3072 next 87
2020-12-13 13:55:09.475035: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db39b00 of size 3072 next 92
2020-12-13 13:55:09.475074: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db3a700 of size 3072 next 88
2020-12-13 13:55:09.475112: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db3b300 of size 12288 next 71
2020-12-13 13:55:09.475217: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db3e300 of size 3072 next 126
2020-12-13 13:55:09.475258: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db3ef00 of size 3072 next 84
2020-12-13 13:55:09.475296: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db3fb00 of size 3072 next 127
2020-12-13 13:55:09.475335: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db40700 of size 3072 next 90
2020-12-13 13:55:09.475374: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db41300 of size 3072 next 70
2020-12-13 13:55:09.475459: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db41f00 of size 3072 next 96
2020-12-13 13:55:09.475498: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db42b00 of size 3072 next 108
2020-12-13 13:55:09.475572: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db43700 of size 5120 next 11
2020-12-13 13:55:09.475612: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db44b00 of size 256 next 14
2020-12-13 13:55:09.475650: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db44c00 of size 3072 next 15
2020-12-13 13:55:09.475732: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db45800 of size 256 next 16
2020-12-13 13:55:09.475770: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db45900 of size 3072 next 17
2020-12-13 13:55:09.475807: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db46500 of size 256 next 10
2020-12-13 13:55:09.475844: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db46600 of size 256 next 20
2020-12-13 13:55:09.475883: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db46700 of size 256 next 19
2020-12-13 13:55:09.475919: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db46800 of size 12288 next 44
2020-12-13 13:55:09.475959: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db49800 of size 3072 next 25
2020-12-13 13:55:09.475998: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db4a400 of size 3072 next 93
2020-12-13 13:55:09.476036: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db4b000 of size 3072 next 82
2020-12-13 13:55:09.476073: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db4bc00 of size 3072 next 125
2020-12-13 13:55:09.476112: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db4c800 of size 3072 next 115
2020-12-13 13:55:09.476150: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db4d400 of size 3072 next 139
2020-12-13 13:55:09.476187: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db4e000 of size 3072 next 155
2020-12-13 13:55:09.476224: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db4ec00 of size 3072 next 149
2020-12-13 13:55:09.476261: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db4f800 of size 3072 next 145
2020-12-13 13:55:09.476298: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db50400 of size 12288 next 157
2020-12-13 13:55:09.476336: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db53400 of size 3072 next 190
2020-12-13 13:55:09.476373: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db54000 of size 3072 next 165
2020-12-13 13:55:09.476410: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db54c00 of size 3072 next 148
2020-12-13 13:55:09.476486: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db55800 of size 3072 next 133
2020-12-13 13:55:09.476532: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db56400 of size 3072 next 174
2020-12-13 13:55:09.476570: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db57000 of size 3072 next 159
2020-12-13 13:55:09.476660: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db57c00 of size 3072 next 173
2020-12-13 13:55:09.476701: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db58800 of size 3072 next 223
2020-12-13 13:55:09.476740: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db59400 of size 3072 next 147
2020-12-13 13:55:09.476827: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db5a000 of size 3072 next 220
2020-12-13 13:55:09.476869: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db5ac00 of size 3072 next 202
2020-12-13 13:55:09.476910: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 21db5b800 of size 79872 next 177
2020-12-13 13:55:09.476949: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db6f000 of size 3072 next 183
2020-12-13 13:55:09.476988: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db6fc00 of size 12288 next 184
2020-12-13 13:55:09.477028: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db72c00 of size 3072 next 153
2020-12-13 13:55:09.477067: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db73800 of size 3072 next 166
2020-12-13 13:55:09.477106: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db74400 of size 3072 next 195
2020-12-13 13:55:09.477145: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db75000 of size 3072 next 194
2020-12-13 13:55:09.477183: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db75c00 of size 3072 next 161
2020-12-13 13:55:09.477221: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db76800 of size 3072 next 200
2020-12-13 13:55:09.477260: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db77400 of size 3072 next 204
2020-12-13 13:55:09.477299: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db78000 of size 12288 next 203
2020-12-13 13:55:09.477337: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db7b000 of size 3072 next 207
2020-12-13 13:55:09.477376: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db7bc00 of size 3072 next 187
2020-12-13 13:55:09.477415: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db7c800 of size 3072 next 201
2020-12-13 13:55:09.477454: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db7d400 of size 3072 next 188
2020-12-13 13:55:09.477492: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db7e000 of size 3072 next 18
2020-12-13 13:55:09.477548: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 21db7ec00 of size 46080 next 28
2020-12-13 13:55:09.477589: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db8a000 of size 3072 next 41
2020-12-13 13:55:09.477628: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db8ac00 of size 3072 next 49
2020-12-13 13:55:09.477699: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db8b800 of size 3072 next 72
2020-12-13 13:55:09.477739: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db8c400 of size 3072 next 48
2020-12-13 13:55:09.477777: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db8d000 of size 3072 next 33
2020-12-13 13:55:09.477849: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db8dc00 of size 3072 next 55
2020-12-13 13:55:09.477888: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db8e800 of size 12288 next 103
2020-12-13 13:55:09.477968: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db91800 of size 3072 next 94
2020-12-13 13:55:09.478010: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db92400 of size 3072 next 111
2020-12-13 13:55:09.478050: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db93000 of size 3072 next 76
2020-12-13 13:55:09.478089: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db93c00 of size 3072 next 74
2020-12-13 13:55:09.478129: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db94800 of size 3072 next 57
2020-12-13 13:55:09.478168: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db95400 of size 3072 next 56
2020-12-13 13:55:09.478207: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db96000 of size 3072 next 58
2020-12-13 13:55:09.478245: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db96c00 of size 12288 next 112
2020-12-13 13:55:09.478284: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db99c00 of size 3072 next 144
2020-12-13 13:55:09.478322: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db9a800 of size 3072 next 124
2020-12-13 13:55:09.478361: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db9b400 of size 3072 next 137
2020-12-13 13:55:09.478399: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db9c000 of size 3072 next 132
2020-12-13 13:55:09.478437: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db9cc00 of size 3072 next 130
2020-12-13 13:55:09.478476: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21db9d800 of size 12288 next 129
2020-12-13 13:55:09.478515: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dba0800 of size 3072 next 110
2020-12-13 13:55:09.478553: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dba1400 of size 3072 next 168
2020-12-13 13:55:09.478605: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dba2000 of size 3072 next 164
2020-12-13 13:55:09.478646: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dba2c00 of size 3072 next 167
2020-12-13 13:55:09.478684: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dba3800 of size 3072 next 170
2020-12-13 13:55:09.478723: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dba4400 of size 12288 next 206
2020-12-13 13:55:09.478761: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dba7400 of size 3072 next 162
2020-12-13 13:55:09.478837: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dba8000 of size 3072 next 152
2020-12-13 13:55:09.478879: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dba8c00 of size 3072 next 179
2020-12-13 13:55:09.478917: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dba9800 of size 3072 next 160
2020-12-13 13:55:09.478992: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbaa400 of size 3072 next 185
2020-12-13 13:55:09.479033: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbab000 of size 3072 next 180
2020-12-13 13:55:09.479072: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbabc00 of size 3072 next 186
2020-12-13 13:55:09.479211: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbac800 of size 3072 next 99
2020-12-13 13:55:09.479259: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbad400 of size 3072 next 106
2020-12-13 13:55:09.479292: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbae000 of size 3072 next 105
2020-12-13 13:55:09.479323: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbaec00 of size 3072 next 109
2020-12-13 13:55:09.479354: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbaf800 of size 3072 next 121
2020-12-13 13:55:09.479384: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbb0400 of size 3072 next 116
2020-12-13 13:55:09.479415: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbb1000 of size 3072 next 102
2020-12-13 13:55:09.479446: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbb1c00 of size 3072 next 104
2020-12-13 13:55:09.479477: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbb2800 of size 12288 next 158
2020-12-13 13:55:09.479508: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbb5800 of size 3072 next 143
2020-12-13 13:55:09.479539: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbb6400 of size 3072 next 98
2020-12-13 13:55:09.479569: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbb7000 of size 3072 next 135
2020-12-13 13:55:09.479600: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbb7c00 of size 3072 next 119
2020-12-13 13:55:09.479642: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbb8800 of size 3072 next 62
2020-12-13 13:55:09.479677: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbb9400 of size 3072 next 141
2020-12-13 13:55:09.479708: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbba000 of size 3072 next 118
2020-12-13 13:55:09.479738: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbbac00 of size 12288 next 198
2020-12-13 13:55:09.479769: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbbdc00 of size 3072 next 228
2020-12-13 13:55:09.479799: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbbe800 of size 3072 next 193
2020-12-13 13:55:09.479834: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbbf400 of size 3072 next 205
2020-12-13 13:55:09.479870: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 21dbc0000 of size 9216 next 210
2020-12-13 13:55:09.479902: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbc2400 of size 3072 next 225
2020-12-13 13:55:09.479965: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 21dbc3000 of size 12288 next 211
2020-12-13 13:55:09.479997: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbc6000 of size 3072 next 217
2020-12-13 13:55:09.480028: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21dbc6c00 of size 3072 next 214
2020-12-13 13:55:09.480097: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 21dbc7800 of size 2107392 next 78
2020-12-13 13:55:09.480130: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21ddca000 of size 2359296 next 59
2020-12-13 13:55:09.480164: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21e00a000 of size 2359296 next 30
2020-12-13 13:55:09.480234: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21e24a000 of size 2359296 next 35
2020-12-13 13:55:09.480267: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21e48a000 of size 2359296 next 36
2020-12-13 13:55:09.480298: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 21e6ca000 of size 2359296 next 37
2020-12-13 13:55:09.480330: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21e90a000 of size 2359296 next 216
2020-12-13 13:55:09.480362: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21eb4a000 of size 2359296 next 68
2020-12-13 13:55:09.480394: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21ed8a000 of size 2359296 next 54
2020-12-13 13:55:09.480425: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 21efca000 of size 2359296 next 51
2020-12-13 13:55:09.480456: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21f20a000 of size 2359296 next 63
2020-12-13 13:55:09.480487: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21f44a000 of size 2359296 next 45
2020-12-13 13:55:09.480518: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21f68a000 of size 2359296 next 60
2020-12-13 13:55:09.480549: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21f8ca000 of size 2359296 next 67
2020-12-13 13:55:09.480580: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21fb0a000 of size 2359296 next 43
2020-12-13 13:55:09.480611: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 21fd4a000 of size 9437184 next 26
2020-12-13 13:55:09.480644: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 22064a000 of size 9437184 next 80
2020-12-13 13:55:09.480681: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 220f4a000 of size 2359296 next 23
2020-12-13 13:55:09.480714: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 22118a000 of size 2359296 next 61
2020-12-13 13:55:09.480745: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 2213ca000 of size 2359296 next 85
2020-12-13 13:55:09.480776: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 22160a000 of size 2359296 next 65
2020-12-13 13:55:09.480807: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 22184a000 of size 9437184 next 53
2020-12-13 13:55:09.480838: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 22214a000 of size 9437184 next 86
2020-12-13 13:55:09.480869: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 222a4a000 of size 2359296 next 97
2020-12-13 13:55:09.480900: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 222c8a000 of size 2359296 next 122
2020-12-13 13:55:09.480964: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 222eca000 of size 2359296 next 134
2020-12-13 13:55:09.481005: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 22310a000 of size 3718656 next 3
2020-12-13 13:55:09.481040: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 223495e00 of size 256 next 4
2020-12-13 13:55:09.481071: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 223495f00 of size 9437184 next 101
2020-12-13 13:55:09.481133: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 223d95f00 of size 9437184 next 75
2020-12-13 13:55:09.481165: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 224695f00 of size 2359296 next 47
2020-12-13 13:55:09.481196: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 2248d5f00 of size 2359296 next 73
2020-12-13 13:55:09.481226: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 224b15f00 of size 2359296 next 95
2020-12-13 13:55:09.481283: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 224d55f00 of size 2359296 next 39
2020-12-13 13:55:09.481316: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 224f95f00 of size 9437184 next 128
2020-12-13 13:55:09.481346: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 225895f00 of size 9437184 next 89
2020-12-13 13:55:09.481377: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 226195f00 of size 2359296 next 91
2020-12-13 13:55:09.481408: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 2263d5f00 of size 2359296 next 120
2020-12-13 13:55:09.481438: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 226615f00 of size 2359296 next 113
2020-12-13 13:55:09.481469: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 226855f00 of size 2359296 next 83
2020-12-13 13:55:09.481499: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 226a95f00 of size 9437184 next 77
2020-12-13 13:55:09.481529: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 227395f00 of size 9437184 next 69
2020-12-13 13:55:09.481559: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 227c95f00 of size 2359296 next 140
2020-12-13 13:55:09.481589: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 227ed5f00 of size 2359296 next 163
2020-12-13 13:55:09.481620: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 228115f00 of size 2359296 next 175
2020-12-13 13:55:09.481651: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 228355f00 of size 2359296 next 181
2020-12-13 13:55:09.482728: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 228595f00 of size 2359296 next 146
2020-12-13 13:55:09.482789: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 2287d5f00 of size 2359296 next 221
2020-12-13 13:55:09.482845: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 228a15f00 of size 4110336 next 5
2020-12-13 13:55:09.482899: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 228e01700 of size 256 next 6
2020-12-13 13:55:09.482956: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 228e01800 of size 9437184 next 150
2020-12-13 13:55:09.483010: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 229701800 of size 9437184 next 117
2020-12-13 13:55:09.483064: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 22a001800 of size 2359296 next 100
2020-12-13 13:55:09.483148: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 22a241800 of size 2359296 next 142
2020-12-13 13:55:09.483210: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 22a481800 of size 2359296 next 151
2020-12-13 13:55:09.483263: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 22a6c1800 of size 2359296 next 114
2020-12-13 13:55:09.483316: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 22a901800 of size 9437184 next 107
2020-12-13 13:55:09.483374: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 22b201800 of size 9437184 next 7
2020-12-13 13:55:09.483438: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 22bb01800 of size 2359296 next 131
2020-12-13 13:55:09.483493: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 22bd41800 of size 2359296 next 156
2020-12-13 13:55:09.483574: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 22bf81800 of size 2359296 next 169
2020-12-13 13:55:09.483621: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 22c1c1800 of size 2359296 next 138
2020-12-13 13:55:09.483665: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 22c401800 of size 9437184 next 123
2020-12-13 13:55:09.483747: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 22cd01800 of size 9437184 next 154
2020-12-13 13:55:09.483800: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 22d601800 of size 18266112 next 219
2020-12-13 13:55:09.483845: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 22e76d000 of size 93763584 next 176
2020-12-13 13:55:09.483928: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 2340d8800 of size 9437184 next 21
2020-12-13 13:55:09.483976: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 2349d8800 of size 9437184 next 136
2020-12-13 13:55:09.484021: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 2352d8800 of size 2359296 next 199
2020-12-13 13:55:09.484068: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 235518800 of size 2359296 next 192
2020-12-13 13:55:09.484140: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 235758800 of size 2359296 next 189
2020-12-13 13:55:09.484186: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 235998800 of size 2359296 next 191
2020-12-13 13:55:09.484229: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 235bd8800 of size 9437184 next 171
2020-12-13 13:55:09.484301: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 2364d8800 of size 9437184 next 212
2020-12-13 13:55:09.484346: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 236dd8800 of size 2359296 next 222
2020-12-13 13:55:09.484390: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 237018800 of size 2359296 next 42
2020-12-13 13:55:09.484433: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 237258800 of size 4718592 next 182
2020-12-13 13:55:09.484476: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 2376d8800 of size 9437184 next 172
2020-12-13 13:55:09.484520: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 237fd8800 of size 9437184 next 209
2020-12-13 13:55:09.491241: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] Free at 2388d8800 of size 9437184 next 196
2020-12-13 13:55:09.491378: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 2391d8800 of size 9437184 next 218
2020-12-13 13:55:09.491478: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 239ad8800 of size 9437184 next 31
2020-12-13 13:55:09.491581: I tensorflow/core/common_runtime/bfc_allocator.cc:1026] InUse at 23a3d8800 of size 108115968 next 18446744073709551615
2020-12-13 13:55:09.491628: I tensorflow/core/common_runtime/bfc_allocator.cc:1031] Summary of in-use Chunks by size:
2020-12-13 13:55:09.491748: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 11 Chunks of size 256 totalling 2.8KiB
2020-12-13 13:55:09.491857: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 1280 totalling 1.2KiB
2020-12-13 13:55:09.491907: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 109 Chunks of size 3072 totalling 327.0KiB
2020-12-13 13:55:09.491954: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 5120 totalling 5.0KiB
2020-12-13 13:55:09.491997: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 5632 totalling 5.5KiB
2020-12-13 13:55:09.492040: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 12 Chunks of size 12288 totalling 144.0KiB
2020-12-13 13:55:09.492083: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 47 Chunks of size 2359296 totalling 105.75MiB
2020-12-13 13:55:09.492126: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 3718656 totalling 3.55MiB
2020-12-13 13:55:09.492170: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 4110336 totalling 3.92MiB
2020-12-13 13:55:09.492253: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 24 Chunks of size 9437184 totalling 216.00MiB
2020-12-13 13:55:09.492331: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 93763584 totalling 89.42MiB
2020-12-13 13:55:09.492410: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 108115968 totalling 103.11MiB
2020-12-13 13:55:09.492456: I tensorflow/core/common_runtime/bfc_allocator.cc:1038] Sum Total of in-use chunks: 522.22MiB
2020-12-13 13:55:09.492495: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] total_region_allocated_bytes_: 586981376 memory_limit_: 586981376 available bytes: 0 curr_region_allocation_bytes_: 1173962752
2020-12-13 13:55:09.492548: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Stats:
Limit: 586981376
InUse: 547585024
MaxInUse: 555047680
NumAllocs: 4236
MaxAllocSize: 108115968
Reserved: 0
PeakReserved: 0
LargestFreeBlock: 0

2020-12-13 13:55:09.492660: W tensorflow/core/common_runtime/bfc_allocator.cc:439] ***************___********************xx
2020-12-13 13:55:09.492752: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at resource_variable_ops.cc:410 : Resource exhausted: OOM when allocating tensor with shape[30522,768] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File “”, line 1, in
File “/usr/local/lib/python3.6/dist-packages/transformers/models/auto/modeling_tf_auto.py”, line 561, in from_pretrained
pretrained_model_name_or_path, *model_args, config=config, **kwargs
File “/usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_utils.py”, line 736, in from_pretrained
missing_keys, unexpected_keys = load_tf_weights(model, resolved_archive_file)
File “/usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_utils.py”, line 321, in load_tf_weights
K.batch_set_value(weight_value_tuples)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/dispatch.py”, line 201, in wrapper
return target(*args, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/backend.py”, line 3576, in batch_set_value
x.assign(np.asarray(value, dtype=dtype(x)))
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py”, line 860, in assign
self.handle, value_tensor, name=name)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_resource_variable_ops.py”, line 147, in assign_variable_op
_ops.raise_from_not_ok_status(e, name)
File “/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py”, line 6843, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File “”, line 3, in raise_from
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[30522,768] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:AssignVariableOp]

tegrastats extract:
sudo tegrastats
RAM 5587/7772MB (lfb 140x4MB) SWAP 83/7982MB (cached 1MB) CPU [10%'AT’1190,6%'AT’1190,5%'AT’1190,2%'AT’1190,1%'AT’1190,0%'AT’1420] EMC_FREQ 0%'AT’1600 GR3D_FREQ 0%'AT’114 APE 150 MTS fg 0% bg 1% AO’AT’40C GPU’AT’41C PMIC’AT’100C AUX’AT’41C CPU’AT’42C thermal’AT’41.3C VDD_IN 4007/4007 VDD_CPU_GPU_CV 408/408 VDD_SOC 1102/1102
RAM 5588/7772MB (lfb 140x4MB) SWAP 83/7982MB (cached 1MB) CPU [14%'AT’1190,5%'AT’1190,5%'AT’1190,7%'AT’1344,2%'AT’1343,6%'AT’1420] EMC_FREQ 0%'AT’1600 GR3D_FREQ 0%'AT’114 APE 150 MTS fg 0% bg 2% AO’AT’40.5C GPU’AT’41C PMIC’AT’100C AUX’AT’41C CPU’AT’42C thermal’AT’41.45C VDD_IN 4089/4048 VDD_CPU_GPU_CV 449/428 VDD_SOC 1102/1102
RAM 5590/7772MB (lfb 140x4MB) SWAP 83/7982MB (cached 1MB) CPU [10%'AT’1420,1%'AT’1420,0%'AT’1420,0%'AT’1420,15%'AT’1420,10%'AT’1420] EMC_FREQ 0%'AT’1600 GR3D_FREQ 14%'AT’114 APE 150 MTS fg 0% bg 2% AO’AT’40C GPU’AT’41.5C PMIC’AT’100C AUX’AT’41C CPU’AT’42.5C thermal’AT’41.45C VDD_IN 4130/4075 VDD_CPU_GPU_CV 490/449 VDD_SOC 1102/1102
RAM 5588/7772MB (lfb 140x4MB) SWAP 83/7982MB (cached 1MB) CPU [16%'AT’1190,5%'AT’1190,2%'AT’1190,7%'AT’1190,7%'AT’1420,9%'AT’1420] EMC_FREQ 0%'AT’1600 GR3D_FREQ 0%'AT’114 APE 150 MTS fg 0% bg 3% AO’AT’40.5C GPU’AT’41.5C PMIC’AT’100C AUX’AT’41C CPU’AT’42.5C thermal’AT’41.6C VDD_IN 4089/4078 VDD_CPU_GPU_CV 449/449 VDD_SOC 1102/1102
RAM 5600/7772MB (lfb 139x4MB) SWAP 83/7982MB (cached 1MB) CPU [19%'AT’1420,13%'AT’1420,4%'AT’1420,13%'AT’1420,18%'AT’1420,13%'AT’1420] EMC_FREQ 1%'AT’1600 GR3D_FREQ 15%'AT’114 APE 150 MTS fg 0% bg 4% AO’AT’40.5C GPU’AT’41.5C PMIC’AT’100C AUX’AT’41C CPU’AT’43C thermal’AT’41.6C VDD_IN 4621/4187 VDD_CPU_GPU_CV 858/530 VDD_SOC 1143/1110
RAM 5592/7772MB (lfb 140x4MB) SWAP 83/7982MB (cached 1MB) CPU [24%'AT’1420,12%'AT’1420,33%'AT’1420,18%'AT’1420,15%'AT’1420,14%'AT’1420] EMC_FREQ 3%'AT’1600 GR3D_FREQ 27%'AT’114 APE 150 MTS fg 0% bg 5% AO’AT’40.5C GPU’AT’41.5C PMIC’AT’100C AUX’AT’41.5C CPU’AT’42.5C thermal’AT’41.6C VDD_IN 4743/4279 VDD_CPU_GPU_CV 940/599 VDD_SOC 1184/1122
RAM 5588/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [25%'AT’1190,11%'AT’1190,20%'AT’1190,18%'AT’1190,20%'AT’1190,16%'AT’1420] EMC_FREQ 3%'AT’1600 GR3D_FREQ 0%'AT’114 APE 150 MTS fg 0% bg 5% AO’AT’40.5C GPU’AT’41.5C PMIC’AT’100C AUX’AT’41.5C CPU’AT’42.5C thermal’AT’41.95C VDD_IN 4457/4305 VDD_CPU_GPU_CV 736/618 VDD_SOC 1143/1125
RAM 5591/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [23%'AT’1420,12%'AT’1420,16%'AT’1420,6%'AT’1420,14%'AT’1420,11%'AT’1420] EMC_FREQ 2%'AT’1600 GR3D_FREQ 0%'AT’114 APE 150 MTS fg 0% bg 4% AO’AT’41C GPU’AT’41.5C PMIC’AT’100C AUX’AT’41.5C CPU’AT’43C thermal’AT’41.95C VDD_IN 4416/4319 VDD_CPU_GPU_CV 695/628 VDD_SOC 1143/1127
RAM 5590/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [17%'AT’1190,13%'AT’1190,2%'AT’1190,1%'AT’1190,8%'AT’1190,12%'AT’1190] EMC_FREQ 2%'AT’1600 GR3D_FREQ 0%'AT’114 APE 150 MTS fg 0% bg 2% AO’AT’41C GPU’AT’42C PMIC’AT’100C AUX’AT’41.5C CPU’AT’42.5C thermal’AT’41.95C VDD_IN 4171/4302 VDD_CPU_GPU_CV 531/617 VDD_SOC 1102/1124
RAM 5587/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [28%'AT’1190,25%'AT’1190,26%'AT’1190,20%'AT’1190,18%'AT’1267,11%'AT’1267] EMC_FREQ 3%'AT’1600 GR3D_FREQ 69%'AT’204 APE 150 MTS fg 0% bg 8% AO’AT’41C GPU’AT’42C PMIC’AT’100C AUX’AT’42C CPU’AT’43C thermal’AT’42.1C VDD_IN 4662/4338 VDD_CPU_GPU_CV 858/641 VDD_SOC 1143/1126
RAM 5588/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [21%'AT’1190,9%'AT’1190,11%'AT’1190,17%'AT’1190,6%'AT’1190,13%'AT’1190] EMC_FREQ 3%'AT’1600 GR3D_FREQ 0%'AT’204 APE 150 MTS fg 0% bg 3% AO’AT’41C GPU’AT’42C PMIC’AT’100C AUX’AT’42C CPU’AT’42.5C thermal’AT’42.3C VDD_IN 4253/4330 VDD_CPU_GPU_CV 572/635 VDD_SOC 1102/1124
RAM 5587/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [20%'AT’1190,11%'AT’1190,0%'AT’1190,0%'AT’1267,3%'AT’1267,7%'AT’1420] EMC_FREQ 1%'AT’1600 GR3D_FREQ 0%'AT’204 APE 150 MTS fg 0% bg 2% AO’AT’41C GPU’AT’42C PMIC’AT’100C AUX’AT’42C CPU’AT’43C thermal’AT’42.45C VDD_IN 4130/4314 VDD_CPU_GPU_CV 490/623 VDD_SOC 1102/1122
RAM 5588/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [25%'AT’1190,13%'AT’1190,10%'AT’1190,12%'AT’1190,8%'AT’1190,9%'AT’1190] EMC_FREQ 2%'AT’1600 GR3D_FREQ 0%'AT’204 APE 150 MTS fg 0% bg 4% AO’AT’41C GPU’AT’42C PMIC’AT’100C AUX’AT’42C CPU’AT’43C thermal’AT’42.45C VDD_IN 4294/4312 VDD_CPU_GPU_CV 654/625 VDD_SOC 1143/1124
RAM 5592/7772MB (lfb 140x4MB) SWAP 83/7982MB (cached 1MB) CPU [13%'AT’1190,6%'AT’1190,1%'AT’1190,5%'AT’1190,18%'AT’1190,6%'AT’1190] EMC_FREQ 1%'AT’1600 GR3D_FREQ 14%'AT’204 APE 150 MTS fg 0% bg 3% AO’AT’41C GPU’AT’42.5C PMIC’AT’100C AUX’AT’42C CPU’AT’43C thermal’AT’42.45C VDD_IN 4294/4311 VDD_CPU_GPU_CV 613/624 VDD_SOC 1143/1125
RAM 5586/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [20%'AT’1190,8%'AT’1190,9%'AT’1190,2%'AT’1190,2%'AT’1190,1%'AT’1190] EMC_FREQ 1%'AT’1600 GR3D_FREQ 0%'AT’204 APE 150 MTS fg 0% bg 1% AO’AT’41.5C GPU’AT’42.5C PMIC’AT’100C AUX’AT’42C CPU’AT’43C thermal’AT’42.45C VDD_IN 4130/4299 VDD_CPU_GPU_CV 490/615 VDD_SOC 1102/1123
RAM 5591/7772MB (lfb 140x4MB) SWAP 83/7982MB (cached 1MB) CPU [35%'AT’1190,7%'AT’1267,5%'AT’1267,7%'AT’1420,10%'AT’1420,10%'AT’1420] EMC_FREQ 2%'AT’1600 GR3D_FREQ 50%'AT’204 APE 150 MTS fg 0% bg 3% AO’AT’41.5C GPU’AT’42.5C PMIC’AT’100C AUX’AT’42C CPU’AT’43.5C thermal’AT’42.45C VDD_IN 4498/4311 VDD_CPU_GPU_CV 777/625 VDD_SOC 1143/1125
RAM 5588/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [17%'AT’1190,7%'AT’1190,13%'AT’1190,10%'AT’1190,8%'AT’1190,10%'AT’1190] EMC_FREQ 2%'AT’1600 GR3D_FREQ 0%'AT’204 APE 150 MTS fg 0% bg 3% AO’AT’41.5C GPU’AT’42.5C PMIC’AT’100C AUX’AT’42C CPU’AT’43.5C thermal’AT’42.6C VDD_IN 4294/4310 VDD_CPU_GPU_CV 613/624 VDD_SOC 1143/1126
RAM 5588/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [29%'AT’1420,12%'AT’1420,6%'AT’1420,6%'AT’1420,6%'AT’1420,4%'AT’1420] EMC_FREQ 1%'AT’1600 GR3D_FREQ 15%'AT’204 APE 150 MTS fg 0% bg 2% AO’AT’41.5C GPU’AT’42.5C PMIC’AT’100C AUX’AT’42.5C CPU’AT’43.5C thermal’AT’42.45C VDD_IN 4457/4318 VDD_CPU_GPU_CV 736/631 VDD_SOC 1143/1127
RAM 5595/7772MB (lfb 140x4MB) SWAP 83/7982MB (cached 1MB) CPU [24%'AT’1420,22%'AT’1420,20%'AT’1420,19%'AT’1420,17%'AT’1420,22%'AT’1420] EMC_FREQ 3%'AT’1600 GR3D_FREQ 10%'AT’306 APE 150 MTS fg 0% bg 7% AO’AT’41.5C GPU’AT’42.5C PMIC’AT’100C AUX’AT’42.5C CPU’AT’43.5C thermal’AT’42.8C VDD_IN 4662/4336 VDD_CPU_GPU_CV 899/645 VDD_SOC 1184/1130
RAM 5590/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [29%'AT’1190,20%'AT’1190,9%'AT’1190,13%'AT’1190,16%'AT’1190,22%'AT’1343] EMC_FREQ 4%'AT’1600 GR3D_FREQ 0%'AT’306 APE 150 MTS fg 0% bg 6% AO’AT’41.5C GPU’AT’42.5C PMIC’AT’100C AUX’AT’42.5C CPU’AT’43.5C thermal’AT’42.8C VDD_IN 4580/4348 VDD_CPU_GPU_CV 817/653 VDD_SOC 1143/1130
RAM 5589/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [20%'AT’1217,10%'AT’1420,10%'AT’1420,11%'AT’1420,9%'AT’1420,7%'AT’1420] EMC_FREQ 3%'AT’1600 GR3D_FREQ 6%'AT’306 APE 150 MTS fg 0% bg 4% AO’AT’42C GPU’AT’42.5C PMIC’AT’100C AUX’AT’42.5C CPU’AT’44C thermal’AT’42.8C VDD_IN 4416/4352 VDD_CPU_GPU_CV 695/655 VDD_SOC 1143/1131
RAM 5587/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [17%'AT’1190,10%'AT’1190,8%'AT’1190,10%'AT’1190,12%'AT’1190,7%'AT’1267] EMC_FREQ 2%'AT’1600 GR3D_FREQ 0%'AT’306 APE 150 MTS fg 0% bg 3% AO’AT’42C GPU’AT’42.5C PMIC’AT’100C AUX’AT’42.5C CPU’AT’43.5C thermal’AT’42.95C VDD_IN 4212/4345 VDD_CPU_GPU_CV 572/651 VDD_SOC 1143/1131
RAM 5589/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [25%'AT’1190,9%'AT’1190,3%'AT’1190,3%'AT’1190,3%'AT’1420,5%'AT’1420] EMC_FREQ 1%'AT’1600 GR3D_FREQ 0%'AT’306 APE 150 MTS fg 0% bg 3% AO’AT’41.5C GPU’AT’43C PMIC’AT’100C AUX’AT’42.5C CPU’AT’43.5C thermal’AT’42.95C VDD_IN 4171/4338 VDD_CPU_GPU_CV 531/646 VDD_SOC 1102/1130
RAM 5590/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [20%'AT’1190,13%'AT’1190,2%'AT’1190,2%'AT’1190,2%'AT’1190,1%'AT’1190] EMC_FREQ 1%'AT’1600 GR3D_FREQ 0%'AT’306 APE 150 MTS fg 0% bg 2% AO’AT’41.5C GPU’AT’43C PMIC’AT’100C AUX’AT’42.5C CPU’AT’43.5C thermal’AT’42.95C VDD_IN 4212/4332 VDD_CPU_GPU_CV 531/641 VDD_SOC 1102/1129
RAM 5590/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [35%'AT’1190,17%'AT’1190,16%'AT’1190,15%'AT’1190,25%'AT’1190,21%'AT’1267] EMC_FREQ 3%'AT’1600 GR3D_FREQ 1%'AT’306 APE 150 MTS fg 0% bg 7% AO’AT’41.5C GPU’AT’43C PMIC’AT’100C AUX’AT’43C CPU’AT’44C thermal’AT’43.3C VDD_IN 4825/4352 VDD_CPU_GPU_CV 981/655 VDD_SOC 1185/1131
RAM 5590/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [20%'AT’1190,8%'AT’1190,4%'AT’1190,10%'AT’1267,10%'AT’1267,4%'AT’1342] EMC_FREQ 2%'AT’1600 GR3D_FREQ 1%'AT’306 APE 150 MTS fg 0% bg 3% AO’AT’42C GPU’AT’43C PMIC’AT’100C AUX’AT’42.5C CPU’AT’44C thermal’AT’43.3C VDD_IN 4416/4354 VDD_CPU_GPU_CV 736/658 VDD_SOC 1143/1132
RAM 5589/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [25%'AT’1190,9%'AT’1190,5%'AT’1190,4%'AT’1190,3%'AT’1190,2%'AT’1190] EMC_FREQ 1%'AT’1600 GR3D_FREQ 0%'AT’306 APE 150 MTS fg 0% bg 1% AO’AT’42C GPU’AT’43C PMIC’AT’100C AUX’AT’43C CPU’AT’44C thermal’AT’43.3C VDD_IN 4212/4349 VDD_CPU_GPU_CV 572/655 VDD_SOC 1143/1132
RAM 5591/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [26%'AT’1190,17%'AT’1190,9%'AT’1190,4%'AT’1190,16%'AT’1190,11%'AT’1420] EMC_FREQ 2%'AT’1600 GR3D_FREQ 0%'AT’306 APE 150 MTS fg 0% bg 3% AO’AT’42C GPU’AT’43C PMIC’AT’100C AUX’AT’43C CPU’AT’44.5C thermal’AT’43.1C VDD_IN 4621/4359 VDD_CPU_GPU_CV 858/662 VDD_SOC 1184/1134
RAM 5597/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [23%'AT’1190,21%'AT’1190,16%'AT’1190,14%'AT’1190,11%'AT’1190,13%'AT’1267] EMC_FREQ 2%'AT’1600 GR3D_FREQ 0%'AT’306 APE 150 MTS fg 0% bg 5% AO’AT’42.5C GPU’AT’43.5C PMIC’AT’100C AUX’AT’43C CPU’AT’44C thermal’AT’43.6C VDD_IN 4621/4368 VDD_CPU_GPU_CV 858/669 VDD_SOC 1145/1134
RAM 5595/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [20%'AT’1190,15%'AT’1190,17%'AT’1283,4%'AT’1344,11%'AT’1190,8%'AT’1190] EMC_FREQ 2%'AT’1600 GR3D_FREQ 0%'AT’306 APE 150 MTS fg 0% bg 4% AO’AT’42.5C GPU’AT’43.5C PMIC’AT’100C AUX’AT’43C CPU’AT’44.5C thermal’AT’43.3C VDD_IN 4580/4375 VDD_CPU_GPU_CV 817/674 VDD_SOC 1143/1134
RAM 5596/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [21%'AT’1190,16%'AT’1190,6%'AT’1190,14%'AT’1190,7%'AT’1190,6%'AT’1190] EMC_FREQ 2%'AT’1600 GR3D_FREQ 0%'AT’306 APE 150 MTS fg 0% bg 3% AO’AT’42.5C GPU’AT’43.5C PMIC’AT’100C AUX’AT’43C CPU’AT’44C thermal’AT’43.6C VDD_IN 4253/4371 VDD_CPU_GPU_CV 572/670 VDD_SOC 1143/1135
RAM 5595/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [15%'AT’1190,5%'AT’1190,0%'AT’1190,0%'AT’1190,12%'AT’1190,10%'AT’1190] EMC_FREQ 1%'AT’1600 GR3D_FREQ 0%'AT’306 APE 150 MTS fg 0% bg 1% AO’AT’42.5C GPU’AT’43.5C PMIC’AT’100C AUX’AT’43C CPU’AT’44C thermal’AT’43.45C VDD_IN 4171/4365 VDD_CPU_GPU_CV 531/666 VDD_SOC 1102/1134
RAM 5598/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [16%'AT’1190,5%'AT’1190,5%'AT’1190,4%'AT’1190,3%'AT’1190,7%'AT’1344] EMC_FREQ 1%'AT’1600 GR3D_FREQ 0%'AT’306 APE 150 MTS fg 0% bg 2% AO’AT’42.5C GPU’AT’43.5C PMIC’AT’100C AUX’AT’43C CPU’AT’44.5C thermal’AT’43.65C VDD_IN 4212/4360 VDD_CPU_GPU_CV 572/663 VDD_SOC 1143/1134
RAM 5596/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [17%'AT’1190,0%'AT’1190,7%'AT’1190,4%'AT’1190,2%'AT’1420,2%'AT’1420] EMC_FREQ 1%'AT’1600 GR3D_FREQ 0%'AT’306 APE 150 MTS fg 0% bg 1% AO’AT’42.5C GPU’AT’43.5C PMIC’AT’100C AUX’AT’43C CPU’AT’44.5C thermal’AT’43.6C VDD_IN 4130/4353 VDD_CPU_GPU_CV 490/658 VDD_SOC 1102/1133
RAM 5596/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [15%'AT’1343,6%'AT’1420,6%'AT’1420,1%'AT’1420,11%'AT’1420,4%'AT’1420] EMC_FREQ 1%'AT’1600 GR3D_FREQ 0%'AT’306 APE 150 MTS fg 0% bg 2% AO’AT’42.5C GPU’AT’44C PMIC’AT’100C AUX’AT’43.5C CPU’AT’44.5C thermal’AT’43.8C VDD_IN 4171/4348 VDD_CPU_GPU_CV 531/654 VDD_SOC 1102/1132
RAM 5598/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [19%'AT’1190,9%'AT’1190,8%'AT’1190,10%'AT’1190,8%'AT’1343,3%'AT’1344] EMC_FREQ 1%'AT’1600 GR3D_FREQ 0%'AT’306 APE 150 MTS fg 0% bg 2% AO’AT’43C GPU’AT’44C PMIC’AT’100C AUX’AT’43.5C CPU’AT’44.5C thermal’AT’43.95C VDD_IN 4334/4348 VDD_CPU_GPU_CV 654/654 VDD_SOC 1143/1132
RAM 5597/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [17%'AT’1190,1%'AT’1190,2%'AT’1190,3%'AT’1190,11%'AT’1190,15%'AT’1190] EMC_FREQ 1%'AT’1600 GR3D_FREQ 0%'AT’306 APE 150 MTS fg 0% bg 2% AO’AT’43C GPU’AT’44C PMIC’AT’100C AUX’AT’43.5C CPU’AT’45C thermal’AT’43.95C VDD_IN 4253/4345 VDD_CPU_GPU_CV 613/653 VDD_SOC 1143/1133
RAM 5598/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [17%'AT’1420,18%'AT’1420,5%'AT’1420,2%'AT’1420,6%'AT’1420,3%'AT’1420] EMC_FREQ 1%'AT’1600 GR3D_FREQ 0%'AT’306 APE 150 MTS fg 0% bg 2% AO’AT’42.5C GPU’AT’44C PMIC’AT’100C AUX’AT’44C CPU’AT’45C thermal’AT’44.1C VDD_IN 4253/4343 VDD_CPU_GPU_CV 572/651 VDD_SOC 1143/1133
RAM 5596/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [14%'AT’1190,0%'AT’1190,0%'AT’1190,1%'AT’1190,6%'AT’1190,12%'AT’1190] EMC_FREQ 1%'AT’1600 GR3D_FREQ 0%'AT’306 APE 150 MTS fg 0% bg 1% AO’AT’43C GPU’AT’44C PMIC’AT’100C AUX’AT’44C CPU’AT’44.5C thermal’AT’44.1C VDD_IN 4130/4337 VDD_CPU_GPU_CV 490/647 VDD_SOC 1102/1132
RAM 5598/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [22%'AT’1420,11%'AT’1420,15%'AT’1420,7%'AT’1420,8%'AT’1420,4%'AT’1420] EMC_FREQ 1%'AT’1600 GR3D_FREQ 0%'AT’306 APE 150 MTS fg 0% bg 3% AO’AT’43C GPU’AT’44C PMIC’AT’100C AUX’AT’44C CPU’AT’45C thermal’AT’44.3C VDD_IN 4457/4340 VDD_CPU_GPU_CV 736/649 VDD_SOC 1143/1132
RAM 5596/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [20%'AT’1420,23%'AT’1420,7%'AT’1420,8%'AT’1420,3%'AT’1420,1%'AT’1420] EMC_FREQ 1%'AT’1600 GR3D_FREQ 0%'AT’306 APE 150 MTS fg 0% bg 5% AO’AT’43C GPU’AT’44C PMIC’AT’100C AUX’AT’44C CPU’AT’45C thermal’AT’44.3C VDD_IN 4375/4341 VDD_CPU_GPU_CV 736/651 VDD_SOC 1143/1133
RAM 5596/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [18%'AT’1190,21%'AT’1190,13%'AT’1190,11%'AT’1420,20%'AT’1420,46%'AT’1420] EMC_FREQ 6%'AT’1600 GR3D_FREQ 37%'AT’306 APE 150 MTS fg 0% bg 14% AO’AT’43C GPU’AT’44.5C PMIC’AT’100C AUX’AT’44C CPU’AT’45C thermal’AT’44.45C VDD_IN 5242/4362 VDD_CPU_GPU_CV 1267/666 VDD_SOC 1267/1136
RAM 5595/7772MB (lfb 141x4MB) SWAP 83/7982MB (cached 1MB) CPU [25%'AT’1420,35%'AT’1420,14%'AT’1420,27%'AT’1420,14%'AT’1420,29%'AT’1420] EMC_FREQ 10%'AT’1600 GR3D_FREQ 0%'AT’408 APE 150 MTS fg 0% bg 16% AO’AT’43C GPU’AT’44.5C PMIC’AT’100C AUX’AT’44C CPU’AT’45.5C thermal’AT’44.6C VDD_IN 5570/4391 VDD_CPU_GPU_CV 1431/684 VDD_SOC 1308/1140
RAM 5688/7772MB (lfb 139x4MB) SWAP 83/7982MB (cached 1MB) CPU [17%'AT’1190,51%'AT’1190,11%'AT’1190,5%'AT’1190,4%'AT’1350,10%'AT’1420] EMC_FREQ 7%'AT’1600 GR3D_FREQ 6%'AT’408 APE 150 MTS fg 0% bg 12% AO’AT’43.5C GPU’AT’44.5C PMIC’AT’100C AUX’AT’44C CPU’AT’45.5C thermal’AT’44.65C VDD_IN 4580/4395 VDD_CPU_GPU_CV 777/686 VDD_SOC 1184/1141
RAM 5689/7772MB (lfb 139x4MB) SWAP 83/7982MB (cached 1MB) CPU [15%'AT’1190,8%'AT’1190,1%'AT’1190,8%'AT’1190,8%'AT’1190,3%'AT’1190] EMC_FREQ 4%'AT’1600 GR3D_FREQ 0%'AT’408 APE 150 MTS fg 0% bg 1% AO’AT’43.5C GPU’AT’44.5C PMIC’AT’100C AUX’AT’44.5C CPU’AT’45.5C thermal’AT’44.45C VDD_IN 4212/4391 VDD_CPU_GPU_CV 531/682 VDD_SOC 1143/1141
RAM 5688/7772MB (lfb 139x4MB) SWAP 83/7982MB (cached 1MB) CPU [16%'AT’1190,9%'AT’1190,7%'AT’1190,1%'AT’1190,5%'AT’1190,8%'AT’1190] EMC_FREQ 2%'AT’1600 GR3D_FREQ 0%'AT’408 APE 150 MTS fg 0% bg 2% AO’AT’43.5C GPU’AT’44.5C PMIC’AT’100C AUX’AT’44.5C CPU’AT’45C thermal’AT’44.45C VDD_IN 4253/4388 VDD_CPU_GPU_CV 572/680 VDD_SOC 1143/1141
RAM 5689/7772MB (lfb 139x4MB) SWAP 83/7982MB (cached 1MB) CPU [16%'AT’1420,14%'AT’1420,4%'AT’1420,13%'AT’1420,1%'AT’1420,1%'AT’1420] EMC_FREQ 1%'AT’1600 GR3D_FREQ 0%'AT’408 APE 150 MTS fg 0% bg 2% AO’AT’43.5C GPU’AT’45C PMIC’AT’100C AUX’AT’44.5C CPU’AT’45.5C thermal’AT’44.6C VDD_IN 4253/4385 VDD_CPU_GPU_CV 613/679 VDD_SOC 1143/1141
RAM 5689/7772MB (lfb 139x4MB) SWAP 83/7982MB (cached 1MB) CPU [15%'AT’1190,5%'AT’1190,8%'AT’1190,6%'AT’1190,2%'AT’1344,2%'AT’1343] EMC_FREQ 1%'AT’1600 GR3D_FREQ 0%'AT’408 APE 150 MTS fg 0% bg 1% AO’AT’43.5C GPU’AT’45C PMIC’AT’100C AUX’AT’44C CPU’AT’45C thermal’AT’44.6C VDD_IN 4171/4380 VDD_CPU_GPU_CV 531/675 VDD_SOC 1143/1141
RAM 5689/7772MB (lfb 139x4MB) SWAP 83/7982MB (cached 1MB) CPU [16%'AT’1190,7%'AT’1190,2%'AT’1190,0%'AT’1190,5%'AT’1190,15%'AT’1190] EMC_FREQ 1%'AT’1600 GR3D_FREQ 0%'AT’408 APE 150 MTS fg 0% bg 1% AO’AT’43.5C GPU’AT’44.5C PMIC’AT’100C AUX’AT’44.5C CPU’AT’45.5C thermal’AT’44.8C VDD_IN 4212/4377 VDD_CPU_GPU_CV 572/673 VDD_SOC 1143/1141
RAM 5692/7772MB (lfb 139x4MB) SWAP 83/7982MB (cached 1MB) CPU [18%'AT’1420,8%'AT’1190,1%'AT’1190,10%'AT’1190,10%'AT’1190,10%'AT’1190] EMC_FREQ 1%'AT’1600 GR3D_FREQ 0%'AT’408 APE 150 MTS fg 0% bg 2% AO’AT’43.5C GPU’AT’44.5C PMIC’AT’100C AUX’AT’44.5C CPU’AT’45.5C thermal’AT’45.1C VDD_IN 4294/4375 VDD_CPU_GPU_CV 654/673 VDD_SOC 1143/1141
RAM 5689/7772MB (lfb 139x4MB) SWAP 83/7982MB (cached 1MB) CPU [13%'AT’1190,8%'AT’1190,0%'AT’1190,3%'AT’1190,7%'AT’1420,4%'AT’1420] EMC_FREQ 1%'AT’1600 GR3D_FREQ 0%'AT’408 APE 150 MTS fg 0% bg 2% AO’AT’44C GPU’AT’44.5C PMIC’AT’100C AUX’AT’44.5C CPU’AT’45.5C thermal’AT’44.8C VDD_IN 4171/4371 VDD_CPU_GPU_CV 531/670 VDD_SOC 1143/1141
RAM 5690/7772MB (lfb 139x4MB) SWAP 83/7982MB (cached 1MB) CPU [10%'AT’1190,5%'AT’1190,5%'AT’1190,8%'AT’1190,2%'AT’1190,0%'AT’1343] EMC_FREQ 0%'AT’1600 GR3D_FREQ 0%'AT’408 APE 150 MTS fg 0% bg 1% AO’AT’43.5C GPU’AT’45C PMIC’AT’100C AUX’AT’44.5C CPU’AT’45.5C thermal’AT’44.95C VDD_IN 4089/4366 VDD_CPU_GPU_CV 449/666 VDD_SOC 1102/1140
RAM 5690/7772MB (lfb 139x4MB) SWAP 83/7982MB (cached 1MB) CPU [19%'AT’1190,9%'AT’1190,0%'AT’1190,0%'AT’1190,1%'AT’1190,0%'AT’1420] EMC_FREQ 0%'AT’1600 GR3D_FREQ 0%'AT’408 APE 150 MTS fg 0% bg 0% AO’AT’44C GPU’AT’45C PMIC’AT’100C AUX’AT’44.5C CPU’AT’45.5C thermal’AT’45.3C VDD_IN 4130/4361 VDD_CPU_GPU_CV 490/663 VDD_SOC 1102/1140
RAM 5691/7772MB (lfb 139x4MB) SWAP 83/7982MB (cached 1MB) CPU [22%'AT’1420,3%'AT’1420,17%'AT’1420,8%'AT’1420,5%'AT’1420,17%'AT’1420] EMC_FREQ 1%'AT’1600 GR3D_FREQ 0%'AT’306 APE 150 MTS fg 0% bg 2% AO’AT’44C GPU’AT’45C PMIC’AT’100C AUX’AT’45C CPU’AT’46C thermal’AT’45.1C VDD_IN 4334/4361 VDD_CPU_GPU_CV 654/662 VDD_SOC 1143/1140
RAM 5688/7772MB (lfb 139x4MB) SWAP 83/7982MB (cached 1MB) CPU [19%'AT’1190,6%'AT’1190,0%'AT’1190,0%'AT’1190,6%'AT’1420,9%'AT’1420] EMC_FREQ 1%'AT’1600 GR3D_FREQ 0%'AT’306 APE 150 MTS fg 0% bg 1% AO’AT’44C GPU’AT’45C PMIC’AT’100C AUX’AT’45C CPU’AT’46C thermal’AT’45.15C VDD_IN 4171/4357 VDD_CPU_GPU_CV 531/660 VDD_SOC 1143/1140
RAM 5689/7772MB (lfb 139x4MB) SWAP 83/7982MB (cached 1MB) CPU [21%'AT’1190,8%'AT’1190,2%'AT’1190,1%'AT’1190,3%'AT’1190,7%'AT’1267] EMC_FREQ 1%'AT’1600 GR3D_FREQ 0%'AT’204 APE 150 MTS fg 0% bg 2% AO’AT’44.5C GPU’AT’45C PMIC’AT’100C AUX’AT’45C CPU’AT’46C thermal’AT’45.3C VDD_IN 4212/4355 VDD_CPU_GPU_CV 531/658 VDD_SOC 1143/1140
RAM 5699/7772MB (lfb 139x4MB) SWAP 83/7982MB (cached 1MB) CPU [20%'AT’1420,14%'AT’1420,11%'AT’1420,12%'AT’1420,21%'AT’1420,27%'AT’1420] EMC_FREQ 2%'AT’1600 GR3D_FREQ 8%'AT’204 APE 150 MTS fg 0% bg 7% AO’AT’44C GPU’AT’45C PMIC’AT’100C AUX’AT’45C CPU’AT’46C thermal’AT’45.3C VDD_IN 4621/4359 VDD_CPU_GPU_CV 858/661 VDD_SOC 1185/1140
RAM 5690/7772MB (lfb 139x4MB) SWAP 83/7982MB (cached 1MB) CPU [22%'AT’1190,8%'AT’1190,8%'AT’1190,3%'AT’1190,7%'AT’1190,11%'AT’1190] EMC_FREQ 1%'AT’1600 GR3D_FREQ 0%'AT’204 APE 150 MTS fg 0% bg 1% AO’AT’44C GPU’AT’45C PMIC’AT’100C AUX’AT’45C CPU’AT’46C thermal’AT’45.3C VDD_IN 4334/4359 VDD_CPU_GPU_CV 613/660 VDD_SOC 1143/1140

ver transformers: 4.0.1
ver tensorflow: 2.3.1+nv20.11
ver keras-bert: 0.86.0

Not sure for your case, but for such an OOM error, you would try adding some swap (here adding 8GB swap):

fallocate -l 8G swapfile
chmod 600 swapfile
mkswap swapfile
sudo swapon swapfile

Hi,

Based on the tegrastats log, the total memory is 7772MB.
However, it seems like TensorFlow can only allocate up to 5699MB.

Since Jetson is a shared memory system, TensorFlow may fail to get all the available memory in some cases.
A quite try is to reboot the XavierNX and launch the BERT sample without any other application running.

More, we also have a BERT example on XavierNX with TensorRT inference:

Thanks.