ResourceExhaustedError: Running TF-TRT integration on Jetson AGX

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
1

Workflow

  1. Freeze a .pb file
  2. Convert the .pb file to .pb file with TF-TRT optimization using the following:
trt_graph = trt.create_inference_graph(
    input_graph_def=tf.get_default_graph().as_graph_def(),
    outputs=output_node,
    max_batch_size=2,
    max_workspace_size_bytes=1 << 25,
    precision_mode="FP32",  # TRT Engine precision "FP32","FP16" or "INT8"
    minimum_segment_size=2  # minimum number of nodes in an engine
    )
    with tf.gfile.GFile("trt.pb", "wb") as f:
        f.write(trt_graph.SerializeToString())
  1. Run inference on trt.pb
    Note: Works on 1080Ti, doesn’t work on Jetson AGX

Error:

ResourceExhaustedError (see above for traceback): Requested batch size is not available and engine cache is full
	 [[Node: scope/my_trt_op_16 = TRTEngineOp[InT=[DT_FLOAT], OutT=[DT_FLOAT], cached_engine_batches=[2], calibration_data="", fixed_input_size=true, input_shapes=[[?,3,3,1]], max_cached_engines_count=1, output_shapes=[[?,1,1,6]], precision_mode="FP32", segment_funcdef_name="scope/my_trt_op_16_native_segment", serialized_segment="6003

ResourceExhaustedError (see above for traceback): Requested batch size is not available and engine cache is full
[[Node: scope/my_trt_op_16 = TRTEngineOpInT=[DT_FLOAT], OutT=[DT_FLOAT], cached_engine_batches=[2], calibration_data=“”, fixed_input_size=true, input_shapes=[[?,3,3,1]], max_cached_engines_count=1, output_shapes=[[?,1,1,6]], precision_mode=“FP32”, segment_funcdef_name=“scope/my_trt_op_16_native_segment”, serialized_segment=“\260\203\0…00\000\000”, static_engine=true, workspace_size_bytes=567516, _device=“/job:localhost/replica:0/task:0/device:GPU:0”]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[Node: scope3/op2/_13 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="scope4/op4", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

...00

ResourceExhaustedError (see above for traceback): Requested batch size is not available and engine cache is full
[[Node: scope/my_trt_op_16 = TRTEngineOpInT=[DT_FLOAT], OutT=[DT_FLOAT], cached_engine_batches=[2], calibration_data=“”, fixed_input_size=true, input_shapes=[[?,3,3,1]], max_cached_engines_count=1, output_shapes=[[?,1,1,6]], precision_mode=“FP32”, segment_funcdef_name=“scope/my_trt_op_16_native_segment”, serialized_segment=“\260\203\0…00\000\000”, static_engine=true, workspace_size_bytes=567516, _device=“/job:localhost/replica:0/task:0/device:GPU:0”]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[Node: scope3/op2/_13 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="scope4/op4", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

00

ResourceExhaustedError (see above for traceback): Requested batch size is not available and engine cache is full
[[Node: scope/my_trt_op_16 = TRTEngineOpInT=[DT_FLOAT], OutT=[DT_FLOAT], cached_engine_batches=[2], calibration_data=“”, fixed_input_size=true, input_shapes=[[?,3,3,1]], max_cached_engines_count=1, output_shapes=[[?,1,1,6]], precision_mode=“FP32”, segment_funcdef_name=“scope/my_trt_op_16_native_segment”, serialized_segment=“\260\203\0…00\000\000”, static_engine=true, workspace_size_bytes=567516, _device=“/job:localhost/replica:0/task:0/device:GPU:0”]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[Node: scope3/op2/_13 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="scope4/op4", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

00", static_engine=true, workspace_size_bytes=567516, _device="/job:localhost/replica:0/task:0/device:GPU:0"](scope2/op_name)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

	 [[Node: scope3/op2/_13 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="scope4/op4", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
2

Hi,

From your log, you meet an out of memory issue:

Hint: If you want to see a list of allocated tensors when <b>OOM happens</b>, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

You can try to set the precision into INT8 or cut the batchsize to 1:

max_batch_size=1,
precision_mode="INT8",

Thanks.

3
  1. Tried batch_size = 1 Same Error
  2. Tried FP16, same issue
  3. Can’t do INT8, that is an open issue i am trying to resolve
    TensorRT INT8 calibration doesn't work with TF r1.12 and TRT 5RC · Issue #22854 · tensorflow/tensorflow · GitHub

The actual bug is that once I convert the tf model to a tf-trt model, the model size is twice that of original. That is why i am able to run the tf model in FP but unable to do so for tf-trt model.

4

Hi,

Could you help to monitor the physical memory status with tegrastats?

sudo ./tegrastats

Thanks.

5

@AastaLLL The output from tegrastats is very unreadable. Also, what stats are you looking for while monitoring the status of physical memory? The program wants more than what is available, that is why it throws me the ResourceExhaustion error

6

Hi,

We want to check if you really run out of physical memory or meet some software limitation.

You can find the RAM usage at the beginning of tegrastats log:

RAM 1109/7854MB (lfb 1506x4MB) CPU [18%@345,off,off,1%@345,1%@345,4%@345] EMC_FREQ 4%@408 GR3D_FREQ 0%@140 APE 150 BCPU@32C MCPU@32C GPU@31C PLL@32C Tboard@28C Tdiode@30C PMIC@100C thermal@31.6C VDD_IN 1911/1568 VDD_CPU 229/158 VDD_GPU 152/152 VDD_SOC 458/384 VDD_WIFI 0/11 VDD_DDR 288/236

Total RAM is 7854MB
Used RAM is 1109MB

Thanks.

7

@AastaLLL Here are the stats:

RAM 2999/15827MB (lfb 2827x4MB) CPU [33%@1190,50%@1190,35%@1190,43%@1190,off,off,off,off] EMC_FREQ 7%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 6% AO@40.5C GPU@40.5C Tboard@42C Tdiode@43.25C AUX@40.5C CPU@42C thermal@40.95C PMIC@100C GPU 310/309 CPU 774/813 SOC 929/929 CV 0/0 VDDRQ 464/232 SYS5V 2136/1747
RAM 3121/15827MB (lfb 2779x4MB) CPU [31%@1190,31%@1190,68%@1190,19%@1190,off,off,off,off] EMC_FREQ 7%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 4% AO@40.5C GPU@40.5C Tboard@42C Tdiode@43.25C AUX@40.5C CPU@42C thermal@40.95C PMIC@100C GPU 310/309 CPU 774/813 SOC 929/929 CV 0/0 VDDRQ 464/235 SYS5V 2019/1751
RAM 3251/15827MB (lfb 2735x4MB) CPU [28%@1190,33%@1190,56%@1190,25%@1190,off,off,off,off] EMC_FREQ 7%@665 GR3D_FREQ 9%@114 APE 150 MTS fg 0% bg 3% AO@40.5C GPU@40.5C Tboard@42C Tdiode@43.5C AUX@40.5C CPU@42C thermal@41.1C PMIC@100C GPU 465/312 CPU 775/812 SOC 929/929 CV 0/0 VDDRQ 310/236 SYS5V 1938/1753
RAM 4851/15827MB (lfb 2330x4MB) CPU [35%@1190,41%@1190,80%@1190,34%@1190,off,off,off,off] EMC_FREQ 8%@665 GR3D_FREQ 24%@114 APE 150 MTS fg 0% bg 2% AO@40.5C GPU@40.5C Tboard@42C Tdiode@43.25C AUX@40.5C CPU@42C thermal@40.95C PMIC@100C GPU 310/312 CPU 930/814 SOC 929/929 CV 0/0 VDDRQ 465/240 SYS5V 1817/1754
RAM 7097/15827MB (lfb 1769x4MB) CPU [30%@1190,28%@1190,100%@1190,44%@1190,off,off,off,off] EMC_FREQ 12%@665 GR3D_FREQ 8%@114 APE 150 MTS fg 0% bg 1% AO@40.5C GPU@40.5C Tboard@42C Tdiode@43.25C AUX@40.5C CPU@42C thermal@41.1C PMIC@100C GPU 310/312 CPU 775/813 SOC 929/929 CV 0/0 VDDRQ 465/243 SYS5V 1776/1755
RAM 9443/15827MB (lfb 1182x4MB) CPU [13%@1190,17%@1190,100%@1190,31%@1190,off,off,off,off] EMC_FREQ 14%@665 GR3D_FREQ 24%@114 APE 150 MTS fg 0% bg 1% AO@40.5C GPU@40.5C Tboard@42C Tdiode@43.5C AUX@40C CPU@41.5C thermal@40.75C PMIC@100C GPU 310/312 CPU 775/813 SOC 929/929 CV 0/0 VDDRQ 465/246 SYS5V 1776/1755
RAM 11833/15827MB (lfb 585x4MB) CPU [19%@1190,21%@1190,100%@1190,11%@1190,off,off,off,off] EMC_FREQ 14%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 0% AO@40.5C GPU@40.5C Tboard@42C Tdiode@43.25C AUX@40.5C CPU@42C thermal@40.75C PMIC@100C GPU 310/312 CPU 775/812 SOC 929/929 CV 0/0 VDDRQ 465/249 SYS5V 1776/1755
RAM 13702/15827MB (lfb 118x4MB) CPU [16%@1190,16%@1190,100%@1190,14%@1190,off,off,off,off] EMC_FREQ 14%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 2% AO@40.5C GPU@40.5C Tboard@42C Tdiode@43.25C AUX@40C CPU@42C thermal@40.95C PMIC@100C GPU 310/312 CPU 775/812 SOC 929/929 CV 0/0 VDDRQ 465/252 SYS5V 1736/1755
RAM 14300/15827MB (lfb 15x4MB) CPU [72%@1190,79%@1190,98%@1190,75%@1190,off,off,off,off] EMC_FREQ 10%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 2% bg 3% AO@40.5C GPU@40.5C Tboard@42C Tdiode@43.25C AUX@40C CPU@42C thermal@40.95C PMIC@100C GPU 310/312 CPU 1084/815 SOC 929/929 CV 0/0 VDDRQ 464/254 SYS5V 1776/1755
RAM 14587/15827MB (lfb 15x4MB) CPU [75%@1190,30%@1190,91%@1190,33%@1190,off,off,off,off] EMC_FREQ 8%@665 GR3D_FREQ 21%@114 APE 150 MTS fg 0% bg 10% AO@40.5C GPU@41C Tboard@42C Tdiode@43.5C AUX@40.5C CPU@42C thermal@40.75C PMIC@100C GPU 310/311 CPU 1084/819 SOC 929/929 CV 0/0 VDDRQ 464/257 SYS5V 1776/1755
RAM 14548/15827MB (lfb 15x4MB) CPU [9%@1190,44%@1190,62%@1190,8%@1190,off,off,off,off] EMC_FREQ 8%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 20% AO@40.5C GPU@40.5C Tboard@42C Tdiode@43.25C AUX@40.5C CPU@42C thermal@40.95C PMIC@100C GPU 310/311 CPU 1084/822 SOC 929/929 CV 0/0 VDDRQ 464/260 SYS5V 1817/1756
RAM 14543/15827MB (lfb 15x4MB) CPU [24%@1190,85%@1190,26%@1190,15%@1190,off,off,off,off] EMC_FREQ 8%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 1% bg 10% AO@40.5C GPU@40.5C Tboard@42C Tdiode@43.25C AUX@40.5C CPU@42C thermal@40.95C PMIC@100C GPU 310/311 CPU 930/823 SOC 929/929 CV 0/0 VDDRQ 464/262 SYS5V 1776/1757
RAM 14605/15827MB (lfb 15x4MB) CPU [9%@1190,46%@1190,14%@1190,61%@1190,off,off,off,off] EMC_FREQ 7%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 10% AO@40.5C GPU@40.5C Tboard@42C Tdiode@43.25C AUX@40C CPU@42C thermal@40.95C PMIC@100C GPU 310/311 CPU 930/825 SOC 929/929 CV 0/0 VDDRQ 310/263 SYS5V 1736/1756
RAM 14579/15827MB (lfb 15x4MB) CPU [9%@1190,22%@1190,7%@1190,94%@1190,off,off,off,off] EMC_FREQ 5%@665 GR3D_FREQ 13%@114 APE 150 MTS fg 0% bg 5% AO@40.5C GPU@40.5C Tboard@42C Tdiode@43.25C AUX@40.5C CPU@42C thermal@40.95C PMIC@100C GPU 310/311 CPU 930/826 SOC 929/929 CV 0/0 VDDRQ 310/263 SYS5V 1776/1756
RAM 14708/15827MB (lfb 15x4MB) CPU [28%@1190,39%@1190,17%@1190,27%@1190,off,off,off,off] EMC_FREQ 5%@665 GR3D_FREQ 5%@114 APE 150 MTS fg 0% bg 3% AO@40.5C GPU@40.5C Tboard@42C Tdiode@43.25C AUX@40.5C CPU@42C thermal@40.8C PMIC@100C GPU 310/311 CPU 775/825 SOC 929/929 CV 0/0 VDDRQ 310/264 SYS5V 1978/1759
RAM 14859/15827MB (lfb 15x4MB) CPU [17%@1190,48%@1190,28%@1190,30%@1190,off,off,off,off] EMC_FREQ 5%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 3% AO@40.5C GPU@40.5C Tboard@42C Tdiode@43.25C AUX@40.5C CPU@42C thermal@40.8C PMIC@100C GPU 310/311 CPU 775/825 SOC 929/929 CV 0/0 VDDRQ 310/265 SYS5V 1978/1762
RAM 15011/15827MB (lfb 15x4MB) CPU [18%@1190,34%@1190,26%@1190,48%@1190,off,off,off,off] EMC_FREQ 5%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 3% AO@40.5C GPU@40.5C Tboard@42C Tdiode@43.5C AUX@40.5C CPU@41.5C thermal@40.95C PMIC@100C GPU 310/311 CPU 775/824 SOC 929/929 CV 0/0 VDDRQ 310/265 SYS5V 1938/1764
RAM 15162/15827MB (lfb 15x4MB) CPU [21%@1190,30%@1190,22%@1190,60%@1190,off,off,off,off] EMC_FREQ 5%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 2% AO@40.5C GPU@40.5C Tboard@42C Tdiode@43.5C AUX@40.5C CPU@41.5C thermal@40.95C PMIC@100C GPU 310/311 CPU 775/824 SOC 929/929 CV 0/0 VDDRQ 310/266 SYS5V 1857/1765
RAM 15231/15827MB (lfb 15x4MB) CPU [23%@1190,34%@1190,22%@1190,70%@1190,off,off,off,off] EMC_FREQ 3%@665 GR3D_FREQ 10%@114 APE 150 MTS fg 0% bg 3% AO@40.5C GPU@40.5C Tboard@42C Tdiode@43.25C AUX@40.5C CPU@42C thermal@40.95C PMIC@100C GPU 310/311 CPU 775/823 SOC 929/929 CV 0/0 VDDRQ 310/266 SYS5V 1978/1767
RAM 15410/15827MB (lfb 15x4MB) CPU [38%@1190,49%@1190,16%@1190,43%@1190,off,off,off,off] EMC_FREQ 3%@665 GR3D_FREQ 32%@114 APE 150 MTS fg 0% bg 3% AO@40.5C GPU@40.5C Tboard@42C Tdiode@43.25C AUX@40.5C CPU@42C thermal@40.95C PMIC@100C GPU 310/311 CPU 775/822 SOC 929/929 CV 0/0 VDDRQ 310/267 SYS5V 1938/1769
RAM 15487/15827MB (lfb 15x4MB) CPU [46%@1190,53%@1190,8%@1190,13%@1190,off,off,off,off] EMC_FREQ 3%@1331 GR3D_FREQ 99%@675 APE 150 MTS fg 0% bg 3% AO@41C GPU@41.5C Tboard@42C Tdiode@43.5C AUX@40.5C CPU@42C thermal@40.95C PMIC@100C GPU 1548/325 CPU 774/822 SOC 1238/933 CV 0/0 VDDRQ 464/269 SYS5V 2298/1775
RAM 15487/15827MB (lfb 15x4MB) CPU [9%@1190,14%@1190,17%@1190,83%@1190,off,off,off,off] EMC_FREQ 18%@1331 GR3D_FREQ 96%@675 APE 150 MTS fg 0% bg 2% AO@41C GPU@42C Tboard@42C Tdiode@43.75C AUX@40.5C CPU@42.5C thermal@41.25C PMIC@100C GPU 3556/362 CPU 773/821 SOC 1700/941 CV 0/0 VDDRQ 927/276 SYS5V 2378/1782
RAM 15487/15827MB (lfb 15x4MB) CPU [3%@1190,94%@1190,11%@1190,15%@1190,off,off,off,off] EMC_FREQ 17%@1331 GR3D_FREQ 99%@675 APE 150 MTS fg 0% bg 2% AO@41C GPU@42C Tboard@42C Tdiode@44C AUX@41C CPU@42.5C thermal@41.55C PMIC@100C GPU 4020/402 CPU 773/821 SOC 1546/948 CV 0/0 VDDRQ 927/283 SYS5V 2217/1787
RAM 15488/15827MB (lfb 15x4MB) CPU [8%@1190,99%@1190,14%@1190,1%@1190,off,off,off,off] EMC_FREQ 29%@1331 GR3D_FREQ 99%@675 APE 150 MTS fg 0% bg 3% AO@41C GPU@43C Tboard@42C Tdiode@44C AUX@41C CPU@43C thermal@41.7C PMIC@100C GPU 4943/452 CPU 772/820 SOC 2007/960 CV 0/0 VDDRQ 1544/297 SYS5V 2499/1795
RAM 15488/15827MB (lfb 15x4MB) CPU [7%@1190,100%@1190,8%@1190,14%@1190,off,off,off,off] EMC_FREQ 37%@1331 GR3D_FREQ 97%@675 APE 150 MTS fg 0% bg 1% AO@41.5C GPU@43C Tboard@42C Tdiode@44.25C AUX@41C CPU@43C thermal@42.05C PMIC@100C GPU 4943/501 CPU 772/820 SOC 2007/971 CV 0/0 VDDRQ 1544/311 SYS5V 2499/1802
RAM 15488/15827MB (lfb 15x4MB) CPU [6%@1190,46%@1190,5%@1190,66%@1190,off,off,off,off] EMC_FREQ 38%@1331 GR3D_FREQ 99%@675 APE 150 MTS fg 0% bg 1% AO@41.5C GPU@43C Tboard@42C Tdiode@44.5C AUX@41C CPU@43C thermal@42.2C PMIC@100C GPU 5096/550 CPU 772/819 SOC 2007/982 CV 0/0 VDDRQ 1698/326 SYS5V 2499/1810
RAM 15488/15827MB (lfb 15x4MB) CPU [10%@1190,22%@1190,2%@1190,83%@1190,off,off,off,off] EMC_FREQ 38%@1331 GR3D_FREQ 99%@675 APE 150 MTS fg 0% bg 1% AO@41.5C GPU@43C Tboard@42C Tdiode@44.5C AUX@41C CPU@42.5C thermal@42.2C PMIC@100C GPU 4634/594 CPU 772/819 SOC 2007/993 CV 0/0 VDDRQ 1544/339 SYS5V 2459/1817
RAM 15488/15827MB (lfb 15x4MB) CPU [4%@1190,6%@1190,11%@1190,100%@1190,off,off,off,off] EMC_FREQ 29%@1331 GR3D_FREQ 99%@675 APE 150 MTS fg 0% bg 0% AO@41.5C GPU@43C Tboard@42C Tdiode@44.5C AUX@41.5C CPU@43C thermal@42.2C PMIC@100C GPU 4174/631 CPU 772/818 SOC 1700/1000 CV 0/0 VDDRQ 1081/347 SYS5V 2257/1821
RAM 15488/15827MB (lfb 15x4MB) CPU [76%@1190,5%@1190,11%@1190,29%@1190,off,off,off,off] EMC_FREQ 29%@1331 GR3D_FREQ 99%@675 APE 150 MTS fg 0% bg 0% AO@41.5C GPU@43C Tboard@42C Tdiode@44.5C AUX@41.5C CPU@43C thermal@42.1C PMIC@100C GPU 3710/664 CPU 773/818 SOC 1700/1008 CV 0/0 VDDRQ 1082/354 SYS5V 2298/1826
RAM 15488/15827MB (lfb 15x4MB) CPU [100%@1190,2%@1190,19%@1190,1%@1190,off,off,off,off] EMC_FREQ 21%@1331 GR3D_FREQ 95%@675 APE 150 MTS fg 0% bg 1% AO@41.5C GPU@43C Tboard@42C Tdiode@44.25C AUX@41C CPU@43C thermal@42.4C PMIC@100C GPU 3248/690 CPU 773/817 SOC 1546/1013 CV 0/0 VDDRQ 773/359 SYS5V 2177/1830
RAM 15488/15827MB (lfb 15x4MB) CPU [100%@1190,2%@1190,17%@1190,1%@1190,off,off,off,off] EMC_FREQ 20%@1331 GR3D_FREQ 99%@675 APE 150 MTS fg 0% bg 0% AO@41.5C GPU@43C Tboard@42C Tdiode@44.5C AUX@41.5C CPU@43C thermal@42.4C PMIC@100C GPU 4174/726 CPU 773/817 SOC 1700/1020 CV 0/0 VDDRQ 927/364 SYS5V 2217/1834
RAM 15488/15827MB (lfb 15x4MB) CPU [100%@1190,1%@1190,17%@1190,1%@1190,off,off,off,off] EMC_FREQ 17%@1331 GR3D_FREQ 99%@675 APE 150 MTS fg 0% bg 0% AO@41.5C GPU@43C Tboard@42C Tdiode@44.5C AUX@41C CPU@43C thermal@42.4C PMIC@100C GPU 4174/761 CPU 773/816 SOC 1546/1026 CV 0/0 VDDRQ 927/370 SYS5V 2177/1837
RAM 15535/15827MB (lfb 15x4MB) CPU [52%@1190,40%@1190,11%@1190,9%@1190,off,off,off,off] EMC_FREQ 10%@1331 GR3D_FREQ 1%@420 APE 150 MTS fg 0% bg 6% AO@41.5C GPU@42C Tboard@42C Tdiode@44.25C AUX@41C CPU@43C thermal@42.4C PMIC@100C GPU 1084/764 CPU 774/816 SOC 1393/1029 CV 0/0 VDDRQ 464/371 SYS5V 2499/1844
RAM 15540/15827MB (lfb 15x4MB) CPU [42%@1190,28%@1190,13%@1190,58%@1190,off,off,off,off] EMC_FREQ 11%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 6% AO@41.5C GPU@41.5C Tboard@42C Tdiode@44.25C AUX@41C CPU@43C thermal@41.75C PMIC@100C GPU 464/761 CPU 774/816 SOC 929/1028 CV 0/0 VDDRQ 309/370 SYS5V 2419/1850
RAM 15546/15827MB (lfb 15x4MB) CPU [37%@1190,27%@1190,17%@1190,69%@1190,off,off,off,off] EMC_FREQ 5%@665 GR3D_FREQ 15%@114 APE 150 MTS fg 0% bg 7% AO@41.5C GPU@41.5C Tboard@42C Tdiode@44C AUX@41C CPU@42.5C thermal@41.6C PMIC@100C GPU 309/756 CPU 929/817 SOC 929/1027 CV 0/0 VDDRQ 309/370 SYS5V 2459/1856
RAM 15546/15827MB (lfb 15x4MB) CPU [32%@1190,43%@1190,19%@1190,53%@1190,off,off,off,off] EMC_FREQ 3%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 4% AO@41C GPU@41.5C Tboard@42C Tdiode@44C AUX@41C CPU@42.5C thermal@41.6C PMIC@100C GPU 310/752 CPU 774/816 SOC 929/1026 CV 0/0 VDDRQ 309/369 SYS5V 2419/1861
RAM 15546/15827MB (lfb 15x4MB) CPU [28%@1190,30%@1190,31%@1190,40%@1190,off,off,off,off] EMC_FREQ 3%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 3% AO@41C GPU@41C Tboard@42C Tdiode@44C AUX@41C CPU@42.5C thermal@41.6C PMIC@100C GPU 310/748 CPU 774/816 SOC 929/1025 CV 0/0 VDDRQ 309/369 SYS5V 2419/1867
RAM 15546/15827MB (lfb 15x4MB) CPU [28%@1190,31%@1190,24%@1190,46%@1190,off,off,off,off] EMC_FREQ 1%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 3% AO@41C GPU@41.5C Tboard@42C Tdiode@43.75C AUX@41C CPU@42.5C thermal@41.45C PMIC@100C GPU 310/744 CPU 774/815 SOC 929/1024 CV 0/0 VDDRQ 154/366 SYS5V 2378/1871
RAM 15547/15827MB (lfb 15x4MB) CPU [24%@1190,30%@1190,24%@1190,62%@1190,off,off,off,off] EMC_FREQ 1%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 3% AO@41C GPU@41C Tboard@42C Tdiode@44C AUX@41C CPU@42.5C thermal@41.45C PMIC@100C GPU 309/739 CPU 929/816 SOC 929/1023 CV 0/0 VDDRQ 309/366 SYS5V 2499/1877
RAM 15547/15827MB (lfb 15x4MB) CPU [29%@1190,32%@1190,51%@1190,37%@1190,off,off,off,off] EMC_FREQ 1%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 2% AO@41C GPU@41C Tboard@42C Tdiode@43.75C AUX@41C CPU@42.5C thermal@41.45C PMIC@100C GPU 310/735 CPU 774/816 SOC 929/1023 CV 0/0 VDDRQ 309/365 SYS5V 2419/1882
RAM 15547/15827MB (lfb 15x4MB) CPU [27%@1190,21%@1190,23%@1190,75%@1190,off,off,off,off] EMC_FREQ 1%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 2% AO@40.5C GPU@42C Tboard@42C Tdiode@43.75C AUX@40.5C CPU@42C thermal@41.6C PMIC@100C GPU 0/729 CPU 775/816 SOC 929/1022 CV 0/0 VDDRQ 310/365 SYS5V 2217/1885
RAM 15547/15827MB (lfb 15x4MB) CPU [17%@1190,22%@1190,13%@1190,72%@1190,off,off,off,off] EMC_FREQ 1%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 2% AO@41C GPU@42C Tboard@42C Tdiode@43.75C AUX@41C CPU@42C thermal@41.4C PMIC@100C GPU 0/722 CPU 775/815 SOC 930/1021 CV 0/0 VDDRQ 310/364 SYS5V 2136/1888
RAM 15545/15827MB (lfb 15x4MB) CPU [27%@1190,25%@1190,47%@1190,16%@1190,off,off,off,off] EMC_FREQ 1%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 1% AO@41C GPU@41C Tboard@42C Tdiode@43.75C AUX@40.5C CPU@42C thermal@41.45C PMIC@100C GPU 310/718 CPU 774/815 SOC 929/1020 CV 0/0 VDDRQ 309/364 SYS5V 2459/1893
RAM 15546/15827MB (lfb 15x4MB) CPU [34%@1190,26%@1190,82%@1190,87%@1190,off,off,off,off] EMC_FREQ 1%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 13% AO@41C GPU@41C Tboard@42C Tdiode@43.75C AUX@41C CPU@42.5C thermal@41.45C PMIC@100C GPU 309/715 CPU 1084/817 SOC 929/1019 CV 0/0 VDDRQ 464/365 SYS5V 2499/1898
RAM 15545/15827MB (lfb 15x4MB) CPU [21%@1190,16%@1190,46%@1190,48%@1190,off,off,off,off] EMC_FREQ 4%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 3% AO@41C GPU@41C Tboard@42C Tdiode@43.75C AUX@40.5C CPU@42.5C thermal@41.25C PMIC@100C GPU 310/711 CPU 774/817 SOC 929/1018 CV 0/0 VDDRQ 309/364 SYS5V 2378/1903
RAM 15485/15827MB (lfb 17x4MB) CPU [34%@1190,19%@1190,28%@1190,34%@1190,off,off,off,off] EMC_FREQ 4%@665 GR3D_FREQ 8%@114 APE 150 MTS fg 0% bg 14% AO@40.5C GPU@41C Tboard@42C Tdiode@43.75C AUX@40.5C CPU@42.5C thermal@41.25C PMIC@100C GPU 309/707 CPU 1084/819 SOC 929/1018 CV 0/0 VDDRQ 309/364 SYS5V 2298/1906
RAM 15508/15827MB (lfb 17x4MB) CPU [32%@1190,33%@1190,33%@1190,74%@1190,off,off,off,off] EMC_FREQ 4%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 1% bg 23% AO@41C GPU@41C Tboard@42C Tdiode@43.75C AUX@40.5C CPU@42.5C thermal@41.25C PMIC@100C GPU 309/704 CPU 1239/823 SOC 929/1017 CV 0/0 VDDRQ 309/363 SYS5V 2217/1909
RAM 15547/15827MB (lfb 17x4MB) CPU [30%@1190,59%@1190,20%@1190,74%@1190,off,off,off,off] EMC_FREQ 4%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 11% AO@40.5C GPU@41C Tboard@42C Tdiode@43.75C AUX@40.5C CPU@42C thermal@41.25C PMIC@100C GPU 309/700 CPU 929/824 SOC 929/1016 CV 0/0 VDDRQ 309/363 SYS5V 2298/1912
RAM 15556/15827MB (lfb 17x4MB) CPU [24%@1190,46%@1190,25%@1190,79%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 14% AO@40.5C GPU@41C Tboard@42C Tdiode@43.75C AUX@41C CPU@42.5C thermal@41.45C PMIC@100C GPU 309/697 CPU 1084/826 SOC 929/1015 CV 0/0 VDDRQ 309/362 SYS5V 2298/1916
RAM 15555/15827MB (lfb 17x4MB) CPU [29%@1190,29%@1190,35%@1190,76%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 6% AO@41C GPU@41C Tboard@42C Tdiode@43.75C AUX@40.5C CPU@42C thermal@41.3C PMIC@100C GPU 310/694 CPU 774/826 SOC 929/1015 CV 0/0 VDDRQ 309/362 SYS5V 2378/1920
RAM 15552/15827MB (lfb 17x4MB) CPU [17%@1190,22%@1190,7%@1190,94%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 1% AO@41C GPU@42C Tboard@42C Tdiode@43.75C AUX@41C CPU@42C thermal@41.6C PMIC@100C GPU 0/688 CPU 775/825 SOC 929/1014 CV 0/0 VDDRQ 155/360 SYS5V 2177/1922
RAM 15553/15827MB (lfb 16x4MB) CPU [26%@1190,14%@1190,12%@1190,87%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 2% AO@41C GPU@41C Tboard@42C Tdiode@43.5C AUX@40.5C CPU@42.5C thermal@41.3C PMIC@100C GPU 310/685 CPU 774/825 SOC 929/1013 CV 0/0 VDDRQ 154/358 SYS5V 2419/1926
RAM 15553/15827MB (lfb 16x4MB) CPU [25%@1190,21%@1190,18%@1190,79%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 2% AO@41C GPU@42C Tboard@42C Tdiode@43.75C AUX@40.5C CPU@42C thermal@41.45C PMIC@100C GPU 0/679 CPU 775/824 SOC 929/1012 CV 0/0 VDDRQ 155/357 SYS5V 2257/1929
RAM 15556/15827MB (lfb 16x4MB) CPU [21%@1190,43%@1190,43%@1190,38%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 4% AO@40.5C GPU@42.5C Tboard@42C Tdiode@43.5C AUX@40.5C CPU@42.5C thermal@41.7C PMIC@100C GPU 0/673 CPU 775/824 SOC 929/1012 CV 0/0 VDDRQ 155/355 SYS5V 2177/1931
RAM 15557/15827MB (lfb 16x4MB) CPU [21%@1190,25%@1190,22%@1190,74%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 2% AO@41C GPU@42C Tboard@42C Tdiode@43.5C AUX@41C CPU@42C thermal@41.6C PMIC@100C GPU 0/668 CPU 930/825 SOC 929/1011 CV 0/0 VDDRQ 310/355 SYS5V 2140/1932
RAM 15557/15827MB (lfb 15x4MB) CPU [27%@1190,90%@1190,15%@1190,9%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 2% AO@41C GPU@42.5C Tboard@42C Tdiode@43.5C AUX@40.5C CPU@42.5C thermal@41.4C PMIC@100C GPU 0/662 CPU 775/824 SOC 930/1010 CV 0/0 VDDRQ 155/353 SYS5V 2136/1934
RAM 15557/15827MB (lfb 15x4MB) CPU [23%@1190,47%@1190,15%@1190,57%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 3% AO@41C GPU@42.5C Tboard@42C Tdiode@43.5C AUX@40.5C CPU@42.5C thermal@41.6C PMIC@100C GPU 0/657 CPU 775/824 SOC 930/1010 CV 0/0 VDDRQ 310/353 SYS5V 2059/1935
RAM 15557/15827MB (lfb 15x4MB) CPU [32%@1190,57%@1190,64%@1190,63%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 9% AO@41C GPU@41C Tboard@42C Tdiode@43.75C AUX@40.5C CPU@42.5C thermal@41.45C PMIC@100C GPU 309/654 CPU 929/825 SOC 929/1009 CV 0/0 VDDRQ 309/352 SYS5V 2378/1939
RAM 15497/15827MB (lfb 15x4MB) CPU [21%@1190,27%@1190,83%@1190,11%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 2% AO@41C GPU@41C Tboard@42C Tdiode@43.5C AUX@41C CPU@42C thermal@41.3C PMIC@100C GPU 310/652 CPU 774/824 SOC 929/1008 CV 0/0 VDDRQ 309/352 SYS5V 2298/1941
RAM 15541/15827MB (lfb 15x4MB) CPU [25%@1190,41%@1190,32%@1190,34%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 12% AO@41C GPU@41C Tboard@42C Tdiode@43.75C AUX@41C CPU@42.5C thermal@41.25C PMIC@100C GPU 309/649 CPU 929/825 SOC 929/1008 CV 0/0 VDDRQ 309/352 SYS5V 2298/1944
RAM 15551/15827MB (lfb 16x4MB) CPU [32%@1190,49%@1190,56%@1190,45%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 10%@114 APE 150 MTS fg 1% bg 18% AO@41C GPU@41C Tboard@42C Tdiode@43.75C AUX@40.5C CPU@42.5C thermal@41.45C PMIC@100C GPU 309/646 CPU 1084/827 SOC 929/1007 CV 0/0 VDDRQ 309/351 SYS5V 2217/1946
RAM 15585/15827MB (lfb 16x4MB) CPU [27%@1190,46%@1190,38%@1190,54%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 8% AO@40.5C GPU@41C Tboard@42C Tdiode@43.75C AUX@41C CPU@42C thermal@41.3C PMIC@100C GPU 310/644 CPU 929/828 SOC 929/1007 CV 0/0 VDDRQ 309/351 SYS5V 2257/1949
RAM 15594/15827MB (lfb 16x4MB) CPU [47%@1190,29%@1190,33%@1190,48%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 12% AO@41C GPU@41C Tboard@42C Tdiode@43.5C AUX@41C CPU@42C thermal@41.25C PMIC@100C GPU 310/641 CPU 929/829 SOC 929/1006 CV 0/0 VDDRQ 309/351 SYS5V 2298/1951
RAM 15595/15827MB (lfb 16x4MB) CPU [24%@1190,15%@1190,28%@1190,40%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 9% AO@41C GPU@41C Tboard@42C Tdiode@43.5C AUX@41C CPU@42.5C thermal@41.3C PMIC@100C GPU 310/638 CPU 929/830 SOC 929/1005 CV 0/0 VDDRQ 309/350 SYS5V 2257/1954
RAM 15541/15827MB (lfb 21x4MB) CPU [20%@1190,28%@1190,13%@1190,34%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 4% AO@40.5C GPU@41C Tboard@42C Tdiode@43.75C AUX@40.5C CPU@42C thermal@41.45C PMIC@100C GPU 310/636 CPU 774/829 SOC 929/1005 CV 0/0 VDDRQ 309/350 SYS5V 2338/1957
RAM 15542/15827MB (lfb 15x4MB) CPU [51%@1190,35%@1190,43%@1190,35%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 1% bg 11% AO@41C GPU@41C Tboard@42C Tdiode@43.75C AUX@40.5C CPU@42C thermal@41.45C PMIC@100C GPU 310/633 CPU 1084/831 SOC 929/1004 CV 0/0 VDDRQ 309/350 SYS5V 2217/1959
RAM 15521/15827MB (lfb 16x4MB) CPU [48%@1190,60%@1190,70%@1190,75%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 1% bg 14% AO@41C GPU@41C Tboard@42C Tdiode@43.75C AUX@40.5C CPU@42.5C thermal@41.45C PMIC@100C GPU 309/631 CPU 1239/834 SOC 929/1004 CV 0/0 VDDRQ 309/349 SYS5V 2217/1961
RAM 15541/15827MB (lfb 15x4MB) CPU [39%@1190,53%@1190,56%@1190,40%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 21% AO@41C GPU@41C Tboard@42C Tdiode@43.75C AUX@41C CPU@42.5C thermal@41.45C PMIC@100C GPU 309/629 CPU 1239/837 SOC 929/1003 CV 0/0 VDDRQ 309/349 SYS5V 2298/1963
RAM 12992/15827MB (lfb 202x4MB) CPU [34%@1190,38%@1190,26%@1190,49%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 2%@114 APE 150 MTS fg 0% bg 14% AO@40.5C GPU@41C Tboard@42C Tdiode@43.5C AUX@41C CPU@42.5C thermal@41.3C PMIC@100C GPU 309/626 CPU 1084/839 SOC 929/1003 CV 0/0 VDDRQ 309/349 SYS5V 2136/1964
RAM 4056/15827MB (lfb 2406x4MB) CPU [68%@1190,65%@1190,75%@1190,90%@1190,off,off,off,off] EMC_FREQ 6%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 2% bg 9% AO@41C GPU@41C Tboard@42C Tdiode@43.75C AUX@41C CPU@42.5C thermal@41.45C PMIC@100C GPU 309/624 CPU 1239/842 SOC 929/1002 CV 0/0 VDDRQ 464/350 SYS5V 1938/1964
RAM 4074/15827MB (lfb 2406x4MB) CPU [38%@1190,63%@1190,45%@1190,67%@1190,off,off,off,off] EMC_FREQ 7%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 1% bg 15% AO@41C GPU@42C Tboard@42C Tdiode@43.75C AUX@40.5C CPU@42C thermal@41.45C PMIC@100C GPU 310/622 CPU 1085/844 SOC 929/1001 CV 0/0 VDDRQ 310/349 SYS5V 1817/1963
RAM 4090/15827MB (lfb 2406x4MB) CPU [10%@1190,9%@1190,4%@1190,85%@1190,off,off,off,off] EMC_FREQ 4%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 18% AO@40.5C GPU@41C Tboard@42C Tdiode@43.5C AUX@41C CPU@42C thermal@41.3C PMIC@100C GPU 310/619 CPU 930/844 SOC 929/1001 CV 0/0 VDDRQ 309/349 SYS5V 1898/1963
RAM 4113/15827MB (lfb 2406x4MB) CPU [49%@1190,68%@1190,59%@1190,56%@1190,off,off,off,off] EMC_FREQ 4%@665 GR3D_FREQ 9%@114 APE 150 MTS fg 1% bg 15% AO@40.5C GPU@41C Tboard@42C Tdiode@43.75C AUX@41C CPU@42.5C thermal@41.3C PMIC@100C GPU 465/618 CPU 1239/847 SOC 929/1000 CV 0/0 VDDRQ 309/349 SYS5V 1898/1962
RAM 4136/15827MB (lfb 2406x4MB) CPU [40%@1190,63%@1190,47%@1190,81%@1190,off,off,off,off] EMC_FREQ 4%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 1% bg 15% AO@41C GPU@41C Tboard@42C Tdiode@43.75C AUX@41C CPU@42.5C thermal@41.45C PMIC@100C GPU 465/617 CPU 1239/850 SOC 929/1000 CV 0/0 VDDRQ 309/348 SYS5V 1898/1962
RAM 4161/15827MB (lfb 2406x4MB) CPU [30%@1190,24%@1190,15%@1190,88%@1190,off,off,off,off] EMC_FREQ 4%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 1% bg 12% AO@40.5C GPU@41C Tboard@42C Tdiode@43.5C AUX@41C CPU@42C thermal@41.3C PMIC@100C GPU 310/615 CPU 930/850 SOC 929/999 CV 0/0 VDDRQ 310/348 SYS5V 1857/1961
RAM 4175/15827MB (lfb 2406x4MB) CPU [6%@1190,7%@1190,3%@1190,92%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 15% AO@41C GPU@41C Tboard@42C Tdiode@43.75C AUX@41C CPU@42C thermal@41.25C PMIC@100C GPU 310/613 CPU 930/851 SOC 929/999 CV 0/0 VDDRQ 310/348 SYS5V 1817/1960
RAM 4115/15827MB (lfb 2406x4MB) CPU [2%@1190,1%@1190,2%@1190,58%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 3% AO@41C GPU@42C Tboard@42C Tdiode@43.5C AUX@41C CPU@42C thermal@41.6C PMIC@100C GPU 155/610 CPU 465/848 SOC 930/998 CV 0/0 VDDRQ 155/347 SYS5V 1615/1958
RAM 4116/15827MB (lfb 2406x4MB) CPU [1%@1190,2%@1190,0%@1190,4%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 1% AO@41C GPU@42C Tboard@42C Tdiode@43.75C AUX@40.5C CPU@42C thermal@41.4C PMIC@100C GPU 0/606 CPU 465/846 SOC 930/998 CV 0/0 VDDRQ 155/345 SYS5V 1494/1954
RAM 4116/15827MB (lfb 2407x4MB) CPU [3%@1190,2%@1190,3%@1190,4%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 2% AO@40.5C GPU@42C Tboard@42C Tdiode@43.5C AUX@40.5C CPU@42C thermal@41.4C PMIC@100C GPU 0/601 CPU 465/843 SOC 930/997 CV 0/0 VDDRQ 155/344 SYS5V 1494/1951
RAM 4116/15827MB (lfb 2407x4MB) CPU [0%@1190,0%@1190,0%@1190,0%@1190,off,off,off,off] EMC_FREQ 0%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 0% AO@40.5C GPU@42C Tboard@42C Tdiode@43.75C AUX@40.5C CPU@42C thermal@41.4C PMIC@100C GPU 0/597 CPU 310/839 SOC 930/997 CV 0/0 VDDRQ 155/343 SYS5V 1453/1948
RAM 4116/15827MB (lfb 2407x4MB) CPU [0%@1190,0%@1190,0%@1190,1%@1190,off,off,off,off] EMC_FREQ 0%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 0% AO@41C GPU@42C Tboard@42C Tdiode@43.75C AUX@40.5C CPU@42C thermal@41.4C PMIC@100C GPU 0/593 CPU 465/837 SOC 930/997 CV 0/0 VDDRQ 155/341 SYS5V 1453/1944
RAM 4116/15827MB (lfb 2407x4MB) CPU [1%@1190,0%@1190,0%@1190,0%@1190,off,off,off,off] EMC_FREQ 0%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 0% AO@40.5C GPU@42C Tboard@42C Tdiode@43.5C AUX@40.5C CPU@42C thermal@41.4C PMIC@100C GPU 0/589 CPU 465/834 SOC 930/996 CV 0/0 VDDRQ 155/340 SYS5V 1453/1941
RAM 4116/15827MB (lfb 2407x4MB) CPU [2%@1190,1%@1190,0%@1190,0%@1190,off,off,off,off] EMC_FREQ 0%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 0% AO@40.5C GPU@42C Tboard@42C Tdiode@43.5C AUX@40.5C CPU@42C thermal@41.4C PMIC@100C GPU 0/585 CPU 465/832 SOC 930/996 CV 0/0 VDDRQ 155/339 SYS5V 1453/1938
RAM 4117/15827MB (lfb 2407x4MB) CPU [0%@1190,0%@1190,1%@1190,0%@1190,off,off,off,off] EMC_FREQ 0%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 0% AO@40.5C GPU@42C Tboard@42C Tdiode@43.5C AUX@41C CPU@42C thermal@41.3C PMIC@100C GPU 0/582 CPU 465/829 SOC 930/995 CV 0/0 VDDRQ 155/338 SYS5V 1453/1935
RAM 4117/15827MB (lfb 2407x4MB) CPU [0%@1190,0%@1190,0%@1190,0%@1190,off,off,off,off] EMC_FREQ 0%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 0% AO@40.5C GPU@42C Tboard@42C Tdiode@43.5C AUX@40.5C CPU@42C thermal@41.1C PMIC@100C GPU 0/578 CPU 465/827 SOC 930/995 CV 0/0 VDDRQ 155/336 SYS5V 1494/1932
RAM 4116/15827MB (lfb 2407x4MB) CPU [0%@1190,0%@1190,1%@1190,0%@1190,off,off,off,off] EMC_FREQ 0%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 0% AO@41C GPU@41.5C Tboard@42C Tdiode@43.5C AUX@40.5C CPU@41.5C thermal@41.4C PMIC@100C GPU 0/574 CPU 310/824 SOC 930/994 CV 0/0 VDDRQ 155/335 SYS5V 1494/1929
RAM 4116/15827MB (lfb 2407x4MB) CPU [0%@1190,0%@1190,0%@1190,0%@1190,off,off,off,off] EMC_FREQ 0%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 0% AO@40.5C GPU@41.5C Tboard@42C Tdiode@43.5C AUX@40.5C CPU@41.5C thermal@41.1C PMIC@100C GPU 0/570 CPU 310/820 SOC 930/994 CV 0/0 VDDRQ 155/334 SYS5V 1453/1926
RAM 4116/15827MB (lfb 2407x4MB) CPU [0%@1190,0%@1190,0%@1190,1%@1190,off,off,off,off] EMC_FREQ 0%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 0% AO@41C GPU@42C Tboard@42C Tdiode@43.5C AUX@40.5C CPU@42C thermal@41.1C PMIC@100C GPU 0/566 CPU 465/818 SOC 930/994 CV 0/0 VDDRQ 155/333 SYS5V 1453/1923
RAM 4116/15827MB (lfb 2407x4MB) CPU [0%@1190,0%@1190,0%@1190,1%@1190,off,off,off,off] EMC_FREQ 0%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 0% AO@41C GPU@41.5C Tboard@42C Tdiode@43.75C AUX@41C CPU@41.5C thermal@41.4C PMIC@100C GPU 0/563 CPU 465/816 SOC 930/993 CV 0/0 VDDRQ 155/332 SYS5V 1494/1920
RAM 4116/15827MB (lfb 2407x4MB) CPU [1%@1190,0%@1190,0%@1190,0%@1190,off,off,off,off] EMC_FREQ 0%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 0% AO@40.5C GPU@42C Tboard@42C Tdiode@43.5C AUX@40.5C CPU@42C thermal@41.4C PMIC@100C GPU 0/559 CPU 310/813 SOC 930/993 CV 0/0 VDDRQ 155/331 SYS5V 1494/1917
RAM 4116/15827MB (lfb 2407x4MB) CPU [0%@1190,1%@1190,0%@1190,0%@1190,off,off,off,off] EMC_FREQ 0%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 1% AO@41C GPU@41.5C Tboard@42C Tdiode@43.5C AUX@40.5C CPU@41.5C thermal@41.4C PMIC@100C GPU 0/556 CPU 465/810 SOC 930/992 CV 0/0 VDDRQ 155/330 SYS5V 1494/1915
8

Hi,

You are running out of physical memory:

RAM <b>15541/15827MB</b> (lfb 15x4MB) CPU [25%@1190,41%@1190,32%@1190,34%@1190,off,off,off,off] EMC_FREQ 2%@665 GR3D_FREQ 0%@114 APE 150 MTS fg 0% bg 12% AO@41C GPU@41C Tboard@42C Tdiode@43.75C AUX@41C CPU@42.5C thermal@41.25C PMIC@100C GPU 309/649 CPU 929/825 SOC 929/1008 CV 0/0 VDDRQ 309/352 SYS5V 2298/1944

Total RAM is 15541MB
Used RAM is 15827MB

When creating TensorRT in the TensorFlow interface, it will generate two GPU model(TF&TRT) and takes twice memory amount.

If your model can fit into 16G memory(test this with pure TF), you can try to convert the model into UFF.
And launch it with native TensorRT to avoid double memory issue.

Here is a tutorial for your reference:
https://github.com/NVIDIA-AI-IOT/tf_trt_models

Thanks.

9

Like I said in #3, it works with pure TF, and the model size doubles with TF-TRT integration. The only reason I am using TF-TRT instead of native TRT is that my model has too many unsupported layers, more than I can count on the console. Is there a way to just get inference timings without actually writing “correct” custom layer plugins?

10

Hi,

If pure TensorRT is not a option for you, it’s recommended to use TensorFlow since your model requires lots of memory.

TF-TRT intends to duplicate the model: one for TensorFlow and the other for TensorRT.
Although you can get some performance improvement, the increase of memory is not affordable for your use case.

Thanks.