Cuda Error in executeInternal: 700 (an illegal memory access was encountered)

igal.kroyter · October 13, 2021, 11:24am

Hello,

I’m trying to infer a tensor with different batch size utilizing the TensorRT. The tensor size if {x,128,3} where I was increasing the x value from 2 up to 3010. Any value above 3010 I’m getting the dump below. It has to be noted that I’ve used an onnx model to construct the network:

./tester_onnx --batch 4096
&&&& RUNNING TensorRT.tester_onnx # ./tester_onnx --batch 4096
Input filename: data/model.onnx
ONNX IR version: 0.0.4
Opset version: 9
Producer name: tf2onnx
Producer version: 1.9.2
*Domain: *
Model version: 0
Doc string:
[10/13/2021-09:25:42] [W] [TRT] onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
Preparation time: 51419783974[ns]
[10/13/2021-09:26:32] [E] [TRT] engine.cpp (986) - Cuda Error in executeInternal: 700 (an illegal memory access was encountered)
[10/13/2021-09:26:32] [E] [TRT] FAILED_EXECUTION: std::exception
failed executeV2
&&&& FAILED TensorRT.tester_onnx # ./tester_onnx --batch 4096
[10/13/2021-09:26:32] [E] [TRT] engine.cpp (179) - Cuda Error in ~ExecutionContext: 700 (an illegal memory access was encountered)
[10/13/2021-09:26:32] [E] [TRT] INTERNAL_ERROR: std::exception
[10/13/2021-09:26:32] [E] [TRT] Parameter check failed at: …/rtSafe/safeContext.cpp::terminateCommonContext::155, condition: cudnnDestroy(context.cudnn) failure.
[10/13/2021-09:26:32] [E] [TRT] Parameter check failed at: …/rtSafe/safeContext.cpp::terminateCommonContext::165, condition: cudaEventDestroy(context.start) failure.
[10/13/2021-09:26:32] [E] [TRT] Parameter check failed at: …/rtSafe/safeContext.cpp::terminateCommonContext::170, condition: cudaEventDestroy(context.stop) failure.
[10/13/2021-09:26:32] [E] [TRT] …/rtSafe/safeRuntime.cpp (32) - Cuda Error in free: 700 (an illegal memory access was encountered)
terminate called after throwing an instance of ‘nvinfer1::CudaError’
what(): std::exception
Aborted (core dumped)

Please advise how to resolve the issue.

AastaLLL · October 14, 2021, 2:46am

Hi,

Based on your testing, it’s quite possible that you are running out of memory when increasing batch size to 4096.
You can double-check this via monitoring the system with tegrastats at the same time.

May I know which JetPack version do you use?
Are you using TensorRT v8.0 from JetPack 4.6?

Thanks.

igal.kroyter · October 14, 2021, 6:10am

hi @AastaLLL,

Thanks for the reply.

The NN model that I’m working with was established by Tensorflow 2.3.0 which is supported by JP 4.4.1.
Thus I had to install JP 4.4.1. the TensorRT that I utilize is 7.1.3

dpkg -l | grep TensorRT

Moreover, at the moment there is some issue with Tensorflow which is being resolved by NVidia Cannot import TF 2.6.0 correctly on Xavier NX - #8 by AastaLLL (by you :) )

Following is a tegrastats dump, prior, during and after the crashing:

**sudo tegrastats**
RAM 2780/15823MB (lfb 2739x4MB) SWAP 0/7911MB (cached 0MB) CPU [0%@1190,0%@1190,1%@1190,4%@1190,3%@1190,2%@1190,0%@1330,0%@1343] EMC_FREQ 1%@665 GR3D_FREQ 0%@318 APE 150 MTS fg 0% bg 7% AO@41.5C GPU@41C Tdiode@44.75C PMIC@100C AUX@41C CPU@42.5C thermal@41.45C Tboard@42C GPU 464/464 CPU 773/773 SOC 2010/2010 CV 0/0 VDDRQ 154/154 SYS5V 2059/2059
RAM 2761/15823MB (lfb 2740x4MB) SWAP 0/7911MB (cached 0MB) CPU [12%@1267,10%@1343,11%@1565,9%@1574,6%@1873,6%@1881,9%@1881,8%@1881] EMC_FREQ 2%@665 GR3D_FREQ 0%@318 APE 150 MTS fg 0% bg 15% AO@41.5C GPU@41C Tdiode@44.75C PMIC@100C AUX@41.5C CPU@42.5C thermal@41.45C Tboard@42C GPU 619/541 CPU 928/850 SOC 1237/1623 CV 0/0 VDDRQ 154/154 SYS5V 1820/1939
RAM 2761/15823MB (lfb 2740x4MB) SWAP 0/7911MB (cached 0MB) CPU [10%@1190,9%@1190,7%@1190,10%@1190,6%@1414,7%@1420,6%@1420,5%@1621] EMC_FREQ 3%@665 GR3D_FREQ 0%@318 APE 150 MTS fg 0% bg 12% AO@41.5C GPU@41C Tdiode@44.75C PMIC@100C AUX@41C CPU@42.5C thermal@41.45C Tboard@42C GPU 619/567 CPU 1083/928 SOC 1082/1443 CV 0/0 VDDRQ 154/154 SYS5V 1820/1899
RAM 2761/15823MB (lfb 2740x4MB) SWAP 0/7911MB (cached 0MB) CPU [7%@1190,7%@1190,0%@1190,2%@1190,0%@1190,0%@1190,3%@1329,1%@1343] EMC_FREQ 2%@665 GR3D_FREQ 0%@318 APE 150 MTS fg 0% bg 7% AO@41.5C GPU@41C Tdiode@44.5C PMIC@100C AUX@41C CPU@42.5C thermal@41.45C Tboard@42C GPU 464/541 CPU 774/889 SOC 1083/1353 CV 0/0 VDDRQ 154/154 SYS5V 1779/1869
RAM 2761/15823MB (lfb 2740x4MB) SWAP 0/7911MB (cached 0MB) CPU [2%@1190,1%@1190,4%@1190,7%@1331,0%@1344,1%@1487,1%@1497,3%@1685] EMC_FREQ 2%@665 GR3D_FREQ 0%@318 APE 150 MTS fg 0% bg 10% AO@41.5C GPU@41C Tdiode@44.75C PMIC@100C AUX@41.5C CPU@42.5C thermal@41.45C Tboard@42C GPU 464/526 CPU 774/866 SOC 1083/1299 CV 0/0 VDDRQ 154/154 SYS5V 1739/1843
RAM 2759/15823MB (lfb 2740x4MB) SWAP 0/7911MB (cached 0MB) CPU [4%@1190,1%@1190,8%@1190,6%@1325,1%@1343,0%@1343,2%@1490,1%@1499] EMC_FREQ 0%@2133 GR3D_FREQ 0%@318 APE 150 MTS fg 0% bg 10% AO@41.5C GPU@41C Tdiode@44.5C PMIC@100C AUX@41C CPU@42.5C thermal@41.3C Tboard@42C GPU 464/515 CPU 773/850 SOC 2164/1443 CV 0/0 VDDRQ 154/154 SYS5V 2099/1886
RAM 2955/15823MB (lfb 2734x4MB) SWAP 0/7911MB (cached 0MB) CPU [3%@2265,4%@2265,9%@2265,9%@2265,57%@2265,3%@2265,3%@2265,1%@2265] EMC_FREQ 1%@2133 GR3D_FREQ 6%@318 APE 150 MTS fg 0% bg 25% AO@41.5C GPU@41C Tdiode@44.75C PMIC@100C AUX@41.5C CPU@43.5C thermal@41.65C Tboard@42C GPU 617/530 CPU 2934/1148 SOC 2624/1611 CV 0/0 VDDRQ 308/176 SYS5V 2342/1951
RAM 3145/15823MB (lfb 2693x4MB) SWAP 0/7911MB (cached 0MB) CPU [46%@2265,9%@2265,16%@2265,40%@2265,18%@2265,9%@2265,14%@2265,11%@2265] EMC_FREQ 2%@2133 GR3D_FREQ 36%@318 APE 150 MTS fg 1% bg 9% AO@41.5C GPU@41.5C Tdiode@45C PMIC@100C AUX@41.5C CPU@44.5C thermal@41.8C Tboard@42C GPU 771/560 CPU 4935/1621 SOC 2929/1776 CV 0/0 VDDRQ 308/192 SYS5V 2423/2010
RAM 3146/15823MB (lfb 2692x4MB) SWAP 0/7911MB (cached 0MB) CPU [100%@2265,9%@2265,39%@2265,23%@2265,13%@2265,17%@2265,10%@2265,24%@2265] EMC_FREQ 10%@665 GR3D_FREQ 30%@318 APE 150 MTS fg 0% bg 0% AO@41.5C GPU@41.5C Tdiode@45C PMIC@100C AUX@42C CPU@44.5C thermal@42.6C Tboard@42C GPU 771/583 CPU 4935/1989 SOC 2929/1904 CV 0/0 VDDRQ 308/205 SYS5V 2463/2060
RAM 3148/15823MB (lfb 2691x4MB) SWAP 0/7911MB (cached 0MB) CPU [99%@2265,3%@2265,2%@2265,2%@2265,2%@2265,17%@2265,4%@2265,5%@2265] EMC_FREQ 2%@2133 GR3D_FREQ 3%@318 APE 150 MTS fg 0% bg 0% AO@41.5C GPU@41C Tdiode@45C PMIC@100C AUX@42C CPU@44C thermal@42.3C Tboard@42C GPU 617/587 CPU 3243/2115 SOC 2777/1991 CV 0/0 VDDRQ 154/200 SYS5V 2382/2092
RAM 3148/15823MB (lfb 2691x4MB) SWAP 0/7911MB (cached 0MB) CPU [3%@2265,1%@2265,6%@2265,7%@2265,1%@2265,2%@2265,97%@2265,0%@2265] EMC_FREQ 1%@2133 GR3D_FREQ 19%@318 APE 150 MTS fg 0% bg 15% AO@41.5C GPU@41.5C Tdiode@45C PMIC@100C AUX@42C CPU@44.5C thermal@42.45C Tboard@42C GPU 617/589 CPU 3243/2217 SOC 2777/2063 CV 0/0 VDDRQ 154/196 SYS5V 2382/2118
RAM 3132/15823MB (lfb 2693x4MB) SWAP 0/7911MB (cached 0MB) CPU [2%@2265,2%@2265,1%@2265,0%@2265,1%@2265,0%@2265,100%@2265,7%@2265] EMC_FREQ 1%@2133 GR3D_FREQ 0%@318 APE 150 MTS fg 0% bg 22% AO@42C GPU@41.5C Tdiode@45C PMIC@100C AUX@42C CPU@44.5C thermal@42.6C Tboard@42C GPU 463/579 CPU 3397/2316 SOC 2777/2122 CV 0/0 VDDRQ 154/192 SYS5V 2382/2140
RAM 3132/15823MB (lfb 2693x4MB) SWAP 0/7911MB (cached 0MB) CPU [7%@2265,8%@2265,10%@2265,13%@2265,4%@2265,8%@2265,100%@2265,4%@2265] EMC_FREQ 1%@2133 GR3D_FREQ 13%@318 APE 150 MTS fg 0% bg 24% AO@42C GPU@41.5C Tdiode@45.25C PMIC@100C AUX@42C CPU@45C thermal@42.6C Tboard@42C GPU 617/582 CPU 4011/2446 SOC 2930/2184 CV 0/0 VDDRQ 308/201 SYS5V 2382/2159
RAM 3138/15823MB (lfb 2692x4MB) SWAP 0/7911MB (cached 0MB) CPU [12%@2265,12%@2265,22%@2265,13%@2265,8%@2265,14%@2265,100%@2265,11%@2265] EMC_FREQ 3%@2133 GR3D_FREQ 11%@318 APE 150 MTS fg 0% bg 24% AO@42C GPU@41.5C Tdiode@45.25C PMIC@100C AUX@42.5C CPU@44.5C thermal@42.6C Tboard@42C GPU 771/595 CPU 4320/2580 SOC 2929/2237 CV 0/0 VDDRQ 308/209 SYS5V 2463/2181
RAM 3137/15823MB (lfb 2692x4MB) SWAP 0/7911MB (cached 0MB) CPU [6%@2265,6%@2265,8%@2265,5%@2265,5%@2265,10%@2265,100%@2265,2%@2265] EMC_FREQ 1%@2133 GR3D_FREQ 0%@318 APE 150 MTS fg 0% bg 17% AO@42C GPU@41.5C Tdiode@45.5C PMIC@100C AUX@42.5C CPU@44.5C thermal@42.6C Tboard@42C GPU 463/586 CPU 3397/2634 SOC 2777/2273 CV 0/0 VDDRQ 154/205 SYS5V 2382/2194
RAM 3186/15823MB (lfb 2675x4MB) SWAP 0/7911MB (cached 0MB) CPU [15%@2265,12%@2265,18%@2265,14%@2265,15%@2265,20%@2265,77%@2265,10%@2265] EMC_FREQ 3%@2133 GR3D_FREQ 27%@318 APE 150 MTS fg 0% bg 25% AO@42C GPU@42C Tdiode@45.5C PMIC@100C AUX@42.5C CPU@44.5C thermal@42.75C Tboard@42C GPU 925/607 CPU 3703/2701 SOC 2930/2314 CV 0/0 VDDRQ 462/221 SYS5V 2665/2223
RAM 3395/15823MB (lfb 2607x4MB) SWAP 0/7911MB (cached 0MB) CPU [8%@2265,9%@2265,55%@2265,22%@2265,4%@2111,3%@2111,10%@2251,23%@2265] EMC_FREQ 3%@2133 GR3D_FREQ 6%@318 APE 150 MTS fg 0% bg 17% AO@42C GPU@41.5C Tdiode@45.5C PMIC@100C AUX@42C CPU@44C thermal@42.9C Tboard@42C GPU 617/608 CPU 2934/2715 SOC 2777/2342 CV 0/0 VDDRQ 308/226 SYS5V 2665/2249
RAM 3622/15823MB (lfb 2534x4MB) SWAP 0/7911MB (cached 0MB) CPU [11%@2265,13%@2265,46%@2265,26%@2265,4%@2188,4%@2232,10%@2265,6%@2265] EMC_FREQ 2%@2133 GR3D_FREQ 4%@420 APE 150 MTS fg 0% bg 15% AO@42C GPU@41.5C Tdiode@45.5C PMIC@100C AUX@42C CPU@44C thermal@42.6C Tboard@42C GPU 617/608 CPU 2625/2710 SOC 2778/2366 CV 0/0 VDDRQ 308/231 SYS5V 2665/2272
RAM 4054/15823MB (lfb 2393x4MB) SWAP 0/7911MB (cached 0MB) CPU [7%@2265,8%@2265,23%@2077,41%@1881,5%@1881,6%@1881,8%@1881,5%@1881] EMC_FREQ 8%@1065 GR3D_FREQ 0%@522 APE 150 MTS fg 0% bg 18% AO@42C GPU@41.5C Tdiode@45.5C PMIC@100C AUX@42C CPU@44C thermal@42.75C Tboard@42C GPU 617/609 CPU 2008/2673 SOC 2624/2379 CV 0/0 VDDRQ 308/235 SYS5V 2948/2308
RAM 4132/15823MB (lfb 2316x4MB) SWAP 0/7911MB (cached 0MB) CPU [7%@2265,5%@2265,23%@2265,16%@2265,6%@2265,8%@2265,7%@2265,6%@2265] EMC_FREQ 3%@2133 GR3D_FREQ 31%@828 APE 150 MTS fg 0% bg 12% AO@42C GPU@42C Tdiode@45.5C PMIC@100C AUX@42C CPU@44.5C thermal@42.3C Tboard@42C GPU 1081/632 CPU 2007/2639 SOC 2624/2392 CV 0/0 VDDRQ 308/238 SYS5V 2988/2342
RAM 4407/15823MB (lfb 2225x4MB) SWAP 0/7911MB (cached 0MB) CPU [9%@2265,2%@2265,48%@2265,15%@2265,8%@2265,6%@2265,9%@2265,6%@2265] EMC_FREQ 3%@2133 GR3D_FREQ 3%@675 APE 150 MTS fg 0% bg 14% AO@42C GPU@41.5C Tdiode@45.5C PMIC@100C AUX@42C CPU@44C thermal@42.6C Tboard@43C GPU 617/632 CPU 2316/2624 SOC 2932/2417 CV 0/0 VDDRQ 308/242 SYS5V 2826/2365
RAM 4576/15823MB (lfb 2181x4MB) SWAP 0/7911MB (cached 0MB) CPU [24%@1701,13%@2029,26%@1824,36%@1727,8%@1870,10%@1883,11%@2155,11%@2188] EMC_FREQ 19%@2133 GR3D_FREQ 59%@1236 APE 150 MTS fg 0% bg 14% AO@42.5C GPU@44C Tdiode@45.75C PMIC@100C AUX@42C CPU@45C thermal@43.05C Tboard@43C GPU 4463/806 CPU 2925/2638 SOC 3692/2475 CV 0/0 VDDRQ 1076/279 SYS5V 2907/2390
RAM 4576/15823MB (lfb 2181x4MB) SWAP 0/7911MB (cached 0MB) CPU [12%@1190,10%@1190,27%@1190,36%@1190,3%@1190,10%@1401,8%@1420,2%@1691] EMC_FREQ 36%@2133 GR3D_FREQ 86%@1377 APE 150 MTS fg 0% bg 17% AO@43C GPU@45.5C Tdiode@46.25C PMIC@100C AUX@42.5C CPU@44.5C thermal@43.85C Tboard@43C GPU 9369/1178 CPU 1536/2590 SOC 3991/2541 CV 0/0 VDDRQ 1228/321 SYS5V 2907/2412
RAM 4581/15823MB (lfb 2178x4MB) SWAP 0/7911MB (cached 0MB) CPU [19%@2265,14%@2265,43%@2265,33%@2265,12%@2265,10%@2265,12%@2265,11%@2265] EMC_FREQ 41%@2133 GR3D_FREQ 9%@1377 APE 150 MTS fg 0% bg 24% AO@43C GPU@44C Tdiode@46.5C PMIC@100C AUX@42.5C CPU@46.5C thermal@43.85C Tboard@43C GPU 6917/1417 CPU 3382/2623 SOC 3841/2595 CV 0/0 VDDRQ 1075/352 SYS5V 2867/2431
RAM 4594/15823MB (lfb 2174x4MB) SWAP 0/7911MB (cached 0MB) CPU [9%@2265,8%@2265,99%@2265,4%@2265,6%@2265,9%@2265,4%@2265,4%@2265] EMC_FREQ 20%@2133 GR3D_FREQ 0%@1377 APE 150 MTS fg 0% bg 46% AO@42.5C GPU@43C Tdiode@46.25C PMIC@100C AUX@42.5C CPU@46C thermal@43.85C Tboard@43C GPU 2004/1441 CPU 4779/2709 SOC 2926/2608 CV 0/0 VDDRQ 308/350 SYS5V 2463/2432
RAM 4594/15823MB (lfb 2174x4MB) SWAP 0/7911MB (cached 0MB) CPU [2%@2265,1%@2265,100%@2265,0%@2265,0%@2265,0%@2265,3%@2235,4%@2265] EMC_FREQ 9%@2133 GR3D_FREQ 0%@1377 APE 150 MTS fg 0% bg 49% AO@43C GPU@43C Tdiode@46C PMIC@100C AUX@42.5C CPU@45C thermal@43.7C Tboard@43C GPU 1234/1433 CPU 4011/2759 SOC 2930/2621 CV 0/0 VDDRQ 154/343 SYS5V 2423/2432
RAM 4595/15823MB (lfb 2174x4MB) SWAP 0/7911MB (cached 0MB) CPU [0%@2265,0%@2265,46%@2265,52%@2265,0%@2265,0%@2265,3%@2265,4%@2265] EMC_FREQ 4%@2133 GR3D_FREQ 0%@1377 APE 150 MTS fg 0% bg 46% AO@42.5C GPU@42.5C Tdiode@46.25C PMIC@100C AUX@42.5C CPU@45C thermal@43.4C Tboard@43C GPU 1080/1420 CPU 3703/2794 SOC 2776/2627 CV 0/0 VDDRQ 154/336 SYS5V 2382/2430
RAM 4596/15823MB (lfb 2174x4MB) SWAP 0/7911MB (cached 0MB) CPU [2%@2265,0%@2265,49%@2265,51%@2265,1%@2265,0%@2265,3%@2265,5%@2265] EMC_FREQ 2%@2133 GR3D_FREQ 0%@1377 APE 150 MTS fg 0% bg 39% AO@42.5C GPU@43C Tdiode@46C PMIC@100C AUX@42.5C CPU@45C thermal@43.55C Tboard@43C GPU 1080/1407 CPU 3396/2815 SOC 2777/2632 CV 0/0 VDDRQ 154/329 SYS5V 2382/2428
RAM 4595/15823MB (lfb 2174x4MB) SWAP 0/7911MB (cached 0MB) CPU [1%@1253,1%@1703,47%@2265,53%@2265,0%@2265,1%@2265,1%@2265,4%@2265] EMC_FREQ 1%@2133 GR3D_FREQ 0%@1377 APE 150 MTS fg 0% bg 37% AO@42.5C GPU@43C Tdiode@46C PMIC@100C AUX@42.5C CPU@45C thermal@43.4C Tboard@43C GPU 1080/1396 CPU 3396/2835 SOC 2777/2637 CV 0/0 VDDRQ 154/323 SYS5V 2382/2427
RAM 4596/15823MB (lfb 2174x4MB) SWAP 0/7911MB (cached 0MB) CPU [4%@1958,2%@1997,44%@2112,52%@2112,1%@2111,0%@2229,4%@2265,2%@2265] EMC_FREQ 2%@2133 GR3D_FREQ 17%@1377 APE 150 MTS fg 0% bg 35% AO@42.5C GPU@44C Tdiode@46C PMIC@100C AUX@42.5C CPU@45C thermal@43.4C Tboard@43C GPU 2467/1432 CPU 2931/2839 SOC 2929/2647 CV 0/0 VDDRQ 308/323 SYS5V 2463/2428
RAM 4596/15823MB (lfb 2171x4MB) SWAP 0/7911MB (cached 0MB) CPU [2%@2265,3%@2265,43%@2265,37%@2035,4%@1743,4%@1881,5%@1881,3%@2198] EMC_FREQ 28%@2133 GR3D_FREQ 50%@1377 APE 150 MTS fg 0% bg 17% AO@43C GPU@45C Tdiode@46.5C PMIC@100C AUX@42.5C CPU@45.5C thermal@43.85C Tboard@43C GPU 6461/1594 CPU 1846/2807 SOC 3998/2690 CV 0/0 VDDRQ 1076/347 SYS5V 2867/2442
RAM 4596/15823MB (lfb 2171x4MB) SWAP 0/7911MB (cached 0MB) CPU [5%@2035,3%@2035,33%@2035,40%@2213,7%@2265,4%@2265,7%@2265,4%@2265] EMC_FREQ 39%@2133 GR3D_FREQ 61%@1377 APE 150 MTS fg 0% bg 15% AO@43.5C GPU@45C Tdiode@46.75C PMIC@100C AUX@42.5C CPU@45.5C thermal@44C Tboard@43C GPU 6920/1760 CPU 1692/2772 SOC 3996/2731 CV 0/0 VDDRQ 1229/374 SYS5V 2907/2457
RAM 4597/15823MB (lfb 2170x4MB) SWAP 0/7911MB (cached 0MB) CPU [6%@2265,6%@2265,29%@2265,29%@2265,10%@2265,9%@2265,10%@2265,6%@2265] EMC_FREQ 51%@2133 GR3D_FREQ 49%@1377 APE 150 MTS fg 0% bg 14% AO@44C GPU@45.5C Tdiode@47C PMIC@100C AUX@43C CPU@46C thermal@44.35C Tboard@43C GPU 7071/1921 CPU 1998/2748 SOC 4302/2779 CV 0/0 VDDRQ 1382/405 SYS5V 2948/2471
RAM 4596/15823MB (lfb 2170x4MB) SWAP 0/7911MB (cached 0MB) CPU [5%@1343,1%@1344,27%@1343,40%@1344,4%@1381,5%@1572,7%@1574,4%@1779] EMC_FREQ 50%@2133 GR3D_FREQ 55%@1377 APE 150 MTS fg 0% bg 15% AO@43.5C GPU@45.5C Tdiode@47C PMIC@100C AUX@43C CPU@45.5C thermal@44.5C Tboard@43C GPU 6923/2068 CPU 1384/2708 SOC 3996/2814 CV 0/0 VDDRQ 1229/429 SYS5V 2907/2484
RAM 4597/15823MB (lfb 2170x4MB) SWAP 0/7911MB (cached 0MB) CPU [14%@1574,4%@1574,13%@1590,28%@1804,7%@1521,6%@1404,14%@1574,21%@1716] EMC_FREQ 55%@2133 GR3D_FREQ 16%@1377 APE 150 MTS fg 0% bg 11% AO@44C GPU@45.5C Tdiode@47.25C PMIC@100C AUX@43.5C CPU@45.5C thermal@44.85C Tboard@43C GPU 7686/2229 CPU 1844/2683 SOC 4148/2852 CV 0/0 VDDRQ 1228/452 SYS5V 2907/2496
RAM 4597/15823MB (lfb 2170x4MB) SWAP 0/7911MB (cached 0MB) CPU [5%@2265,4%@2265,36%@2265,31%@2265,8%@2265,10%@2265,13%@2265,2%@2265] EMC_FREQ 60%@2133 GR3D_FREQ 54%@1377 APE 150 MTS fg 0% bg 14% AO@44C GPU@45.5C Tdiode@47.25C PMIC@100C AUX@43.5C CPU@46C thermal@45C Tboard@43C GPU 7225/2368 CPU 1999/2664 SOC 4302/2893 CV 0/0 VDDRQ 1382/478 SYS5V 2988/2510
RAM 4597/15823MB (lfb 2170x4MB) SWAP 0/7911MB (cached 0MB) CPU [6%@1346,5%@1497,25%@1481,40%@1189,0%@1190,1%@1339,6%@1343,5%@1550] EMC_FREQ 48%@2133 GR3D_FREQ 80%@1377 APE 150 MTS fg 0% bg 17% AO@44C GPU@45.5C Tdiode@47.25C PMIC@100C AUX@43.5C CPU@45.5C thermal@45C Tboard@43C GPU 7385/2503 CPU 1846/2642 SOC 3689/2914 CV 0/0 VDDRQ 922/490 SYS5V 2746/2516
RAM 4597/15823MB (lfb 2170x4MB) SWAP 0/7911MB (cached 0MB) CPU [2%@2265,1%@2265,1%@2265,96%@2265,2%@2265,0%@2265,4%@2265,2%@2265] EMC_FREQ 23%@2133 GR3D_FREQ 0%@1377 APE 150 MTS fg 0% bg 33% AO@43.5C GPU@44C Tdiode@47C PMIC@100C AUX@43.5C CPU@45.5C thermal@44.8C Tboard@43C GPU 1851/2486 CPU 3086/2654 SOC 2930/2915 CV 0/0 VDDRQ 308/485 SYS5V 2423/2514
RAM 4597/15823MB (lfb 2170x4MB) SWAP 0/7911MB (cached 0MB) CPU [0%@2265,0%@2265,1%@2265,100%@2265,0%@2265,0%@2265,5%@2265,3%@2265] EMC_FREQ 10%@2133 GR3D_FREQ 0%@1377 APE 150 MTS fg 0% bg 34% AO@43C GPU@43.5C Tdiode@46.75C PMIC@100C AUX@43C CPU@45.5C thermal@44.25C Tboard@43C GPU 1234/2454 CPU 3087/2665 SOC 2931/2915 CV 0/0 VDDRQ 154/477 SYS5V 2423/2511
RAM 4597/15823MB (lfb 2170x4MB) SWAP 0/7911MB (cached 0MB) CPU [1%@2265,0%@1191,1%@2180,98%@2265,0%@2265,1%@2265,3%@2265,3%@2265] EMC_FREQ 5%@2133 GR3D_FREQ 0%@1377 APE 150 MTS fg 0% bg 34% AO@43.5C GPU@43.5C Tdiode@46.75C PMIC@100C AUX@43.5C CPU@45.5C thermal@43.9C Tboard@43C GPU 1080/2420 CPU 3087/2676 SOC 2777/2912 CV 0/0 VDDRQ 154/468 SYS5V 2423/2509
RAM 4598/15823MB (lfb 2170x4MB) SWAP 0/7911MB (cached 0MB) CPU [0%@2265,0%@1265,0%@2037,100%@2265,2%@2265,0%@2265,1%@2265,4%@2265] EMC_FREQ 3%@2133 GR3D_FREQ 5%@1377 APE 150 MTS fg 0% bg 35% AO@43C GPU@43.5C Tdiode@46.75C PMIC@100C AUX@43C CPU@45.5C thermal@44.1C Tboard@43C GPU 1080/2387 CPU 3241/2689 SOC 2777/2908 CV 0/0 VDDRQ 154/461 SYS5V 2423/2507
RAM 4598/15823MB (lfb 2170x4MB) SWAP 0/7911MB (cached 0MB) CPU [1%@2265,0%@2265,95%@2265,4%@2265,1%@2265,0%@2265,2%@2265,2%@2265] EMC_FREQ 1%@2133 GR3D_FREQ 0%@1377 APE 150 MTS fg 0% bg 36% AO@43C GPU@43.5C Tdiode@46.75C PMIC@100C AUX@43C CPU@46C thermal@44.1C Tboard@43C GPU 1080/2356 CPU 3241/2702 SOC 2777/2905 CV 0/0 VDDRQ 154/453 SYS5V 2382/2504
RAM 4598/15823MB (lfb 2170x4MB) SWAP 0/7911MB (cached 0MB) CPU [3%@2265,1%@2265,59%@2265,39%@2265,0%@2265,0%@2265,2%@2265,4%@2265] EMC_FREQ 2%@2133 GR3D_FREQ 0%@1377 APE 150 MTS fg 0% bg 31% AO@43C GPU@43.5C Tdiode@46.75C PMIC@100C AUX@43.5C CPU@45.5C thermal@43.9C Tboard@43C GPU 1389/2333 CPU 3087/2711 SOC 2931/2906 CV 0/0 VDDRQ 154/446 SYS5V 2423/2502
RAM 4598/15823MB (lfb 2170x4MB) SWAP 0/7911MB (cached 0MB) CPU [3%@2265,1%@2265,0%@2265,100%@2265,0%@2265,0%@2265,2%@2265,1%@2265] EMC_FREQ 1%@2133 GR3D_FREQ 0%@1377 APE 150 MTS fg 0% bg 31% AO@43C GPU@43.5C Tdiode@46.5C PMIC@100C AUX@43.5C CPU@45.5C thermal@43.9C Tboard@43C GPU 1080/2305 CPU 3087/2720 SOC 2777/2903 CV 0/0 VDDRQ 154/440 SYS5V 2382/2499
RAM 4599/15823MB (lfb 2170x4MB) SWAP 0/7911MB (cached 0MB) CPU [3%@2265,0%@2265,2%@2265,99%@2265,0%@2265,0%@2265,3%@2265,3%@2265] EMC_FREQ 1%@2133 GR3D_FREQ 0%@1377 APE 150 MTS fg 0% bg 34% AO@43C GPU@43.5C Tdiode@46.5C PMIC@100C AUX@43C CPU@45.5C thermal@44.05C Tboard@43C GPU 1080/2278 CPU 3241/2731 SOC 2777/2900 CV 0/0 VDDRQ 154/433 SYS5V 2382/2497
RAM 4599/15823MB (lfb 2170x4MB) SWAP 0/7911MB (cached 0MB) CPU [1%@2265,0%@2265,1%@2265,100%@2265,0%@2265,0%@2265,5%@2265,3%@2265] EMC_FREQ 1%@2133 GR3D_FREQ 0%@1377 APE 150 MTS fg 0% bg 33% AO@43C GPU@43.5C Tdiode@46.5C PMIC@100C AUX@43C CPU@45.5C thermal@43.95C Tboard@43C GPU 1080/2252 CPU 3087/2739 SOC 2777/2897 CV 0/0 VDDRQ 154/427 SYS5V 2382/2494
RAM 4599/15823MB (lfb 2170x4MB) SWAP 0/7911MB (cached 0MB) CPU [1%@2265,0%@2265,0%@2265,99%@2265,0%@2265,0%@2265,1%@2265,4%@2265] EMC_FREQ 1%@2133 GR3D_FREQ 0%@1377 APE 150 MTS fg 0% bg 33% AO@43C GPU@43.5C Tdiode@46.5C PMIC@100C AUX@43C CPU@45.5C thermal@43.95C Tboard@43C GPU 1080/2227 CPU 3087/2747 SOC 2777/2895 CV 0/0 VDDRQ 154/422 SYS5V 2382/2492
RAM 4598/15823MB (lfb 2170x4MB) SWAP 0/7911MB (cached 0MB) CPU [1%@2265,0%@2265,2%@2265,100%@2265,0%@2265,0%@2265,2%@2265,4%@2265] EMC_FREQ 1%@2133 GR3D_FREQ 0%@828 APE 150 MTS fg 0% bg 33% AO@43C GPU@43C Tdiode@46.5C PMIC@100C AUX@43C CPU@45.5C thermal@43.95C Tboard@43C GPU 617/2193 CPU 3088/2754 SOC 2777/2892 CV 0/0 VDDRQ 154/416 SYS5V 2382/2490
RAM 4613/15823MB (lfb 2167x4MB) SWAP 0/7911MB (cached 0MB) CPU [6%@1574,3%@1343,36%@1343,45%@1574,0%@1573,1%@1574,6%@1564,4%@1651] EMC_FREQ 28%@2133 GR3D_FREQ 69%@1236 APE 150 MTS fg 0% bg 13% AO@43.5C GPU@45C Tdiode@47C PMIC@100C AUX@43.5C CPU@45.5C thermal@43.75C Tboard@43C GPU 4925/2249 CPU 1847/2735 SOC 4154/2918 CV 0/0 VDDRQ 1384/436 SYS5V 2948/2499
RAM 4613/15823MB (lfb 2167x4MB) SWAP 0/7911MB (cached 0MB) CPU [8%@2252,4%@2265,26%@2094,43%@1496,3%@1573,3%@1984,5%@2265,2%@2265] EMC_FREQ 50%@2133 GR3D_FREQ 75%@1377 APE 150 MTS fg 0% bg 10% AO@44C GPU@46C Tdiode@47.25C PMIC@100C AUX@43.5C CPU@46C thermal@44.65C Tboard@43C GPU 8755/2379 CPU 1382/2708 SOC 4452/2949 CV 0/0 VDDRQ 1535/458 SYS5V 3064/2510
RAM 4613/15823MB (lfb 2167x4MB) SWAP 0/7911MB (cached 0MB) CPU [4%@2265,4%@2265,34%@2265,38%@2265,9%@2265,6%@2265,10%@1957,3%@1978] EMC_FREQ 50%@2133 GR3D_FREQ 29%@1377 APE 150 MTS fg 0% bg 12% AO@44.5C GPU@45.5C Tdiode@47.5C PMIC@100C AUX@43.5C CPU@46C thermal@45C Tboard@43C GPU 7382/2477 CPU 1692/2688 SOC 3843/2966 CV 0/0 VDDRQ 1075/470 SYS5V 2826/2516
RAM 4612/15823MB (lfb 2167x4MB) SWAP 0/7911MB (cached 0MB) CPU [3%@2265,3%@2265,36%@2265,34%@2265,6%@2265,7%@2265,9%@2265,6%@2265] EMC_FREQ 53%@2133 GR3D_FREQ 75%@1377 APE 150 MTS fg 0% bg 14% AO@44.5C GPU@46C Tdiode@47.5C PMIC@100C AUX@43.5C CPU@46C thermal@45C Tboard@44C GPU 7532/2574 CPU 1844/2672 SOC 4148/2989 CV 0/0 VDDRQ 1228/484 SYS5V 2948/2525
RAM 4613/15823MB (lfb 2167x4MB) SWAP 0/7911MB (cached 0MB) CPU [5%@1344,4%@1343,28%@1343,45%@1343,3%@1421,2%@1420,6%@1691,4%@1727] EMC_FREQ 48%@2133 GR3D_FREQ 50%@1377 APE 150 MTS fg 0% bg 13% AO@44.5C GPU@46C Tdiode@47.75C PMIC@100C AUX@43.5C CPU@46C thermal@45.15C Tboard@44C GPU 6769/2653 CPU 1692/2653 SOC 3844/3005 CV 0/0 VDDRQ 1076/495 SYS5V 2826/2530
RAM 4613/15823MB (lfb 2167x4MB) SWAP 0/7911MB (cached 0MB) CPU [8%@2265,5%@2265,39%@2174,30%@1727,6%@1728,6%@2062,8%@2111,4%@2251] EMC_FREQ 54%@2133 GR3D_FREQ 55%@1377 APE 150 MTS fg 0% bg 11% AO@44.5C GPU@46C Tdiode@47.75C PMIC@100C AUX@44C CPU@46.5C thermal@45C Tboard@44C GPU 7074/2735 CPU 1537/2633 SOC 4150/3026 CV 0/0 VDDRQ 1229/509 SYS5V 2948/2538
RAM 4612/15823MB (lfb 2167x4MB) SWAP 0/7911MB (cached 0MB) CPU [5%@1689,4%@2188,48%@2188,29%@2199,3%@2265,3%@2265,10%@2226,3%@2264] EMC_FREQ 57%@2133 GR3D_FREQ 51%@1377 APE 150 MTS fg 0% bg 15% AO@44.5C GPU@46C Tdiode@47.75C PMIC@100C AUX@44C CPU@46.5C thermal@45.15C Tboard@44C GPU 7228/2817 CPU 1844/2618 SOC 4302/3049 CV 0/0 VDDRQ 1382/525 SYS5V 2948/2546
RAM 4613/15823MB (lfb 2167x4MB) SWAP 0/7911MB (cached 0MB) CPU [4%@1190,1%@1253,30%@1420,39%@1420,5%@1330,3%@1485,6%@1497,4%@1704] EMC_FREQ 53%@2133 GR3D_FREQ 54%@1377 APE 150 MTS fg 0% bg 15% AO@44.5C GPU@46C Tdiode@48C PMIC@100C AUX@44C CPU@46C thermal@45.35C Tboard@44C GPU 6769/2887 CPU 1538/2599 SOC 3998/3066 CV 0/0 VDDRQ 1229/537 SYS5V 2907/2552
RAM 4612/15823MB (lfb 2167x4MB) SWAP 0/7911MB (cached 0MB) CPU [8%@1223,4%@1190,22%@1339,25%@1414,13%@1420,19%@1420,5%@1420,5%@1497] EMC_FREQ 60%@2133 GR3D_FREQ 80%@1377 APE 150 MTS fg 0% bg 10% AO@45C GPU@46.5C Tdiode@48C PMIC@100C AUX@44C CPU@46.5C thermal@45.5C Tboard@44C GPU 7840/2974 CPU 1998/2589 SOC 4300/3088 CV 0/0 VDDRQ 1381/552 SYS5V 2948/2559
RAM 4613/15823MB (lfb 2167x4MB) SWAP 0/7911MB (cached 0MB) CPU [6%@2265,3%@2265,33%@2265,44%@2265,5%@2265,2%@2265,7%@2265,3%@2265] EMC_FREQ 61%@2133 GR3D_FREQ 61%@1377 APE 150 MTS fg 0% bg 13% AO@44.5C GPU@46.5C Tdiode@48C PMIC@100C AUX@44C CPU@46.5C thermal@45.65C Tboard@44C GPU 7225/3048 CPU 1845/2576 SOC 4302/3109 CV 0/0 VDDRQ 1382/567 SYS5V 2988/2566
RAM 4612/15823MB (lfb 2167x4MB) SWAP 0/7911MB (cached 0MB) CPU [5%@1190,2%@1190,43%@1190,32%@1190,2%@1190,2%@1267,6%@1267,8%@1413] EMC_FREQ 59%@2133 GR3D_FREQ 50%@1377 APE 150 MTS fg 0% bg 11% AO@45C GPU@46.5C Tdiode@48C PMIC@100C AUX@44C CPU@46.5C thermal@45.65C Tboard@44C GPU 7228/3118 CPU 1384/2555 SOC 4150/3127 CV 0/0 VDDRQ 1229/578 SYS5V 2907/2572
RAM 4613/15823MB (lfb 2167x4MB) SWAP 0/7911MB (cached 0MB) CPU [7%@2265,4%@2265,52%@2265,26%@2265,3%@2265,8%@2265,5%@2265,2%@2265] EMC_FREQ 62%@2133 GR3D_FREQ 72%@1377 APE 150 MTS fg 0% bg 11% AO@45C GPU@47C Tdiode@48.25C PMIC@100C AUX@44C CPU@47C thermal@45.5C Tboard@44C GPU 7990/3200 CPU 1843/2544 SOC 4300/3146 CV 0/0 VDDRQ 1381/591 SYS5V 2983/2579
RAM 4612/15823MB (lfb 2165x4MB) SWAP 0/7911MB (cached 0MB) CPU [2%@2265,1%@2265,18%@2265,68%@2265,5%@2265,3%@2265,6%@2265,1%@2265] EMC_FREQ 41%@2133 GR3D_FREQ 2%@1377 APE 150 MTS fg 0% bg 21% AO@44.5C GPU@45C Tdiode@48C PMIC@100C AUX@44C CPU@47C thermal@45.65C Tboard@44C GPU 4006/3213 CPU 2926/2550 SOC 3387/3150 CV 0/0 VDDRQ 615/592 SYS5V 2624/2580
RAM 3406/15823MB (lfb 2291x4MB) SWAP 0/7911MB (cached 0MB) CPU [10%@1190,6%@1190,18%@1190,5%@1190,2%@1190,11%@1190,2%@1190,35%@1400] EMC_FREQ 23%@2133 GR3D_FREQ 0%@1377 APE 150 MTS fg 0% bg 21% AO@44.5C GPU@44.5C Tdiode@48C PMIC@100C AUX@44C CPU@46.5C thermal@46.1C Tboard@44C GPU 2161/3196 CPU 2006/2541 SOC 3084/3149 CV 0/0 VDDRQ 308/587 SYS5V 2463/2578
RAM 3406/15823MB (lfb 2291x4MB) SWAP 0/7911MB (cached 0MB) CPU [4%@1497,1%@1497,1%@1497,1%@1497,2%@1811,4%@1891,0%@2112,0%@2124] EMC_FREQ 10%@2133 GR3D_FREQ 0%@318 APE 150 MTS fg 0% bg 5% AO@44.5C GPU@44C Tdiode@47.5C PMIC@100C AUX@44C CPU@45.5C thermal@44.75C Tboard@44C GPU 1082/3162 CPU 773/2513 SOC 2626/3141 CV 0/0 VDDRQ 154/580 SYS5V 2261/2573
RAM 3406/15823MB (lfb 2291x4MB) SWAP 0/7911MB (cached 0MB) CPU [3%@1190,1%@1190,1%@1190,2%@1190,3%@1190,3%@1190,3%@1326,0%@1344] EMC_FREQ 16%@665 GR3D_FREQ 0%@318 APE 150 MTS fg 0% bg 6% AO@44C GPU@43.5C Tdiode@47.25C PMIC@100C AUX@43.5C CPU@44.5C thermal@44.15C Tboard@44C GPU 619/3123 CPU 774/2486 SOC 1237/3111 CV 0/0 VDDRQ 154/573 SYS5V 1820/2561
RAM 3406/15823MB (lfb 2291x4MB) SWAP 0/7911MB (cached 0MB) CPU [25%@1190,19%@1190,5%@1190,2%@1190,9%@1190,8%@1190,2%@1408,3%@1420] EMC_FREQ 8%@665 GR3D_FREQ 5%@318 APE 150 MTS fg 0% bg 12% AO@44C GPU@43.5C Tdiode@47.25C PMIC@100C AUX@43.5C CPU@44.5C thermal@43.8C Tboard@44C GPU 619/3084 CPU 1083/2464 SOC 1237/3082 CV 0/0 VDDRQ 154/567 SYS5V 1820/2550
RAM 3406/15823MB (lfb 2291x4MB) SWAP 0/7911MB (cached 0MB) CPU [7%@1190,11%@1190,1%@1190,3%@1190,3%@1190,5%@1190,3%@1190,0%@1414] EMC_FREQ 5%@665 GR3D_FREQ 0%@318 APE 150 MTS fg 0% bg 9% AO@43.5C GPU@43C Tdiode@47C PMIC@100C AUX@43.5C CPU@44.5C thermal@43.95C Tboard@44C GPU 464/3044 CPU 774/2439 SOC 1083/3052 CV 0/0 VDDRQ 154/561 SYS5V 1779/2538
RAM 3406/15823MB (lfb 2291x4MB) SWAP 0/7911MB (cached 0MB) CPU [10%@1190,6%@1190,1%@1190,0%@1190,1%@1267,0%@1267,2%@1267,2%@1479] EMC_FREQ 3%@665 GR3D_FREQ 0%@318 APE 150 MTS fg 0% bg 5% AO@43.5C GPU@43C Tdiode@47C PMIC@100C AUX@43.5C CPU@44.5C thermal@43.65C Tboard@44C GPU 464/3006 CPU 774/2414 SOC 1083/3022 CV 0/0 VDDRQ 154/555 SYS5V 1779/2527
RAM 3406/15823MB (lfb 2291x4MB) SWAP 0/7911MB (cached 0MB) CPU [3%@1190,1%@1190,0%@1190,1%@1190,0%@1190,1%@1190,6%@1338,3%@1344] EMC_FREQ 2%@665 GR3D_FREQ 0%@318 APE 150 MTS fg 0% bg 5% AO@43.5C GPU@43C Tdiode@46.75C PMIC@100C AUX@43C CPU@44.5C thermal@43.45C Tboard@44C GPU 464/2968 CPU 774/2390 SOC 1237/2996 CV 0/0 VDDRQ 154/549 SYS5V 1820/2516
RAM 3406/15823MB (lfb 2291x4MB) SWAP 0/7911MB (cached 0MB) CPU [3%@1190,4%@1190,2%@1190,1%@1190,0%@1190,0%@1190,6%@1384,2%@1420] EMC_FREQ 2%@665 GR3D_FREQ 0%@318 APE 150 MTS fg 0% bg 7% AO@43C GPU@43C Tdiode@46.5C PMIC@100C AUX@43C CPU@44.5C thermal@43.45C Tboard@44C GPU 464/2932 CPU 774/2366 SOC 1392/2973 CV 0/0 VDDRQ 154/543 SYS5V 1820/2506

I do not see any RAM suspicious raise during activation.

Regards.

AastaLLL · October 19, 2021, 7:56am

Hi,

Could you share the source so we can test this in our environment?

Please noted that it is possible to deploy it on JetPack4.6.
Since you don’t need the TensorFlow but just the ONNX model for TensorRT deployment.

Thanks.

AastaLLL · October 25, 2021, 8:49am

Hi,

Thanks for your sharing.

Just check your model, it can work correctly on batchsize=4096 with trtexec.

$ /usr/src/tensorrt/bin/trtexec --onnx=data/model.onnx --batch=4096 --explicitBatch

It looks like the issue is from the TesterOnnxRADA.
Could you check if all the buffer can allow 4096 batches first?

Thanks.

igal.kroyter · October 26, 2021, 5:46am

hi @AastaLLL,

Thanks for investigating, I’ve run the above line with multiple batch sizes: 4096, 30k, 100k values and it looks like the average inference time is lowering the higher the batch size, while when I run it with the code I uploaded the average time increases. So I’m not sure what exactly is done in trtexec. BTW, my code was based on the SampleOnnxMNIST just without the data conversion network which preceded the “actual” inference network.

Could you please point to how to debug the memory allocation?

Regards.

AastaLLL · October 29, 2021, 5:55am

Hi,

Could you check the input/output buffer of TensorRT and make sure it can allow the batch size?
trtexec is also open-sourced. You can find it in the OSS GitHub directly:

Thanks.

igal.kroyter · October 31, 2021, 5:21pm

hi @AastaLLL,

how should I “check the input/output buffer of TensorRT” ?

AastaLLL · November 12, 2021, 5:42am

Hi,

You can enable the verbose output to get more information about the error.

bool SampleDynamicReshape::build()
{
    ...
    sample::gLogger.setReportableSeverity(nvinfer1::ILogger::Severity::kVERBOSE);
    return buildPredictionEngine(builder) /*&& buildPreprocessorEngine(builder)*/;
}

We got the following log when deploying the app with batchsize=4096.

It looks like the error occurs when TensorRT wants to write out the output tensor.
So please double-check the output buffer of enqueue function:

...
[11/12/2021-13:30:58] [V] [TRT] Layer(Reformat): StatefulPartitionedCall/functional_1/projects.rada_tracks._layers.unet_1d_v1_1/classify/conv1d__464:0 copy, Tactic: 0, StatefulPartitionedCall/functional_1/projects.rada_tracks._layers.unet_1d_v1_1/classify/conv1d__464:0[Float(128,1)] -> tf_op_layer_concat[Float(128,1)]
[11/12/2021-13:30:58] [V] [TRT] Layer(Reformat): StatefulPartitionedCall/functional_1/projects.rada_tracks._layers.unet_1d_v1_2/classify/conv1d__372:0 copy, Tactic: 0, StatefulPartitionedCall/functional_1/projects.rada_tracks._layers.unet_1d_v1_2/classify/conv1d__372:0[Float(128,1)] -> tf_op_layer_concat[Float(128,1)]
[11/12/2021-13:30:58] [V] [TRT] Layer(Reformat): StatefulPartitionedCall/functional_1/projects.rada_tracks._layers.unet_1d_v1_3/classify/conv1d__280:0 copy, Tactic: 0, StatefulPartitionedCall/functional_1/projects.rada_tracks._layers.unet_1d_v1_3/classify/conv1d__280:0[Float(128,1)] -> tf_op_layer_concat[Float(128,1)]
 Preparation time: 73061951044[ns]
[11/12/2021-13:30:58] [E] [TRT] engine.cpp (986) - Cuda Error in executeInternal: 700 (an illegal memory access was encountered)
[11/12/2021-13:30:58] [E] [TRT] FAILED_EXECUTION: std::exception
 failed executeV2
&&&& FAILED TensorRT.tester_onnx_RADA # ./tester_onnx_rada --batch=4096
[11/12/2021-13:30:58] [E] [TRT] engine.cpp (179) - Cuda Error in ~ExecutionContext: 700 (an illegal memory access was encountered)
[11/12/2021-13:30:58] [E] [TRT] INTERNAL_ERROR: std::exception
[11/12/2021-13:30:58] [E] [TRT] Parameter check failed at: ../rtSafe/safeContext.cpp::terminateCommonContext::155, condition: cudnnDestroy(context.cudnn) failure.
[11/12/2021-13:30:58] [E] [TRT] Parameter check failed at: ../rtSafe/safeContext.cpp::terminateCommonContext::165, condition: cudaEventDestroy(context.start) failure.
[11/12/2021-13:30:58] [E] [TRT] Parameter check failed at: ../rtSafe/safeContext.cpp::terminateCommonContext::170, condition: cudaEventDestroy(context.stop) failure.
[11/12/2021-13:30:58] [E] [TRT] ../rtSafe/safeRuntime.cpp (32) - Cuda Error in free: 700 (an illegal memory access was encountered)
terminate called after throwing an instance of 'nvinfer1::CudaError'
  what():  std::exception
Aborted (core dumped)

Thanks.

igal.kroyter · November 18, 2021, 7:59am

hi @AastaLLL,

I’ve looked into the allocation functions in buffer.h file and saw no issue with allocation, and found no enqueue function. Please advise.

system · December 2, 2021, 8:00am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.