Regnet use group conv for all backbones.
I tested when input size is 512x384 regnet800mf could work but regnet1.6gf will raise some bugs.
I think this may be caused by DLA, and it may have some bugs with group conv, we also test resnet50 and it work well and is much larger than regnet1.6gf
here is the full traceback:
nvidia@nvidia-desktop:/xavierSD/jiangbin/devspace/workdir/fastld/testconv$ /usr/src/tensorrt/bin/trtexec --onnx=./rmdet.onnx --saveEngine=./out.trt --useDLACore=1 --allowGPUFallback
&&&& RUNNING TensorRT.trtexec # /usr/src/tensorrt/bin/trtexec --onnx=./rmdet.onnx --saveEngine=./out.trt --useDLACore=1 --allowGPUFallback
[04/22/2021-17:45:30] [I] === Model Options ===
[04/22/2021-17:45:30] [I] Format: ONNX
[04/22/2021-17:45:30] [I] Model: ./rmdet.onnx
[04/22/2021-17:45:30] [I] Output:
[04/22/2021-17:45:30] [I] === Build Options ===
[04/22/2021-17:45:30] [I] Max batch: 1
[04/22/2021-17:45:30] [I] Workspace: 16 MB
[04/22/2021-17:45:30] [I] minTiming: 1
[04/22/2021-17:45:30] [I] avgTiming: 8
[04/22/2021-17:45:30] [I] Precision: FP32
[04/22/2021-17:45:30] [I] Calibration:
[04/22/2021-17:45:30] [I] Safe mode: Disabled
[04/22/2021-17:45:30] [I] Save engine: ./out.trt
[04/22/2021-17:45:30] [I] Load engine:
[04/22/2021-17:45:30] [I] Builder Cache: Enabled
[04/22/2021-17:45:30] [I] NVTX verbosity: 0
[04/22/2021-17:45:30] [I] Inputs format: fp32:CHW
[04/22/2021-17:45:30] [I] Outputs format: fp32:CHW
[04/22/2021-17:45:30] [I] Input build shapes: model
[04/22/2021-17:45:30] [I] Input calibration shapes: model
[04/22/2021-17:45:30] [I] === System Options ===
[04/22/2021-17:45:30] [I] Device: 0
[04/22/2021-17:45:30] [I] DLACore: 1(With GPU fallback)
[04/22/2021-17:45:30] [I] Plugins:
[04/22/2021-17:45:30] [I] === Inference Options ===
[04/22/2021-17:45:30] [I] Batch: 1
[04/22/2021-17:45:30] [I] Input inference shapes: model
[04/22/2021-17:45:30] [I] Iterations: 10
[04/22/2021-17:45:30] [I] Duration: 3s (+ 200ms warm up)
[04/22/2021-17:45:30] [I] Sleep time: 0ms
[04/22/2021-17:45:30] [I] Streams: 1
[04/22/2021-17:45:30] [I] ExposeDMA: Disabled
[04/22/2021-17:45:30] [I] Spin-wait: Disabled
[04/22/2021-17:45:30] [I] Multithreading: Disabled
[04/22/2021-17:45:30] [I] CUDA Graph: Disabled
[04/22/2021-17:45:30] [I] Skip inference: Disabled
[04/22/2021-17:45:30] [I] Inputs:
[04/22/2021-17:45:30] [I] === Reporting Options ===
[04/22/2021-17:45:30] [I] Verbose: Disabled
[04/22/2021-17:45:30] [I] Averages: 10 inferences
[04/22/2021-17:45:30] [I] Percentile: 99
[04/22/2021-17:45:30] [I] Dump output: Disabled
[04/22/2021-17:45:30] [I] Profile: Disabled
[04/22/2021-17:45:30] [I] Export timing to JSON file:
[04/22/2021-17:45:30] [I] Export output to JSON file:
[04/22/2021-17:45:30] [I] Export profile to JSON file:
[04/22/2021-17:45:30] [I]
----------------------------------------------------------------
Input filename: ./rmdet.onnx
ONNX IR version: 0.0.6
Opset version: 11
Producer name: pytorch
Producer version: 1.7
Domain:
Model version: 0
Doc string:
----------------------------------------------------------------
[04/22/2021-17:45:31] [W] [TRT] Internal DLA error for layer Conv_120. Switching to GPU fallback.
[04/22/2021-17:45:31] [W] [TRT] Internal DLA error for layer Conv_120. Switching to GPU fallback.
[04/22/2021-17:45:31] [W] [TRT] Internal DLA error for layer Conv_127. Switching to GPU fallback.
[04/22/2021-17:45:31] [W] [TRT] Internal DLA error for layer Conv_127. Switching to GPU fallback.
[04/22/2021-17:45:31] [W] [TRT] Internal DLA error for layer ConvTranspose_138. Switching to GPU fallback.
[04/22/2021-17:45:31] [W] [TRT] Internal DLA error for layer ConvTranspose_138. Switching to GPU fallback.
[04/22/2021-17:45:31] [W] [TRT] Internal DLA error for layer ConvTranspose_144. Switching to GPU fallback.
[04/22/2021-17:45:31] [W] [TRT] Internal DLA error for layer ConvTranspose_144. Switching to GPU fallback.
[04/22/2021-17:45:31] [W] [TRT] Internal DLA error for layer ConvTranspose_150. Switching to GPU fallback.
[04/22/2021-17:45:31] [W] [TRT] Internal DLA error for layer ConvTranspose_150. Switching to GPU fallback.
[04/22/2021-17:45:31] [I] [TRT]
[04/22/2021-17:45:31] [I] [TRT] --------------- Layers running on DLA:
[04/22/2021-17:45:31] [I] [TRT] {Conv_0,Relu_1,Conv_2,Conv_3,Relu_4,Conv_5,Relu_6,Conv_7,Add_8,Relu_9,Conv_10,Relu_11,Conv_12,Relu_13,Conv_14,Add_15,Relu_16,Conv_17,Conv_18,Relu_19,Conv_20,Relu_21,Conv_22,Add_23,Relu_24,Conv_25,Relu_26,Conv_27,Relu_28,Conv_29,Add_30,Relu_31,Conv_32,Relu_33,Conv_34,Relu_35,Conv_36,Add_37,Relu_38,Conv_39,Relu_40,Conv_41,Relu_42,Conv_43,Add_44,Relu_45,Conv_46,Conv_47,Relu_48,Conv_49,Relu_50,Conv_51,Add_52,Relu_53,Conv_54,Relu_55,Conv_56,Relu_57,Conv_58,Add_59,Relu_60,Conv_61,Relu_62,Conv_63,Relu_64,Conv_65,Add_66,Relu_67,Conv_68,Relu_69,Conv_70,Relu_71,Conv_72,Add_73,Relu_74,Conv_75,Relu_76,Conv_77,Relu_78,Conv_79,Add_80,Relu_81,Conv_82,Relu_83,Conv_84,Relu_85,Conv_86,Add_87,Relu_88,Conv_89,Relu_90,Conv_91,Relu_92,Conv_93,Add_94,Relu_95,Conv_96,Relu_97,Conv_98,Relu_99,Conv_100,Add_101,Relu_102,Conv_103,Relu_104,Conv_105,Relu_106,Conv_107,Add_108,Relu_109,Conv_110,Relu_111,Conv_112,Relu_113,Conv_114,Add_115,Relu_116,Conv_117,Conv_118,Relu_119}, {Relu_121,Conv_122,Add_123,Relu_124,Conv_125,Relu_126}, {Relu_128,Conv_129,Add_130,Relu_131,Conv_132,Conv_133,Conv_134,Conv_135,Conv_136,Relu_137}, {BatchNormalization_139,Relu_140,Add_141,Conv_142,Relu_143}, {BatchNormalization_145,Relu_146,Add_147,Conv_148,Relu_149}, {BatchNormalization_151,Relu_152,Add_153,Conv_154,Conv_155,Relu_156,Conv_157,Conv_158,Relu_159,Conv_160,Sigmoid_161,MaxPool_162,Conv_163,Conv_164,Conv_165,Sigmoid_166,MaxPool_167,Conv_168,Conv_169},
[04/22/2021-17:45:31] [I] [TRT] --------------- Layers running on GPU:
[04/22/2021-17:45:31] [I] [TRT] Conv_120, Conv_127, ConvTranspose_138, ConvTranspose_144, ConvTranspose_150,
[04/22/2021-17:45:43] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
[04/22/2021-17:45:49] [I] [TRT] Detected 1 inputs and 9 output network tensors.
[04/22/2021-17:45:55] [I] Starting inference threads
NVMEDIA_DLA : 1928, ERROR: Submit failed.
[04/22/2021-17:45:55] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:55] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:55] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:55] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:55] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:55] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:55] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:55] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:55] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:55] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:55] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:55] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:55] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:55] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:55] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:55] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:55] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:55] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:55] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:55] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:55] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:55] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:55] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:55] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:55] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:55] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:55] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:55] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:55] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:55] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:55] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:55] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:55] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:55] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:56] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:56] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:57] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:57] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
NVMEDIA_DLA : 885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.
[04/22/2021-17:45:58] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[04/22/2021-17:45:58] [E] [TRT] FAILED_EXECUTION: std::exception
[04/22/2021-17:45:58] [I] Warmup completed 7 queries over 200 ms
[04/22/2021-17:45:58] [I] Timing trace has 115 queries over 3.04403 s
[04/22/2021-17:45:58] [I] Trace averages of 10 runs:
[04/22/2021-17:45:58] [I] Average on 10 runs - GPU latency: 26.2923 ms - Host latency: 26.4721 ms (end to end 26.4801 ms, enqueue 3.62892 ms)
[04/22/2021-17:45:58] [I] Average on 10 runs - GPU latency: 26.2915 ms - Host latency: 26.4755 ms (end to end 26.4828 ms, enqueue 3.48925 ms)
[04/22/2021-17:45:58] [I] Average on 10 runs - GPU latency: 26.2969 ms - Host latency: 26.4778 ms (end to end 26.4861 ms, enqueue 3.47683 ms)
[04/22/2021-17:45:58] [I] Average on 10 runs - GPU latency: 26.2912 ms - Host latency: 26.4739 ms (end to end 26.4827 ms, enqueue 3.19267 ms)
[04/22/2021-17:45:58] [I] Average on 10 runs - GPU latency: 26.2953 ms - Host latency: 26.4777 ms (end to end 26.486 ms, enqueue 3.43838 ms)
[04/22/2021-17:45:58] [I] Average on 10 runs - GPU latency: 26.2878 ms - Host latency: 26.4702 ms (end to end 26.4778 ms, enqueue 3.47576 ms)
[04/22/2021-17:45:58] [I] Average on 10 runs - GPU latency: 26.2661 ms - Host latency: 26.4487 ms (end to end 26.4564 ms, enqueue 3.32141 ms)
[04/22/2021-17:45:58] [I] Average on 10 runs - GPU latency: 26.266 ms - Host latency: 26.4484 ms (end to end 26.4561 ms, enqueue 3.35016 ms)
[04/22/2021-17:45:58] [I] Average on 10 runs - GPU latency: 26.2612 ms - Host latency: 26.4438 ms (end to end 26.4513 ms, enqueue 3.38496 ms)
[04/22/2021-17:45:58] [I] Average on 10 runs - GPU latency: 26.2674 ms - Host latency: 26.45 ms (end to end 26.457 ms, enqueue 3.39702 ms)
[04/22/2021-17:45:58] [I] Average on 10 runs - GPU latency: 26.2727 ms - Host latency: 26.4549 ms (end to end 26.4618 ms, enqueue 3.33569 ms)
[04/22/2021-17:45:58] [I] Host Latency
[04/22/2021-17:45:58] [I] min: 26.3635 ms (end to end 26.3716 ms)
[04/22/2021-17:45:58] [I] max: 26.5569 ms (end to end 26.5606 ms)
[04/22/2021-17:45:58] [I] mean: 26.4621 ms (end to end 26.4698 ms)
[04/22/2021-17:45:58] [I] median: 26.4604 ms (end to end 26.4679 ms)
[04/22/2021-17:45:58] [I] percentile: 26.5449 ms at 99% (end to end 26.5551 ms at 99%)
[04/22/2021-17:45:58] [I] throughput: 37.7788 qps
[04/22/2021-17:45:58] [I] walltime: 3.04403 s
[04/22/2021-17:45:58] [I] Enqueue Time
[04/22/2021-17:45:58] [I] min: 2.87952 ms
[04/22/2021-17:45:58] [I] max: 4.52925 ms
[04/22/2021-17:45:58] [I] median: 3.33911 ms
[04/22/2021-17:45:58] [I] GPU Compute
[04/22/2021-17:45:58] [I] min: 26.1958 ms
[04/22/2021-17:45:58] [I] max: 26.3768 ms
[04/22/2021-17:45:58] [I] mean: 26.2799 ms
[04/22/2021-17:45:58] [I] median: 26.2778 ms
[04/22/2021-17:45:58] [I] percentile: 26.3618 ms at 99%
[04/22/2021-17:45:58] [I] total compute time: 3.02219 s
&&&& PASSED TensorRT.trtexec # /usr/src/tensorrt/bin/trtexec --onnx=./rmdet.onnx --saveEngine=./out.trt --useDLACore=1 --allowGPUFallback
terminate called after throwing an instance of 'nvinfer1::AssertionFailure'
what(): std::exception
Aborted (core dumped)