Errors inferencing model with DLA core


I have created a model (Mobilenet-v1-SSD) which uses the DLA core using trtexec. When generating this trt engine file, I do not get any errors. It also inferences perfectly using trtexec.

But this same model when run with my inferencer, runs perfectly with only the first frame from the video. This model fails on every subsequent frame thereafter. In fact, the inferencer I coded up is loosely based on the trtexec file provided in the TensorRT Github repository.


TensorRT Version: 7.1.0
GPU Type: Nvidia Xavier AGX
Nvidia Driver Version:
CUDA Version: 10.2
CUDNN Version:
Operating System + Version: Ubuntu 18.04
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

The console output with the errors:

2021-08-18 13:28:07.974] [info] [main.cxx:522] (15, 69), (239, 201), 1 (aeroplane), 0.88964844
[2021-08-18 13:28:07.974] [info] [timeExecution.hpp:96] [UTILS] [TimeExecution (TensorRT Inference) ]: 1490.553682ms (0.670892 Hz)
NVMEDIA_DLA : 1928, ERROR: Submit failed.
[08/18/2021-13:29:26] [08/18/2021-13:29:26] [E] [TRT] ../rtExt/dla/native/dlaUtils.cpp (194) - DLA Error in submit: 7 (Failure to submit program to DLA engine.)
[E] [TRT] FAILED_EXECUTION: std::exception
[2021-08-18 13:29:48.642] [info] [main.cxx:522] (15, 69), (239, 201), 1 (aeroplane), 0.88964844
[2021-08-18 13:29:48.642] [info] [timeExecution.hpp:96] [UTILS] [TimeExecution (TensorRT Inference) ]: 41131.105716ms (0.024312 Hz)
NVMEDIA_DLA :  885, ERROR: runtime registerEvent failed. err: 0x4.
NVMEDIA_DLA : 1849, ERROR: RequestSubmitEvents failed. status: 0x7.

In the above output snippet, I was able to get the first prediction which I confirmed it to be correct. Then every subsequent frame would not be submitted successfully.


Would you mind upgrading your environment into JetPack4.6 first?
JetPack 4.6 includes DLA v1.3.6 with lots of bug fixes and it may help on your use case.
(since we don’t upgrade DLA library for a while)