Error when use DLA with yoloV4 tensorRT

[longvuvan083]

Hello, I got the error when convert yolov4 onnx model to tensorRT engine that support DLA. I used this command with TRTexec:
./trtexec --onnx=yolov4.onnx --fp16 --useDLACore=0 --allowGPUFallback --saveEngine=yolov41.engine

• But i got the errors when convert the model :

...........................................................................
[04/13/2023-02:31:29] [I] [TRT] [GpuLayer] 139_convolutional_reshape_2 copy
[04/13/2023-02:31:29] [I] [TRT] [GpuLayer] 150_convolutional_reshape_2 copy
[04/13/2023-02:31:29] [I] [TRT] [GpuLayer] 161_convolutional_reshape_2 copy
[04/13/2023-02:31:29] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +226, GPU +228, now: CPU 1092, GPU 11836 (MiB)
[04/13/2023-02:31:30] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +307, GPU +309, now: CPU 1399, GPU 12145 (MiB)
[04/13/2023-02:31:30] [W] [TRT] Detected invalid timing cache, setup a local cache instead
[04/13/2023-02:31:33] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
Module_id 33 Severity 2 : NVMEDIA_DLA 684
Module_id 33 Severity 2 : Failed to bind input tensor. err : 0x00000b
Module_id 33 Severity 2 : NVMEDIA_DLA 2866
Module_id 33 Severity 2 : Failed to bind input tensor args. status:  0x000007
[04/13/2023-02:36:45] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1863, GPU 12642 (MiB)
[04/13/2023-02:36:45] [E] Error[1]: [nvdlaUtils.cpp::submit::198] Error Code 1: DLA (Failure to submit program to DLA engine.)
[04/13/2023-02:36:45] [E] Error[2]: [builder.cpp::buildSerializedNetwork::417] Error Code 2: Internal Error (Assertion enginePtr != nullptr failed.)
Segmentation fault (core dumped)

Here is my AGX information:

How can i fix that?

– Update 1:
I try to run similar command in the new version of AGX is jetson Orin(jetpack 5.0.2) then the DLA model is converted successfully. but i have no lucky in the jetson xavier AGX version.

[SivaRamaKrishnaNV]

Dear @longvuvan083,
Could you flash latest jetpack release on Jetson AGX Xavier target and check the issue?

[alex247]

Quite a few customers are running their systems on jetpack 4.6 so we would prefer to resolve this without reflashing if possible

[SivaRamaKrishnaNV]

Dear @alex247,
Jetpack 4.6 seems to have a bug and got fixed. Please upgrade to Jetpack 4.6.3(Preferably latest release) and test.

[alex247]

If we build the model on 4.6.3 and it works, can we move the TRt model to a 4.6 machine do you think ?

[SivaRamaKrishnaNV]

Dear @alex247,

we build the model on 4.6.3 and it works, can we move the TRt model to a 4.6 machine?

The issue seem to be with a DLA lib. I doubt if it works. You may give a try.

[ramc]

Please refer to the DLA FAQ page to learn more about common DLA related questions: Deep-Learning-Accelerator-SW/FAQ