Use TAO converter with DLA enable for pruned TLT model. But the GPU still load when inference

longvuvan083 · April 20, 2023, 9:02pm

Hello,
Currently I follow some instructions to covert the pruned VehicleMakeNet tlt model to DLA engine with tao-converter. My expected is to use this converted engine model only in DLA core. But when i load the engine, the gpu usage still increase? Do I miss some thing?

I found an old topic talk about my problem Transfer Learning Toolkit DLA warning messages. When i convert TLT to DLA engine, it show some layer not support DLA but I thought it just warning and I can ignore that follow that topic?

The model taken from here “VehicleMakeNet | NVIDIA NGC”
The command I used to convert the model to DLA engine

$ ./tao-converter  resnet18_vehiclemakenet_pruned.etlt -k tlt_encode -d  3,224,224 -m 1 -o predictions/Softmax -t fp16 -e resnet18_vehiclemakenet_pruned.engine -u 0

The command I used to test inference:

 /usr/src/tensorrt/bin/trtexec --loadEngine=resnet18_vehiclemakenet_pruned.engine --fp16 --batch=1 --useSpinWait --avgRuns=100000000 --useDLACore=0

As the message Log I can see the [MemUsageChange] from 7547Mb to 8100MB. I also used "sudo tegrastats " to check mem usage

RAM 4339/31929MB GPU 928/1620 CPU 928/2139
RAM 4579/31929MB GPU 1080/1610 CPU 4318/2176
RAM 4982/31929MB GPU 1081/1602 CPU 2779/2186
RAM 5303/31929MB GPU 1234/1596 CPU 2469/2191 
RAM 5303/31929MB GPU 1387/1592 CPU 1850/2185 
RAM 5303/31929MB GPU 1387/1589 CPU 1849/2180 
RAM 5305/31929MB GPU 1387/1586 CPU 2003/2177 
RAM 4341/31929MB GPU 1081/1578 CPU 1391/2165 
RAM 4341/31929MB GPU 927/1568 CPU 1082/2148 
RAM 4340/31929MB GPU 928/1559 CPU 1236/2135 
RAM 4341/31929MB GPU 928/1549 CPU 928/2117 
RAM 4341/31929MB GPU 928/1540 CPU 928/2100

My system information:
image (5)

Pls help me

Morganh · April 21, 2023, 2:16am

Can you run without “–useDLACore=0” in order to run with gpu only, and then compare the result?

longvuvan083 · April 21, 2023, 7:44am

Hello, I ran the test command wihtout “–useDLACore=0” and here is the result:

 /usr/src/tensorrt/bin/trtexec --loadEngine=resnet18_vehiclemakenet_pruned.engine --fp16 --batch=1 --useSpinWait --avgRuns=100000000

Look like the speed result is not have many difference with and without “–useDLACore=0”

Morganh · April 21, 2023, 9:24am

Can you share the full log when you generate the engine?

longvuvan083 · April 21, 2023, 9:36am

Hello @Morganh here is the full log captured when generate the engine:
Command :

./tao-converter  resnet18_vehiclemakenet_pruned.etlt -k tlt_encode -d  3,224,224 -m 1 -o predictions/Softmax -t fp16 -e resnet18_vehiclemakenet_pruned.engine -u 0

[INFO] [MemUsageChange] Init CUDA: CPU +354, GPU +0, now: CPU 372, GPU 11414 (MiB)

[WARNING] Default DLA is enabled but layer conv1/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer bn_conv1/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer bn_conv1/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer bn_conv1/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer bn_conv1/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer bn_conv1/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer bn_conv1/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer bn_conv1/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer bn_conv1/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer bn_conv1/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_conv_1/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_1/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_1/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_1/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_1/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_1/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_1/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_1/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_1/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_1/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_conv_2/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_2/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_2/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_2/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_2/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_2/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_2/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_2/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_2/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_2/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_conv_shortcut/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_shortcut/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_shortcut/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_shortcut/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_shortcut/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_shortcut/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_shortcut/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_shortcut/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_shortcut/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1a_bn_shortcut/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_conv_1/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_1/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_1/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_1/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_1/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_1/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_1/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_1/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_1/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_1/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_conv_2/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_2/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_2/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_2/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_2/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_2/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_2/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_2/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_2/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_2/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_conv_shortcut/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_shortcut/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_shortcut/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_shortcut/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_shortcut/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_shortcut/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_shortcut/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_shortcut/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_shortcut/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_1b_bn_shortcut/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_conv_1/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_1/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_1/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_1/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_1/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_1/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_1/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_1/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_1/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_1/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_conv_2/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_2/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_2/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_2/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_2/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_2/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_2/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_2/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_2/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_2/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_conv_shortcut/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_shortcut/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_shortcut/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_shortcut/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_shortcut/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_shortcut/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_shortcut/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_shortcut/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_shortcut/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2a_bn_shortcut/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_conv_1/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_1/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_1/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_1/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_1/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_1/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_1/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_1/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_1/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_1/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_conv_2/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_2/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_2/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_2/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_2/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_2/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_2/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_2/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_2/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_2/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_conv_shortcut/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_shortcut/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_shortcut/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_shortcut/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_shortcut/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_shortcut/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_shortcut/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_shortcut/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_shortcut/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_2b_bn_shortcut/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_conv_1/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_1/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_1/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_1/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_1/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_1/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_1/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_1/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_1/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_1/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_conv_2/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_2/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_2/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_2/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_2/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_2/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_2/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_2/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_2/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_2/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_conv_shortcut/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_shortcut/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_shortcut/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_shortcut/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_shortcut/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_shortcut/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_shortcut/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_shortcut/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_shortcut/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3a_bn_shortcut/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_conv_1/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_1/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_1/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_1/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_1/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_1/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_1/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_1/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_1/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_1/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_conv_2/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_2/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_2/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_2/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_2/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_2/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_2/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_2/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_2/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_2/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_conv_shortcut/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_shortcut/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_shortcut/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_shortcut/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_shortcut/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_shortcut/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_shortcut/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_shortcut/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_shortcut/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_3b_bn_shortcut/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_conv_1/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_1/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_1/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_1/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_1/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_1/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_1/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_1/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_1/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_1/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_conv_2/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_2/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_2/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_2/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_2/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_2/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_2/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_2/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_2/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_2/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_conv_shortcut/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_shortcut/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_shortcut/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_shortcut/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_shortcut/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_shortcut/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_shortcut/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_shortcut/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_shortcut/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4a_bn_shortcut/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_conv_1/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_1/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_1/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_1/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_1/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_1/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_1/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_1/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_1/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_1/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_conv_2/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_2/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_2/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_2/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_2/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_2/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_2/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_2/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_2/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_2/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_conv_shortcut/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_shortcut/moving_variance is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_shortcut/Reshape_1/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_shortcut/batchnorm/add/y is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_shortcut/gamma is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_shortcut/Reshape_3/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_shortcut/beta is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_shortcut/Reshape_2/shape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_shortcut/moving_mean is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer block_4b_bn_shortcut/Reshape/shape is not supported on DLA, falling back to GPU.
[WARNING] flatten/Reshape: DLA only supports windows in the range of [1-8].
[WARNING] Default DLA is enabled but layer flatten/Reshape is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer predictions/kernel is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer predictions/bias is not supported on DLA, falling back to GPU.
[WARNING] Default DLA is enabled but layer predictions/Softmax is not supported on DLA, falling back to GPU.
[INFO] [MemUsageSnapshot] Builder begin: CPU 405 MiB, GPU 11469 MiB
[INFO] ---------- Layers Running on DLA ----------
[INFO] [DlaLayer] {ForeignNode[conv1/convolution...block_4b_relu/Relu]}
[INFO] [DlaLayer] {ForeignNode[predictions/MatMul...predictions/BiasAdd]}
[INFO] ---------- Layers Running on GPU ----------
[INFO] [GpuLayer] flatten/Reshape
[INFO] [GpuLayer] predictions/Softmax
[INFO] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +226, GPU +209, now: CPU 649, GPU 11699 (MiB)
[INFO] [MemUsageChange] Init cuDNN: CPU +307, GPU +312, now: CPU 956, GPU 12011 (MiB)
[WARNING] Detected invalid timing cache, setup a local cache instead
[INFO] Detected 1 inputs and 1 output network tensors.
[INFO] Total Host Persistent Memory: 1744
[INFO] Total Device Persistent Memory: 0
[INFO] Total Scratch Memory: 0
[INFO] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 32 MiB, GPU 2 MiB
[INFO] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 992, GPU 12099 (MiB)
[INFO] [MemUsageChange] Init cuDNN: CPU +0, GPU +0, now: CPU 992, GPU 12099 (MiB)
[INFO] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 992, GPU 12099 (MiB)
[INFO] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 991, GPU 12099 (MiB)
[INFO] [MemUsageSnapshot] Builder end: CPU 975 MiB, GPU 12099 MiB```

Morganh · April 24, 2023, 3:12am

Thanks for the info. That’s expected. Some layers are running on DLA, and some are running on GPU.

alex247 · April 24, 2023, 5:17am

So the nvidia vehicle make model cannot run fully on DLA unless we manually modify the layers ourself ?

We’re trying to offload processing from the gpu but so far models we’ve tried (mostly our custom ones) always to default layers back to gpu which seems to defeat the point.

ramc · April 26, 2023, 2:23pm

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

Also check out the DLA github page for samples and resources: Recipes and tools for running deep learning workloads on NVIDIA DLA cores for inference applications.

We have a FAQ page that addresses some common questions that we see developers run into: Deep-Learning-Accelerator-SW/FAQ

system · May 30, 2023, 7:22am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.