Tao-converter failed to convert etlt to engine file due to could not find any implementation for node conv1/convolution + activate_1/Relu6

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Please provide the following information when requesting support.

• Hardware (T4/V100/Xavier/Nano/etc)
Jetson Nano 2G, Jetpack 4.6.
• Network Type (Detectnet_v2/Faster_rcnn/Yolo_v4/LPRnet/Mask_rcnn/Classification/etc)
Detectnet_v2
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here)
• Training spec file(If have, please share here)
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)

I’ve retrained a object detection model based on detectnet_v2 and integrated it into my deepstream python app by using .etlt model in PGIE config file, all working good in my Jetson Nano 2g with jetpack 4.6.
For speed up the startup time of the app, I want to convert the .etlt to engine file, what I’ve done on Jetson Nano 2g:

but always got an error:

[INFO] [MemUsageChange] Init CUDA: CPU +203, GPU +0, now: CPU 221, GPU 1940 (MiB)
[INFO] [MemUsageSnapshot] Builder begin: CPU 237 MiB, GPU 1937 MiB
[INFO] ---------- Layers Running on DLA ----------
[INFO] ---------- Layers Running on GPU ----------
[INFO] [GpuLayer] conv1/convolution + activation_1/Relu6
[INFO] [GpuLayer] block_1a_conv_1/convolution + block_1a_relu_1/Relu6
[INFO] [GpuLayer] block_1a_conv_shortcut/convolution
[INFO] [GpuLayer] block_1a_conv_2/convolution + add_1/add + block_1a_relu/Relu6
[INFO] [GpuLayer] block_1b_conv_1/convolution + block_1b_relu_1/Relu6
[INFO] [GpuLayer] block_1b_conv_2/convolution + add_2/add + block_1b_relu/Relu6
[INFO] [GpuLayer] block_2a_conv_1/convolution + block_2a_relu_1/Relu6
[INFO] [GpuLayer] block_2a_conv_shortcut/convolution
[INFO] [GpuLayer] block_2a_conv_2/convolution + add_3/add + block_2a_relu/Relu6
[INFO] [GpuLayer] block_2b_conv_1/convolution + block_2b_relu_1/Relu6
[INFO] [GpuLayer] block_2b_conv_2/convolution + add_4/add + block_2b_relu/Relu6
[INFO] [GpuLayer] block_3a_conv_1/convolution + block_3a_relu_1/Relu6
[INFO] [GpuLayer] block_3a_conv_shortcut/convolution
[INFO] [GpuLayer] block_3a_conv_2/convolution + add_5/add + block_3a_relu/Relu6
[INFO] [GpuLayer] block_3b_conv_1/convolution + block_3b_relu_1/Relu6
[INFO] [GpuLayer] block_3b_conv_2/convolution + add_6/add + block_3b_relu/Relu6
[INFO] [GpuLayer] block_4a_conv_1/convolution + block_4a_relu_1/Relu6
[INFO] [GpuLayer] block_4a_conv_shortcut/convolution
[INFO] [GpuLayer] block_4a_conv_2/convolution + add_7/add + block_4a_relu/Relu6
[INFO] [GpuLayer] block_4b_conv_1/convolution + block_4b_relu_1/Relu6
[INFO] [GpuLayer] block_4b_conv_2/convolution + add_8/add + block_4b_relu/Relu6
[INFO] [GpuLayer] output_bbox/convolution
[INFO] [GpuLayer] output_cov/convolution
[INFO] [GpuLayer] PWN(output_cov/Sigmoid)
[INFO] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +158, GPU +4, now: CPU 403, GPU 1943 (MiB)
[INFO] [MemUsageChange] Init cuDNN: CPU +240, GPU -2, now: CPU 643, GPU 1941 (MiB)
[WARNING] Detected invalid timing cache, setup a local cache instead
[ERROR] Tactic Device request: 450MB Available: 208MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 0 due to oom error on requested size of 450 detected for tactic 1002.
[ERROR] Tactic Device request: 450MB Available: 207MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 1 due to oom error on requested size of 450 detected for tactic 0.
[WARNING] Internal error: cannot reformat, disabling format. Try decreasing the workspace size with IBuilderConfig::setMaxWorkspaceSize().
[WARNING] Memory requirements of format conversion cannot be satisfied during timing, format rejected.
[ERROR] Tactic Device request: 1275MB Available: 199MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 0 due to oom error on requested size of 1275 detected for tactic 0.
[ERROR] Tactic Device request: 1276MB Available: 198MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 1 due to oom error on requested size of 1276 detected for tactic 1.
[ERROR] Tactic Device request: 1283MB Available: 199MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 3 due to oom error on requested size of 1283 detected for tactic 5.
[INFO] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
[ERROR] Tactic Device request: 1276MB Available: 199MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 0 due to oom error on requested size of 1276 detected for tactic 1062367460111450758.
[ERROR] Tactic Device request: 1276MB Available: 199MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 1 due to oom error on requested size of 1276 detected for tactic 1754984623894446479.
[ERROR] Tactic Device request: 1276MB Available: 199MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 2 due to oom error on requested size of 1276 detected for tactic 3611739942397549984.
[ERROR] Tactic Device request: 1276MB Available: 199MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 3 due to oom error on requested size of 1276 detected for tactic 4337000649858996379.
[ERROR] Tactic Device request: 1276MB Available: 199MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 4 due to oom error on requested size of 1276 detected for tactic 4501471010995462441.
[ERROR] Tactic Device request: 1276MB Available: 199MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 5 due to oom error on requested size of 1276 detected for tactic 6645123197870846056.
[ERROR] Tactic Device request: 1276MB Available: 199MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 6 due to oom error on requested size of 1276 detected for tactic -9137461792520977713.
[ERROR] Tactic Device request: 1276MB Available: 199MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 7 due to oom error on requested size of 1276 detected for tactic -8262349710178828730.
[ERROR] Tactic Device request: 1276MB Available: 199MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 8 due to oom error on requested size of 1276 detected for tactic -8133971918129952780.
[ERROR] Tactic Device request: 1276MB Available: 199MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 9 due to oom error on requested size of 1276 detected for tactic -6092040395344634144.
[ERROR] Tactic Device request: 1276MB Available: 199MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 10 due to oom error on requested size of 1276 detected for tactic -4787320710726427159.
[ERROR] Tactic Device request: 1276MB Available: 198MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 11 due to oom error on requested size of 1276 detected for tactic -1218658103698133241.
[INFO] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 648, GPU 1930 (MiB)
[ERROR] 10: [optimizer.cpp::computeCosts::1855] Error Code 10: Internal Error (Could not find any implementation for node conv1/convolution + activation_1/Relu6.)
[ERROR] Unable to create engine
Segmentation fault (core dumped)

questions:

  1. how to fix the convertion error?
  2. can i run the tao-converter on JetsonNano 4g or Jetson NX 8g all with Jetpack 4.6, and then using the converted engine file on JetsonNano 2g?
3

Could you add below in the command line?
-w 1000000000

4

command on JetsonNano 2g:

./tao-converter resnet18_detector.etlt -k tlt_encode -d 3,1280,960 -o output_cov/Sigmoid,output_bbox/BiasAdd -e resnet18_detector.engine -w 1000000000

got same error:


...
...
[ERROR] Tactic Device request: 1276MB Available: 175MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 0 due to oom error on requested size of 1276 detected for tactic 1062367460111450758.
[ERROR] Tactic Device request: 1276MB Available: 175MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 1 due to oom error on requested size of 1276 detected for tactic 1754984623894446479.
[ERROR] Tactic Device request: 1276MB Available: 175MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 2 due to oom error on requested size of 1276 detected for tactic 3611739942397549984.
[ERROR] Tactic Device request: 1276MB Available: 175MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 3 due to oom error on requested size of 1276 detected for tactic 4337000649858996379.
[ERROR] Tactic Device request: 1276MB Available: 175MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 4 due to oom error on requested size of 1276 detected for tactic 4501471010995462441.
[ERROR] Tactic Device request: 1276MB Available: 175MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 5 due to oom error on requested size of 1276 detected for tactic 6645123197870846056.
[ERROR] Tactic Device request: 1276MB Available: 175MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 6 due to oom error on requested size of 1276 detected for tactic -9137461792520977713.
[ERROR] Tactic Device request: 1276MB Available: 175MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 7 due to oom error on requested size of 1276 detected for tactic -8262349710178828730.
[ERROR] Tactic Device request: 1276MB Available: 175MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 8 due to oom error on requested size of 1276 detected for tactic -8133971918129952780.
[ERROR] Tactic Device request: 1276MB Available: 175MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 9 due to oom error on requested size of 1276 detected for tactic -6092040395344634144.
[ERROR] Tactic Device request: 1276MB Available: 175MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 10 due to oom error on requested size of 1276 detected for tactic -4787320710726427159.
[ERROR] Tactic Device request: 1276MB Available: 175MB. Device memory is insufficient to use tactic.
[WARNING] Skipping tactic 11 due to oom error on requested size of 1276 detected for tactic -1218658103698133241.
[INFO] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 648, GPU 1914 (MiB)
[ERROR] 10: [optimizer.cpp::computeCosts::1855] Error Code 10: Internal Error (Could not find any implementation for node conv1/convolution + activation_1/Relu6.)
[ERROR] Unable to create engine
Segmentation fault (core dumped)
5

The issue is due to OOM.

You can try to run on JetsonNano 4g.
Make sure the Tensorrt version is the same as JetsonNano 2g.

6

different Jetson devices (Nano, NX) but using same version of jetpack 4.6, they suppose to having same version of Tensorrt, is that correct? so they can share the same engine file.

7

It should be working. Please have a try.

8

Tested the .engine file can be generated on JetsonNano 4g (though a memory insufficient warning still show), the 2g always aborted with an OOM exception.
But also noticed, my deepstream python app using directly the .engine file had worse performance vs using .etlt file.
I’m using a retrained detectnet_v2 model.

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.