TLT Converter error on Jetson Nano

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

• Hardware: Jetson Nano
• Network Type: Faster RCNN
• TLT Version 3.0

Hi, I’m trying to deploy a Faster RCNN model (trained in a x86 machine) on DeepStream. But when I execute the tlt-converter command on the Jetson Nano, it responds with Cuda Error.

Command: ./tlt-converter -k NnRkcW8zdWk4ZG1hMWpvdW44OGpuODA1azc6ZDZiNDVkMjQtYzJlYS00OTgzLWEzNTUtNmIxMWE4NDc3Nzk5 -d 3,480,640 -o NMS frcnn_kitti_mobilenet_v1_train_fp32.etlt

Response log: [INFO] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [ERROR] ../builder/cudnnBuilderUtils.cpp (427) - Cuda Error in findFastestTactic: 702 (the launch timed out and was terminated) [ERROR] ../rtSafe/safeRuntime.cpp (32) - Cuda Error in free: 702 (the launch timed out and was terminated) terminate called after throwing an instance of 'nvinfer1::CudaError' what(): std::exception Aborted (core dumped)

I tried increasing the workspace using the -w flag, but the response is the same.
Can you help me please?

3

May I know more info about the environment?
What is the CUDA/Cudnn/TensorRT version? Did you install them via Jetpack?

4
  • CUDA ver: 10.2
  • TensorRT ver: 7.1.3
    I installed them via Jetpack.
    How do I check Cudnn version on Jetson?
5

You can run $dpkg -l |grep cuda

6

My Cudnn ver is 8.0.0

7

Please download official faster_rcnn model mentioned in GitHub - NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream
$ wget https://nvidia.box.com/shared/static/i1cer4s3ox4v8svbfkuj5js8yqm3yazo.zip -O models.zip

Retry with below command.
Command:

./tlt-converter -k nvidia_tlt -d 3,544,960 -o NMS xxx.etlt

8

I got the same response as before

[INFO] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [ERROR] ../builder/cudnnBuilderUtils.cpp (427) - Cuda Error in findFastestTactic: 702 (the launch timed out and was terminated) [ERROR] ../rtSafe/safeRuntime.cpp (32) - Cuda Error in free: 702 (the launch timed out and was terminated) terminate called after throwing an instance of 'nvinfer1::CudaError' what(): std::exception Aborted (core dumped)
9

Please reboot and retry.

10

Same error :/

11

Please try other models.
See Tlt-converter failed on Jetson Nano with Cuda Error in loadKernel: 702

It has similar error.
But actually I cannot reproduce. And the error results from end user.

12

Trying with Yolo v3 and Yolo v4 models responds with key error, but using SSD and RetinaNet generates engine successfully.

13

What is the “key error”?

14 
[ERROR] UffParser: Unsupported number of graph 0
[ERROR] Failed to parse the model, please check the encoding key to make sure it's correct
[ERROR] Network must have at least one output
[ERROR] Network validation failed.
[ERROR] Unable to create engine
Segmentation fault (core dumped)
15

What is the command when you run Yolo v3 and Yolo v4 ?

16

./tlt-converter -k nvidia_tlt -d 3,544,960 -o BatchedNMS yolov3_resnet18.etlt
./tlt-converter -k nvidia_tlt -d 3,544,960 -o BatchedNMS yolov4_resnet18.etlt

17

Please run with below.
$ ./tlt-converter -k nvidia_tlt -d 3,544,960 -p Input,1x3x544x960,1x3x544x960,2x3x544x960 yolov3_resnet18.etlt

18

I got the following error:

./tlt-converter: invalid option -- '-'
Unrecognized argument
Aborted (core dumped)
19

How did you download tlt-converter?

20

From Deploying to DeepStream / Generating an Engine Using tlt-converter / Instructions for Jetson , on the Faster RCNN TLT documentation.
Here is the link:
Faster RCNN TLT

21

Can you share the result of “$ ./tlt-converter -h” ?