Deep Learning Inference Benchmarking Instructions

Hi all, below you will find the procedures to run the Jetson Nano deep learning inferencing benchmarks from this blog post with TensorRT:

First, make sure you Nano is in 10W performance mode (which is the default mode) and run jetson_clocks script:

$ sudo nvpmodel -m 0
$ sudo jetson_clocks

SSD-Mobilenet-V2

  1. Copy the ssd-mobilenet-v2 archive from here to the ~/Downloads folder on Nano.
    $ cd ~/Downloads/
    $ wget --no-check-certificate 'https://nvidia.box.com/shared/static/8oqvmd79llr6lq1fr43s4fu1ph37v8nt.gz' -O ssd-mobilenet-v2.tar.gz
    $ tar -xvf ssd-mobilenet-v2.tar.gz
    $ cd ssd-mobilenet-v2
    $ sudo cp -R sampleUffSSD_rect /usr/src/tensorrt/samples
    $ sudo cp sample_unpruned_mobilenet_v2.uff /usr/src/tensorrt/data/ssd/
    $ sudo cp image1.ppm /usr/src/tensorrt/data/ssd/
    
  2. Compile the sample
    $ cd /usr/src/tensorrt/samples/sampleUffSSD_rect
    $ sudo make
    
  3. Run the sample to measure inference performance
    $ cd /usr/src/tensorrt/bin
    $ sudo ./sample_uff_ssd_rect
    

Image Classification (ResNet-50, Inception V4, VGG-19)

  1. The resources needed to run these models are available here. Copy each of these .prototxt files to the /usr/src/tensorrt/data/googlenet folder on your Jetson Nano.
  2. ResNet-50
    $ cd /usr/src/tensorrt/bin
    $ ./trtexec --output=prob --deploy=../data/googlenet/ResNet50_224x224.prototxt --fp16 --batch=1
    
  3. Inception V4
    $ cd /usr/src/tensorrt/bin
    $ ./trtexec --output=prob --deploy=../data/googlenet/inception_v4.prototxt --fp16 --batch=1
    
  4. VGG-19
    $ cd /usr/src/tensorrt/bin
    $ ./trtexec --output=prob --deploy=../data/googlenet/VGG19_N2.prototxt --fp16 --batch=1
    

U-Net Segmentation

  1. Copy the output_graph.uff model file from here to the home folder on your Jetson Nano or any directory of your preference.
  2. Run the U-Net inference benchmark:
    $ cd /usr/src/tensorrt/bin
    $ sudo ./trtexec --uff=~/output_graph.uff --uffInput=input_1,1,512,512 --output=conv2d_19/Sigmoid --fp16
    

Pose Estimation

  1. Copy the pose_estimation.prototxt file from here to the /usr/src/tensorrt/data/googlenet folder of your Nano.
  2. Run the OpenPose inference benchmark:
    $ cd /usr/src/tensorrt/bin/
    $ sudo ./trtexec --output=Mconv7_stage2_L2 --deploy=../data/googlenet/pose_estimation.prototxt --fp16 --batch=1
    

Super Resolution

  1. Download the require files to run inference on the Super Resolution neural network.
    $ sudo wget --no-check-certificate 'https://nvidia.box.com/shared/static/a99l8ttk21p3tubjbyhfn4gh37o45rn8.gz' -O Super-Resolution-BSD500.tar.gz
    
  2. Unzip the downloaded file
    $ sudo tar -xvf Super-Resolution-BSD500.tar.gz
    
  3. Run the Super Resolution inferencing benchmark:
    $ cd /usr/src/tensorrt/bin
    $ sudo ./trtexec --output=output_0 --onnx=<path to the .onnx file in the unzipped folder above> --fp16 --batch=1
    

Tiny YOLO v3

  1. Install pre-requisite packages
    $ sudo apt-get install libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev libgflags-dev
    
  2. Download trt-yolo-app
    $ cd ~
    $ git clone https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps.git
    
  3. Install other requirements
    $ cd ~/deepstream_reference_apps/yolo
    $ sudo sh prebuild.sh
    
  4. Compile and install app
    $ cd apps/trt-yolo
    $ mkdir build && cd build
    $ cmake -D CMAKE_BUILD_TYPE=Release ..
    $ make && sudo make install
    $ cd ../../..
    
  5. For the sample image data set, you can download 500 images (need to be in .png) format to any folder on your Jetson Nano, just use 1 image file, or use a test set of 5 images that we've provided here.
    • Navigate your terminal to:
      $ cd ~/deepstream_reference_apps/yolo/data
      
    • Open the file “test_images.txt”
    • In the above file, you need to provide the full path to each of the 500 images you downloaded. For example, if your first image is located in the Downloads directory, the path you would enter in line 1 would be:
      /home/<username>/Downloads/<image file name>.png
      
    • Alternatively, you could provide the path to just one image and copy that line 500 times in that file.
    • A sample set of images (5 images of varying resolutions, repeated 100 times) along with the test_images.txt file have been uploaded here. You can use this data set if you don’t want to download your own images.
    • Go to the folder ‘config’ and open file ‘yolov3-tiny.txt'
    • In the file yolov3-tiny.txt, search for “--precision=kINT8” and replace “kINT8” with “kHALF” to change the inference precision to FP16.
    • Save the file
  6. Now run the Tiny YOLO inference:
    $ cd ~/deepstream_reference_apps/yolo
    $ sudo trt-yolo-app --flagfile=config/yolov3-tiny.txt