Train custom object detectio model

rsamvelyan · May 4, 2020, 7:46am

Hello
I’m using Jetson Nano.
I need to train custom object detectors for a real time webcam application.
What API/Technology… is best to use in order to run the training on my Jetson Nano?
Please suggest me something.
Thanks in advance!
Ryan

AastaLLL · May 7, 2020, 5:42am

Hi,

Please noticed that Nano is an edge device so we don’t recommend to apply training on it.
You will get a much better training performance on a desktop GPU.

A possible case for Nano is transfer learning.
Please check this document for information:

github.com

dusty-nv/jetson-inference/blob/master/docs/pytorch-transfer-learning.md

<img src="https://github.com/dusty-nv/jetson-inference/raw/master/docs/images/deep-vision-header.jpg" width="100%">
<p align="right"><sup><a href="depthnet.md">Back</a> | <a href="pytorch-cat-dog.md">Next</a> | </sup><a href="../README.md#hello-ai-world"><sup>Contents</sup></a>
<br/>
<sup>Transfer Learning</sup></s></p>

# Transfer Learning with PyTorch

Transfer learning is a technique for re-training a DNN model on a new dataset, which takes less time than training a network from scratch.  With transfer learning, the weights of a pre-trained model are fine-tuned to classify a customized dataset.  In these examples, we'll be using the <a href="https://arxiv.org/abs/1512.03385">ResNet-18</a> and [SSD-Mobilenet](pytorch-ssd.md) networks, although you can experiment with other networks too.

<p align="center"><a href="https://arxiv.org/abs/1512.03385"><img src="https://github.com/dusty-nv/jetson-inference/raw/python/docs/images/pytorch-resnet-18.png" width="600"></a></p>

Although training is typically performed on a PC, server, or cloud instance with discrete GPU(s) due to the often large datasets used and the associated computational demands, by using transfer learning we're able to re-train various networks onboard Jetson to get started with training and deploying our own DNN models.  

<a href=https://pytorch.org/>PyTorch</a> is the machine learning framework that we'll be using, and example datasets along with training scripts are provided to use below, in addition to a camera-based tool for collecting and labeling your own training datasets.  

## Installing PyTorch

If you are [Running the Docker Container](aux-docker.md) or optionally chose to install PyTorch back when you [Built the Project](building-repo-2.md#installing-pytorch), it should already be installed on your Jetson to use.  Otherwise, if you aren't using the container and want to proceed with transfer learning, you can install it now:

``` bash

This file has been truncated. show original

Thanks.

rsamvelyan · May 8, 2020, 4:28pm

Thanks for the reply.
I did this tutorial, which is for Image Classification. The tutorial is great and the Inference library does a great job.
I didn’t find a tutorial on object detection though. Do you know where I can find one?
Thanks again

dusty_nv · May 8, 2020, 5:05pm

Hi @rsamvelyan, see this post: DIGITS or somthing else - #7 by dusty_nv

For PyTorch training of object detection, I’ve been preparing the tutorial. You can find the code here: GitHub - dusty-nv/pytorch-ssd: MobileNetV1, MobileNetV2, VGG based SSD/SSD-lite implementation in PyTorch. Out-of-box support for retraining on Open Images dataset. ONNX and Caffe2 support. Experiment Ideas like CoordConv.

rsamvelyan · May 8, 2020, 8:19pm

thanks a lot! I’m going to do this tutorial

rsamvelyan · May 9, 2020, 3:36pm

And is there an example on how to load the custom trained model into detectnet-camera.py which is in the examplea folder of Jetson-Inference?

I tried to load different pre-trained models, by using --model and and specifying the labels file but it always gives me an error.
Do I need to specify something under --network too?

rsamvelyan · May 9, 2020, 8:44pm

Dusty, first of all thanks a lot for all your contribution!

The tutorial for PyTorch you built requires Caffe. I tried compiling it from source on my Jetson Nano, it troughs c++ errors… total nightmare

Can I install DIGITS on a computer without an NVIDIA GPU? Or can I install it on my Jetson Nano?

What are my options for transfer learning? I’m totally lost.

Can I use the train.py in the jetson-inference/python/training/detection for transfer learning on my custom data-set?
Any specific and complete tutorial on that one, just like the cat_dog example?

At this point, all I can get to work is inference with detectnet on the pre-trained models that came with the installation… :(

Thanks again!!

AastaLLL · May 13, 2020, 7:35am

Hi,

1.
You can find the instruction to load custom model below:
Although this is for imagenet with C++, the procedure for detectnet-camera.py should be similar.

github.com

dusty-nv/jetson-inference/blob/master/docs/imagenet-custom.md#loading-custom-models-on-jetson

<img src="https://github.com/dusty-nv/jetson-inference/raw/master/docs/images/deep-vision-header.jpg" width="100%">
<p align="right"><sup><a href="imagenet-snapshot.md">Back</a> | <a href="detectnet-training.md">Next</a> | </sup><a href="../README.md#two-days-to-a-demo-digits"><sup>Contents</sup></a>
<br/>
<sup>Image Recognition</sup></p> 

# Loading Custom Models on Jetson

The [`imagenet-console`](../examples/imagenet-console/imagenet-console.cpp) and [`imagenet-camera`](../examples/imagenet-camera/imagenet-camera.cpp) programs that we used before also accept extended command line parameters for loading a custom model snapshot.  Set the `$NET` variable below to the path to your extracted snapshot:

``` bash
$ NET=networks/GoogleNet-ILSVRC12-subset

$ ./imagenet-console bird_0.jpg output_0.jpg \
--prototxt=$NET/deploy.prototxt \
--model=$NET/snapshot_iter_184080.caffemodel \
--labels=$NET/labels.txt \
--input_blob=data \
--output_blob=softmax
```

This file has been truncated. show original

2.
DIGITs cannot install on the environment without desktop GPU.
Since it need NCCL support, which requires a desktop GPU.

To install pyTorch on Jetson, please follow this topic:

To install Caffe on Jetson, please follow this tutorial:

Thanks.

rsamvelyan · May 18, 2020, 10:18pm

Hello AastaLLL
Thanks for the reply
I have my custom ONNX model, which is a resnet-18 model, fine tuned with Pytorch with the sample python script provided with Jetson Inference. Then I converted it to ONNX
First of all, I just made a copy for myself, and renamed it with my-detectnet-camera.py
This is what I use to call the detectnet-camera.py:

python3 my-detectnet-camera.py --model=resnet18_smarties_224x224.onnx --class_labels=labels.txt --input_blob=data

The GPU engine file is successfully created: resnet18_smarties_224x224.onnx.1.1.GPU.FP16.engine

Gstreamer starts and my webcam begins to work. I’m showing around 80FPS, but absolutely no detection. Not one single object gets detected, no bounding boxes.

This is what I get in the terminal while the webcam is up and running:
[TRT] detectNet::Detect() – ONNX – coord (nan, nan) (nan, nan) image 640x480
[cuda] cudaDetectionOverlay((float4*)input, (float4*)output, width, height, detections, numDetections, (float4*)mClassColors[1])
[cuda] invalid configuration argument (error 9) (hex 0x09)
[cuda] /home/ryan/jetson-inference/c/detectNet.cpp:945
[TRT] detectNet::Detect() – failed to render overlay
detected 1 objects in image
jetson.inference – PyDetection_Dealloc()
<detectNet.Detection object>
– ClassID: 0
– Confidence: 1
– Left: nan
– Top: nan
– Right: nan
– Bottom: nan
– Width: nan
– Height: nan
– Area: nan
– Center: (nan, nan)

These are all the parameters I can specify:
optional arguments:
-h, --help show this help message and exit
–network NETWORK pre-trained model to load (see below for options)
–overlay OVERLAY detection overlay flags (e.g. --overlay=box,labels,conf)
valid combinations are: ‘box’, ‘labels’, ‘conf’, ‘none’
–threshold THRESHOLD
minimum detection threshold to use
–camera CAMERA index of the MIPI CSI camera to use (e.g. CSI camera 0)
or for VL42 cameras, the /dev/video device to use.
by default, MIPI CSI camera 0 will be used.
–width WIDTH desired width of camera stream (default is 1280 pixels)
–height HEIGHT desired height of camera stream (default is 720 pixels)

detectNet arguments:
–network NETWORK pre-trained model to load, one of the following:
* ssd-mobilenet-v1
* ssd-mobilenet-v2 (default)
* ssd-inception-v2
* pednet
* multiped
* facenet
* coco-airplane
* coco-bottle
* coco-chair
* coco-dog
–model MODEL path to custom model to load (caffemodel, uff, or onnx)
–prototxt PROTOTXT path to custom prototxt to load (for .caffemodel only)
–class_labels LABELS path to text file containing the labels for each class
–threshold THRESHOLD minimum threshold for detection (default is 0.5)
–input_blob INPUT name of the input layer (default is ‘data’)
–output_cvg COVERAGE name of the coverge output layer (default is ‘coverage’)
–output_bbox BOXES name of the bounding output layer (default is ‘bboxes’)
–mean_pixel PIXEL mean pixel value to subtract from input (default is 0.0)
–batch_size BATCH maximum batch size (default is 1)
–alpha ALPHA overlay alpha blending value, range 0-255 (default: 120)
–profile enable layer profiling in TensorRT

usage: my-detectnet-camera.py [-h] [–network NETWORK] [–overlay OVERLAY]
[–threshold THRESHOLD] [–camera CAMERA]
[–width WIDTH] [–height HEIGHT]

Locate objects in a live camera stream using an object detection DNN.

optional arguments:
-h, --help show this help message and exit
–network NETWORK pre-trained model to load (see below for options)
–overlay OVERLAY detection overlay flags (e.g. --overlay=box,labels,conf)
valid combinations are: ‘box’, ‘labels’, ‘conf’, ‘none’
–threshold THRESHOLD
minimum detection threshold to use
–camera CAMERA index of the MIPI CSI camera to use (e.g. CSI camera 0)
or for VL42 cameras, the /dev/video device to use.
by default, MIPI CSI camera 0 will be used.
–width WIDTH desired width of camera stream (default is 1280 pixels)
–height HEIGHT desired height of camera stream (default is 720 pixels)

detectNet arguments:
–network NETWORK pre-trained model to load, one of the following:
* ssd-mobilenet-v1
* ssd-mobilenet-v2 (default)
* ssd-inception-v2
* pednet
* multiped
* facenet
* coco-airplane
* coco-bottle
* coco-chair
* coco-dog
–model MODEL path to custom model to load (caffemodel, uff, or onnx)
–prototxt PROTOTXT path to custom prototxt to load (for .caffemodel only)
–class_labels LABELS path to text file containing the labels for each class
–threshold THRESHOLD minimum threshold for detection (default is 0.5)
–input_blob INPUT name of the input layer (default is ‘data’)
–output_cvg COVERAGE name of the coverge output layer (default is ‘coverage’)
–output_bbox BOXES name of the bounding output layer (default is ‘bboxes’)
–mean_pixel PIXEL mean pixel value to subtract from input (default is 0.0)
–batch_size BATCH maximum batch size (default is 1)
–alpha ALPHA overlay alpha blending value, range 0-255 (default: 120)
–profile enable layer profiling in TensorRT

I’ve even tried with caffe models taken from the network folder provided with Jetson Inference ans still no luck!

I can’t figure out what exactly is that I am doing wrong here?

Thanks!

dusty_nv · May 19, 2020, 4:17pm

Hi @rsamvelyan, resnet-18 is an image classification model, not object detection - so it wouldn’t work with detectNet. You should instead load resnet-18 with the imageNet programs.

If you want to train your own object detection model, I have been working on new part of Hello AI World tutorial for that. You can see the PyTorch training code awhile here: GitHub - dusty-nv/pytorch-ssd: MobileNetV1, MobileNetV2, VGG based SSD/SSD-lite implementation in PyTorch. Out-of-box support for retraining on Open Images dataset. ONNX and Caffe2 support. Experiment Ideas like CoordConv.

rsamvelyan · May 19, 2020, 5:08pm

You’re roght Dusty. Yesterday I realized that I used the wron model.
Nevertheless, for some reason, I haven’t beem able to properly load no other model. Even when loading the same mobilenetv2 I’m getting messages such as wrong class id or something like that.
I was wonderif there is a specific example on how to properly load an ONNX object detection model into detectnet-camera.py?

AastaLLL · June 10, 2020, 3:09am

Hi,

You can find an imageNet with ONNX model sample here:

github.com

dusty-nv/jetson-inference/blob/master/docs/pytorch-plants.md

<img src="https://github.com/dusty-nv/jetson-inference/raw/master/docs/images/deep-vision-header.jpg" width="100%">
<p align="right"><sup><a href="pytorch-cat-dog.md">Back</a> | <a href="pytorch-collect.md">Next</a> | </sup><a href="../README.md#hello-ai-world"><sup>Contents</sup></a>
<br/>
<sup>Transfer Learning - Classification</sup></s></p>

# Re-training on the PlantCLEF Dataset

Next, we'll train a model capable of classifying 20 different varieties of plants and trees from the <a href="https://www.imageclef.org/lifeclef/2017/plant">PlantCLEF</a> dataset.

<a href="https://www.imageclef.org/lifeclef/2017/plant"><img src="https://github.com/dusty-nv/jetson-inference/raw/python/docs/images/pytorch-plants.jpg"></a>

Provided below is a 1.5GB subset that includes 10,475 training images, 1,155 validation images, and 30 test images across 20 classes of plants and trees.  The classes were selected from PlantCLEF 2017 from categories that had at least 500 training images in the original dataset:

```
• ash
• beech
• cat-tail
• cedar
• clover
• cyprus

This file has been truncated. show original

The steps for detection model should be similar.
Thanks.