Low confidence scores of cat and dog in jetson-inference classification

jetsonnvidia · December 3, 2020, 11:39am

I am attempting to follow this to do transfer learning on cat and dog samples:

dusty-nv/jetson-inference/blob/master/docs/pytorch-cat-dog.md

<img src="https://github.com/dusty-nv/jetson-inference/raw/master/docs/images/deep-vision-header.jpg" width="100%">
<p align="right"><sup><a href="pytorch-transfer-learning.md">Back</a> | <a href="pytorch-plants.md">Next</a> | </sup><a href="../README.md#hello-ai-world"><sup>Contents</sup></a>
<br/>
<sup>Transfer Learning - Classification</sup></s></p>

# Re-training on the Cat/Dog Dataset

The first model that we'll be re-training is a simple model that recognizes two classes:  cat or dog.

<img src="https://github.com/dusty-nv/jetson-inference/raw/python/docs/images/pytorch-cat-dog.jpg" width="700">

Provided below is an 800MB dataset that includes 5000 training images, 1000 validation images, and 200 test images, each evenly split between the cat and dog classes.  The set of training images is used for transfer learning, while the validation set is used to evaluate classification accuracy during training, and the test images are to be used by us after training completes.  The network is never directly trained on the validation and test sets, only the training set.

The images from the dataset are made up of many different breeds of dogs and cats, including large felines like tigers and mountain lions since the amount of cat images available was a bit lower than dogs.  Some of the images also picture humans, which the detector is essentially trained to ignore as background and focus on the cat vs. dog content.

To get started, first make sure that you have [PyTorch installed](pytorch-transfer-learning.md#installing-pytorch) on your Jetson, then download the dataset below and kick off the training script.  After that, we'll test the re-trained model in TensorRT on some static images and a live camera feed. 

## Downloading the Data

During this tutorial, we'll store the datasets on the host device under `jetson-inference/python/training/classification/data`, which is one of the directories that is automatically [mounted into the container](aux-docker.md#mounted-data-volumes).  This way the dataset won't be lost when you shutdown the container.

This file has been truncated. show original

In the section “Processing Images with TensorRT” I am finding that I get much lower confidence levels on the single test images of a puma (?) and a dog on the beach. I get 50% confidence for the puma and 70% for the dog, yet I am running exactly the commands as specified in the tutorial.

cat dog

Additionally, in the section “Processing all the Test Images” I get many images of cats classified as dogs to quite high confidence levels (and vice versa for dogs). About 25% of images are misclassified. :-(

What am I doing wrong?

I am concerned because if I am doing something wrong now, I will be doing something wrong later when I use my own training data. Thanks.

dusty_nv · December 4, 2020, 10:44am

Hi @jetsonnvidia, do you know what accuracy your model trained to in PyTorch? How many epochs did you train it for?

You can try this cat/dog model that I previously trained for 100 epochs, and see if you get a different result:

https://nvidia.box.com/s/zlvb4y43djygotpjn6azjhwu0r3j0yxc

You will want to extract my model into a different folder than your own model resides in.

jetsonnvidia · December 3, 2020, 10:40pm

Hi, mine is running for the default of 35 epochs. I will try a larger number tomorrow. Thanks.

jetsonnvidia · December 4, 2020, 10:44am

I ran for 100 epochs overnight.

For the first 2 images, I got much better scores. :-)

Unfortunately, the algorithm really struggles with black-faced cats. Such is life.