Graphing ImageNet confidence values for all classes

Robotics & Edge Computing Jetson & Embedded Systems Jetson Nano
1

I followed Dusty’s excellent imagenet training tutorial and have been having good success with a simple 3 class model.
I have been retraining with different networks / epochs and learning a lot.

When evaluating the effectiveness of each new model I find myself wanting to graph the confidence values for all three classes as they change with varying input. This data is passed through the terminal but I can’t figure out how to get all of the confidence values in my python script for graphing with matplotlib.

class_id, confidence = net.Classify(img)
Only returns the values for the highest confidence solution.

I can see the data I want in the terminal but don’t know how to retrieve it.
If I need a FOR loop I don’t know the correct variable to use

For example
for confidence in classifications:
print(confidence)

Any suggestions?

2

Hi @clay1 - currently there isn’t an API in the imageNet class for this - I will add it to my todo list.

What you could do to test it though, is add a quick little function to jetson-inference/c/imageNet.h (inside the imageNet class, around line 183 or so would work)

// add this inside public imageNet functions
inline float GetConfidence( uint32_t index ) const     { return mOutputs[0].CPU[index]; }

Then, you would need to write the Python binding for this in jetson-inference/python/bindings/PyImageNet.cpp

PyObject* PyImageNet_GetConfidence( PyImageNet_Object* self, PyObject* args )
{
	if( !self || !self->net )
	{
		PyErr_SetString(PyExc_Exception, LOG_PY_INFERENCE "imageNet invalid object instance");
		return NULL;
	}
	
	int classIdx = 0;

	if( !PyArg_ParseTuple(args, "i", &classIdx) )
	{
		PyErr_SetString(PyExc_Exception, LOG_PY_INFERENCE "imageNet failed to parse arguments");
		return NULL;
	}
		
	if( classIdx < 0 || classIdx >= self->net->GetNumClasses() )
	{
		PyErr_SetString(PyExc_Exception, LOG_PY_INFERENCE "imageNet requested class index is out of bounds");
		return NULL;
	}

	return PyFloat_FromDouble(self->net->GetConfidence(classIdx));
}

Finally, add the binding to pyImageNet_Methods[] list to register it with Python:

static PyMethodDef pyImageNet_Methods[] = 
{
    /* add this GetConfidence line below */
    { "GetConfidence", (PyCFunction)PyImageNet_GetConfidence, METH_VARARGS, "Get class confidence"},
	{ "Classify", (PyCFunction)PyImageNet_Classify, METH_VARARGS|METH_KEYWORDS, DOC_CLASSIFY},
	{ "GetNetworkName", (PyCFunction)PyImageNet_GetNetworkName, METH_NOARGS, DOC_GET_NETWORK_NAME},
    { "GetNumClasses", (PyCFunction)PyImageNet_GetNumClasses, METH_NOARGS, DOC_GET_NUM_CLASSES},
    { "GetClassDesc", (PyCFunction)PyImageNet_GetClassDesc, METH_VARARGS, DOC_GET_CLASS_DESC},
	{ "GetClassSynset", (PyCFunction)PyImageNet_GetClassSynset, METH_VARARGS, DOC_GET_CLASS_SYNSET},
	{ "Usage", (PyCFunction)PyImageNet_Usage, METH_NOARGS|METH_STATIC, DOC_USAGE_STRING},
	{NULL}  /* Sentinel */
};

Then you should be able to use it in a loop from Python, like net.GetConfidence(n)

Remember to re-run make and sudo make install after making the changes.

3

Confirmed! Dusty that worked perfectly!

It is really educational to watch a graph of each confidence percentage while trying different scenarios with a custom trained network.

The time you took out of your day to make that lengthy reply and get me over this hump is greatly appreciated. I would have never figured that out.

Thanks so much

4

Glad to hear that you got it working, Clay. That graphing does sound useful - I will have to think about adding it in the examples for the top N classes. What graphing library did you use (i.e. matplotlib?), and did you notice any performance impact?

5

Performance impact was minimal.
It would be a neat feature to add for the top 3 to 5 classes so you can visualize what the “network” is thinking.

To pay you back for your time and give back to the community here are the pertinent parts of the plotting code I used to create a 50 sample rolling plot for 3 classes (Empty, OnePerson, Twopeople). It was modeled after a Paul McHorter Tutorial and altered to work by a total newbie (myself).

Thanks Again!

 #Plotting Imports
 import datetime as dt
 import matplotlib.pyplot as plt
 from drawnow import *   # this must be downloaded, extracted, and installed per: 
 https://pypi.org/project/drawnow/  

 EmptyConfidence = []
 OnePersonConfidence = []
 TwoPeopleConfidence = []

 plt.ion() #Tell matplotlib you want interactive mode to plot live data
 cnt=0

 #plotting code from: https://toptechboy.com/python-with-arduino-lesson-11-plotting-and-graphing-live-data-from-arduino-with-matplotlib/

def makeFig(): #Create a function that makes our desired plot
     plt.ylim(0,1)                                 #Set y min and max values
     plt.title('Confidence %')      #Plot the title
     plt.grid(True)                                  #Turn the grid on
     plt.ylabel('%')                                 #Set ylabels
     plt.plot(EmptyConfidence, 'b--', label='Empty %')       #plot empty
     plt.plot(OnePersonConfidence, 'g--', label='One Person %')      
     plt.plot(TwoPeopleConfidence, 'r--', label='Two People %')
     plt.legend(loc='upper left')                    #plot the legend


Empty = net.GetConfidence(0)
One = net.GetConfidence(1)
Two = net.GetConfidence(2)
print('Empty Confidence =',Empty)
print('One Person Confidence =',One)
print('Two People Confidence =',Two)

EmptyConfidence.append(Empty)  
OnePersonConfidence.append(One)
TwoPeopleConfidence.append(Two)

drawnow(makeFig) #draw the plot figure)
plt.pause(.000001) # maybe unnecessary, untested
cnt=cnt+1

if(cnt>50):   #purge data older than 50 samples
	EmptyConfidence.pop(0)
	OnePersonConfidence.pop(0)
	TwoPeopleConfidence.pop(0)