Rock-Paper-Scissors with Jetson Nano

(Link to the GitHub repository: https://github.com/mokpi/Rock-Paper-Scissors-with-Jetson-Nano)

Jetson Nano can now play Rock-Paper-Scissors with you!

This project utilizes both the GPIO and deep learning capability of the Jetson Nano and created a simple game of rock-paper-scissors to play around with.

When you are playing rock-paper-scissors with Jetson Nano,

  1. GPIO is used to rotate a board, which indicates the device's choice of rock/paper/scissor.
  2. GPIO can also be used to indicate the timing of the game. (Rock-..Paper-..Scissors..Go!)
  3. The deep learning part is used to recognize the player's hand gesture through camera input.
  4. A lot of people ask about using the deep learning part to recognize and predict the player's next hand, but we're trying to create a fair game here. =)
1 Like

May i ask what is the jet cam you mention in the github post and when i try to run the test_game.ipynb file it keeps crashing the kernel and it stuck at the first section of codes?

“jetcam” is a small library for camera input of jetson boards.
Some source code and documentation from nvidia: https://github.com/NVIDIA-AI-IOT/jetcam

Yes, if you don’t have a CSI camera, connected to the jetson nano with a ribbon cable, than the import would not be successful.
The CSI camera is used to get image input.

If you don’t have a CSI camera, you can use a USB camera.
You’d need to change the code a little though: https://github.com/NVIDIA-AI-IOT/jetcam#usage

Hey i have another question to as i tried to run the ipynb files however when i am running the test_game.ipynb it kept crashing on the first cell, may i ask if you know what is the issue?

I’d suggest breaking them up into 3 separate cells and run them in order and see which part cause the crashing.
May be it’s the camera part.

#PyTorch initilize
import torch
import torchvision

#random
import random

### import randoms
import math

### import colors
from colorama import Fore, Back, Style

model = torchvision.models.alexnet(pretrained=False)

#change the "3" when more categories are added
model.classifier[6] = torch.nn.Linear(model.classifier[6].in_features, 3)

#read the model
model.load_state_dict(torch.load('best_model.pth'))
#tranform input into the format of the model
import cv2
import numpy as np

mean = 255.0 * np.array([0.485, 0.456, 0.406])
stdev = 255.0 * np.array([0.229, 0.224, 0.225])

normalize = torchvision.transforms.Normalize(mean, stdev)

def preprocess(camera_value):
    global device, normalize
    x = camera_value
    x = cv2.cvtColor(x, cv2.COLOR_BGR2RGB)
    x = x.transpose((2, 0, 1))
    x = torch.from_numpy(x).float()
    x = normalize(x)
    x = x.to(device)
    x = x[None, ...]
    return x
#acquire camera image
from jetcam.csi_camera import CSICamera
import ipywidgets
from IPython.display import display
from jetcam.utils import bgr8_to_jpeg

camera = CSICamera(width=224, height=224)
image = camera.read()
image_widget = ipywidgets.Image(format='jpeg')
image_widget.value = bgr8_to_jpeg(image)

#Update camera image
camera.running = True

def update_image(change):
    image = change['new']
    image_widget.value = bgr8_to_jpeg(image)
    
camera.observe(update_image, names='value')

#GPU
device = torch.device('cuda')
model = model.to(device)

First cell of https://github.com/mokpi/Rock-Paper-Scissors-with-Jetson-Nano/blob/master/test_game.ipynb.

May i ask if i want to add another image files that show the background as no result, where and how should i put it?

  1. First, you’ll need to add a new category in “collection.ipynb”.
    Cell 3:
#create dataset folders
import os

one_dir = 'dataset/one'
two_dir = 'dataset/two'
three_dir = 'dataset/three'
bg_dir = 'dataset/backgrounds' # ADDED

# we have this "try/except" statement because these next functions can throw an error if the directories exist already
try:
    os.makedirs(one_dir)
    os.makedirs(two_dir)
    os.makedirs(three_dir)
    os.makedirs(bg_dir) # ADDED
except FileExistsError:
    print('Directories not created becasue they already exist')

Cell 4:

#create buttons
import ipywidgets.widgets as widgets

button_layout = widgets.Layout(width='128px', height='64px')
one_button = widgets.Button(description='add to "one"', layout=button_layout)
two_button = widgets.Button(description='add to "two"', layout=button_layout)
three_button = widgets.Button(description='add to "three"', layout=button_layout)
bg_button = widgets.Button(description='add to "backgounds"', layout=button_layout) # ADDED

one_count = widgets.IntText(layout=button_layout, value=len(os.listdir(one_dir)))
two_count = widgets.IntText(layout=button_layout, value=len(os.listdir(two_dir)))
three_count = widgets.IntText(layout=button_layout, value=len(os.listdir(three_dir)))
bg_count = widgets.IntText(layout=button_layout, value=len(os.listdir(bg_dir))) # ADDED

display(widgets.HBox([one_count, one_button]))
display(widgets.HBox([two_count, two_button]))
display(widgets.HBox([three_count, three_button]))
display(widgets.HBox([bg_count, bg_button])) # ADDED

Cell 5:

#Take photo and save at button press
from uuid import uuid1

def save_snapshot(directory):
    image_path = os.path.join(directory, str(uuid1()) + '.jpg')
    with open(image_path, 'wb') as f:
        f.write(image_widget.value)

def save_one():
    global one_dir, one_count
    save_snapshot(one_dir)
    one_count.value = len(os.listdir(one_dir))
    
def save_two():
    global two_dir, two_count
    save_snapshot(two_dir)
    two_count.value = len(os.listdir(two_dir))
    
def save_three():
    global three_dir, three_count
    save_snapshot(three_dir)
    three_count.value = len(os.listdir(three_dir))
    
def save_bg():  # ADDED
    global bg_dir, bg_count
    save_snapshot(bg_dir)
    bg_count.value = len(os.listdir(bg_dir))
    
# attach the callbacks, we use a 'lambda' function to ignore the
# parameter that the on_click event would provide to our function
# because we don't need it.
one_button.on_click(lambda x: save_one())
two_button.on_click(lambda x: save_two())
three_button.on_click(lambda x: save_three())
bg_button.on_click(lambda x: save_bg())  # ADDED

Cell 6:

#Show them together
display(image_widget)
display(widgets.HBox([one_count, one_button]))
display(widgets.HBox([two_count, two_button]))
display(widgets.HBox([three_count, three_button]))
display(widgets.HBox([bg_count, bg_button]))  # ADDED

(These should be done in a loop for a more concise code.)

  1. Run “collection.ipynb” to collect some images for backgrounds.

  2. Then, edit “train_model.ipynb” to add the new category to the model. You’ll need to retrain the model.
    At cell 6, change the “3” to a “4”.

#set to *4* categories
#number of categories need to be adjusted here
NUM_CATEGORIES = 4
model.classifier[6] = torch.nn.Linear(model.classifier[6].in_features, NUM_CATEGORIES)
  1. You can then run cells in order to train the model.

  2. Change line 17 in both the “test_game.ipynb” and “game.ipynb” files.

#number of categories need to be adjusted here
NUM_CATEGORIES = 4
model.classifier[6] = torch.nn.Linear(model.classifier[6].in_features, NUM_CATEGORIES)
  1. In the “Machine Learning: Deduction using the trained network” section, change the following part as shown:
# change lines 19 to 30 to below
    one_blocked = float(y.flatten()[0])
    two_blocked = float(y.flatten()[1])
    three_blocked = float(y.flatten()[2])
    bg_blocked = float(y.flatten()[3])

    # if(one_blocked > two_blocked and one_blocked > three_blocked):
    #     a = 0
    # elif(two_blocked > one_blocked and two_blocked > three_blocked):
    #     a = 1
    # elif(three_blocked > one_blocked and three_blocked > one_blocked):
    #     a = 2
    values = [one_blocked, two_blocked, three_blocked, bg_blocked]
    a = values.index(max(values)) # more consice

    #0 rock 
    #1 paper
    #2 scissors   
    #3 background

(You may also change line 3 to this)

a=3 # background by default
  1. Change line 55 in the “Gameplay: actual gameplay process” section, from
sys.stdout.write("\tR:%f, P:%f, S:%f\t" % (one_blocked, two_blocked, three_blocked) )

to

sys.stdout.write("\tR:%f, P:%f, S:%f\t, BG:%f\t" % (one_blocked, two_blocked, three_blocked, bg_blocked) )
  1. Change the “name_of_value” function.
def name_of_value(val):
    if val == 0:
        return "Rock       ";
    if val == 1:
        return "Paper      ";
    if val == 2:
        return "Scissor    ";
    if val == 3:
        return "Background ";

OR rewrite it like this:

def name_of_value(val):
    return ["Rock       ", "Paper      ", "Scissor    ", "Background "][val];
  1. Modify the “win or lose” logic to make it work correctly.

  2. Run the cells in order to play the game!

Please reply if I missed anything that needs modification.

-------------------------------------------------------------------
11. (Extra-Hint: change cell 3 of “train_model.ipynb” for different proportion of testing and training sample images.)

#Create training and testing data
train_dataset, test_dataset = torch.utils.data.random_split(dataset, [len(dataset) - 25, 25]) # 25 images used for testing.

sorry for not mentioning this but i am trying to change the files from ipynb to .py . However there are some difference in the command use and i cant really solve it, i tried to use tkinter instead of ipywidget as widget seems to be only able to run on ipython. But even after installing tkinter into the system,i is still not detecting it.
As for the IPython.display, what can i replace it with on python?

I have not tried porting it to python with GUI, so I don’t really know the answers to these questions.

However, from these two lines:

image = camera.read()
image_widget = ipywidgets.Image(format='jpeg')

You may see that you can just skip creating a widget and see if there is any tkinter API for showing images…

A Google search gives me this result: https://pythonbasics.org/tkinter-image/

From the documentation of “ImageTk.PhotoImage”(https://pillow.readthedocs.io/en/4.2.x/reference/ImageTk.html#PIL.ImageTk.PhotoImage), it accepts PIL image format.

Reading https://github.com/NVIDIA-AI-IOT/jetcam/blob/master/jetcam/usb_camera.py reveals that the jetcam library uses cv2 openCV image format.

As I was going to research on how to convert from cv2 image format to PIL images, I figured may be this line

cv2.imshow("OpenCV",image)

may be able to show the image immediately since we have cv2 imported already in the python code.

May i ask what is the one_blocked, two_blocked, three_blocked for at the test_game.ipynb?

Those are the confidence values from the neural network for each of the categories one(rock), two(paper) and three(scissors). The highest value would be chosen.

Another question here. Somehow my test accuracy when training is 100%. When i check the problem , i found out that the test error count is 0. To my understanding is that the test error count is from (labels - output.argmax(1)), the values of both labels and output.argmax(1) is both there when i printed it out however when i print test error count the value printed out was 0. May i ask how the label is counted so that it could minus output.argmax(1) and why the value is 0.

It is indeed very weird, but I’m not very familiar with this part of the code.
I did not notice this problem. Or maybe I actually saw non-100% accuracy before. My memory isn’t reliable.

Also, I’m not very familiar with deep learning programming…

I agree that the key here is “labels - outputs.argmax(1)”, which I have not idea why do the author wrote it this way.
I’ll look into this project again when I have time. But now I’m quite busy.

If you have found anything, maybe you can share it here.

May i ask if you know how the train_model.ipynb recognized the difference in picture for rock, paper and scissor. I added the background in and finish training it but when i try to run it on test game the value on background id constantly staying at 0.
And one more thing, for the game.ipynb i tried to run it but it some how showed ‘ModuleNotFoundError: No module named ‘Jetson’’ and i have no idea why that is happening.

For the 2nd question, game.ipynb requires GPIO objects. We used a button to start a new game, a step motor to rotate a board and indicate the devices’ choice on the game.

For the 1st question, what do you mean by “the value on background id constantly staying at 0”? Do you mean the recognition result is always background, or the device never think anything can be the background (value for bg_blocked is always zero)?

1 Like

the device never think anything can be the background (value for bg_blocked is always zero) even after training multiple time and i have increased the training data to 600 pictures.