Issue with pre-processing the camera with Resnet18 Collision Avoidance Model

Autonomous Machines Jetson & Embedded Systems Jetson Nano
, ,
1

I have a dataset consisting of chairs that was trained with the Resnet 18 neural network. The dataset has pictures in a ‘blocked’ and ‘free’ folder. When I try to load the trained model onto the Nano, I get an error. This is the preprocessing function which works:

import torchvision.transforms as transforms
import torch.nn.functional as F
import cv2
import PIL.Image
import numpy as np

mean = torch.Tensor([0.485, 0.456, 0.406]).cuda().half()
std = torch.Tensor([0.229, 0.224, 0.225]).cuda().half()

normalize = torchvision.transforms.Normalize(mean, std)

def preprocess(image):
    image = PIL.Image.fromarray(image)
    image = transforms.functional.to_tensor(image).to(device).half()
    image.sub_(mean[:, None, None]).div_(std[:, None, None])
    return image[None, ...]

This is displaying the camera with a ‘blocked’ slider (which works):

import traitlets
from IPython.display import display
import ipywidgets.widgets as widgets
from jetbot import Camera, bgr8_to_jpeg

camera = Camera.instance(
    camera_type='csi',
    device_id=0,
    width=224,
    height=224,
    capture_fps=30,
    gstreamer_pipeline='nvarguscamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)30/1 ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink'
)

image = widgets.Image(format='jpeg', width=224, height=224)
blocked_slider = widgets.FloatSlider(description='blocked', min=0.0, max=1.0, orientation='vertical')

camera_link = traitlets.dlink((camera, 'value'), (image, 'value'), transform=bgr8_to_jpeg)

display(widgets.HBox([image, blocked_slider]))

This is creating the neural network execution function which works:

from jetbot import Robot

robot = Robot()

import torch.nn.functional as F
import time

def update(change):
    global blocked_slider, robot
    x = change['new'] 
    x = preprocess(x)
    y = model(x)

    y = F.softmax(y, dim=1)
    
    prob_blocked = float(y.flatten()[0])
    
    blocked_slider.value = prob_blocked
    
    time.sleep(0.001)
        
update({'new': camera.value})

All of this runs, however, when I try to use:

camera.observe(update, names='value')

I get an error. It is ‘TypeError: ‘dict’ object is not callable’:

observe_error
observe_error1920×1200 361 KB

I’m not sure how to fix this. My goal is to move the CSI camera which is attached to the Nano infront of an obstacle (chair) and see if the slider moves up to 1. If it does then I know my model works and if it doesn’t, I probably need to fix it. However, I get an error when running this and I’m not sure how to fix it.

Any help would be appreciated. Thanks in advance!

3

Hi,

It looks like you have a variable called “update” and also a function called “update”.
Could you rename either one to see if it helps?

Thanks.

4

You are right, I meant to do

update({‘new’: camera.value})

I get a new issue now. The issue is when I run my blocks, the block which displays the video and slider (5th block) crashes the notebook:

kernelerror
kernelerror1920×1080 301 KB

I tried to run the blocks in a different order when I ran the 5th one first and then first 4 but when I get to the 7th block which is this:

from jetbot import Robot
robot = Robot()
import torch.nn.functional as F
import time
def update(change):
global blocked_slider, robot
x = change[‘new’]
x = preprocess(x)
y = model(x)
y = F.softmax(y, dim=1)
prob_blocked = float(y.flatten()[0])
blocked_slider.value = prob_blocked
time.sleep(0.001)

update({‘new’: camera.value})

the screen just freezes. I also removed the

update({‘new’: camera.value})

and just ran the next block of:

camera.observe(update, names = ‘value’)

and got the same freeze issue. I think it has to do with the ‘update’ function and maybe the ‘preprocess’ one which the ‘update’ calls. I’m not sure. If I run it from top to bottom the whole thing crashes and if I try to run 5, 1-4, 6+ then the block which calls ‘update’ freezes my screen.

5

I think it’s a memory problem and if it is do you know any way to allocate more free memory? Or if it’s a different issue altogether like it has to do with the camera.

6

Hi,

You can try to add some swap memory.
But CUDA-based jobs cannot use swap since it is not accessible via GPU.

Thanks.

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.