Some Issues with Sign Language custom model inference on webcam feed

Autonomous Machines Jetson & Embedded Systems Jetson Nano
#1

Hello,
I have trained a model to recognize when I am performing Sign Language and I have managed to load it along with its state dictionary. The thing I am trying to achieve is to load my model into the notebook, start a stream from my webcam and apply the model to the webcam stream. I then just want it to print out or display on the frame the command that it is seeing,
I did it with the following code:

mod_path= '/nvdli-nano/data/classification/my_model.pth'

dev= torch.device(‘cuda’)
model = torchvision.models.resnet34()
model.fc = torch.nn.Linear(512,6)
model=model.to(dev)
model= model.load_state_dict(torch.load(mod_path))

output= model(image)
model=model.eval()

I am getting this error in model.eval()
AttributeError: ‘_IncompatibleKeys’ object has no attribute ‘eval’

I am also having webcam issues.
Using the code below:

!ls -ltrh /dev/video*
from jetcam.usb_camera import USBCamera
camera = USBCamera(width=224, height=224, capture_device=0)
camera.running = True

image=camera.read()/

camera.read() gives me an error saying:
RuntimeError: Cannot read directly while camera is running
Which is odd since this issue does not happen in the hello_camera notebook

Alternatively I have tried to run this model on my laptop using opencv, but am having trouble getting the model to predict the frames.

cam=cv2.VideoCapture(0)
while True:
ret,frame = cam.read()
cv2.imshow(‘frame’,frame)
ten=transforms.ToTensor()
tensor=ten(frame)
tensor=tensor.unsqueeze(0)
model.eval()
output=model(torch.as_tensor(tensor))
print(out[0])
if cv2.waitKey(1) & 0xFF == ord(‘q’):
break
cam.release()

This gives an error
RuntimeError: Could not infer dtype of ToTensor

Interestingly enough, I am not having issues with loading the models in Jupytre on my laptop. I got the All keys Matched Successfully message, but I’m having trouble with applying the model to webcam frames to classify what it sees in the frames.
Please do let me know if you need me to clarify the issue and I really appreciate any help I can get

#3

Hi,

1. AttributeError: ‘_IncompatibleKeys’ object has no attribute ‘eval’

Could you update the source to the following and try it again?

model=model.to(dev)
model.load_state_dict(torch.load(mod_path))

2. RuntimeError: Cannot read directly while camera is running
This error indicates that there is another application using the same camera.
Please close the app and try it again.

3. RuntimeError: Could not infer dtype of ToTensor
Could you try this?

output=model(torch.tensor(tensor))

Thanks.

#4

Hi, so for the camera I keep having this problem. I turned off the other kernels before trying again but the issue persists. I also checked to see the usb camera notebook in hello_camera and in the classification_interactive notebook example. The camera works just fine in both those cases, but for some reason I cannot get camera.read() to work when I create my own notebook.

As for the model, I redid it with a new line of code and it seems to be working fine:

import torch
import torchvision
import cv2
from torchvision.utils import *

MOD_PATH = ‘/nvdli-nano/data/classification/my_model.pth’
state_D= torch.load(MOD_PATH,map_location=torch.device(‘cuda’))

import torchvision.models as models
resnet34 = models.resnet34(pretrained=False)
resnet34.fc = torch.nn.Linear(512,6)

resnet34.load_state_dict(state_D)
optimizer = torch.optim.Adam(resnet34.parameters())
resnet34 = resnet34.eval() (outputs all the blocks in the model)

What I am trying to do from here is just get the webcam feed and simply run inference on the webcam feed using the model, then have the predictions printed out. Do you know how I can pass this webcam feed to the model(once i get camera.read() to work of course) is it as simple as typing :

image= camera.read()
output = resnet34(image)
print(output)

Thanks

#5

Hi,

Do you get any error or log when the webcam is not working?

Based on the jetcam source, not too much usb camera is tested (only Logitech C270).
You may need some update in configure for your camera:

Thanks.