Combining all jetbot functionalities

Autonomous Machines Jetson & Embedded Systems Jetson Nano
1

Hello there
I recently got the Jetbot robot kit and started to experiment with it and
I did all the examples in the notebook.

Now I am trying to make the robot do all the functionalities at the same time
like obstacle avoidance with road following and object detection.
I built a track that simulate a real world road with signs, people, obstacles and so on and I want the jetbot to navigate through it.

my questions are:

  1. Is it possible to accomplish all of this with the jetbot?

  2. Should I make multiple CNN models for each task and run them at the same time or is there a better way to do so?

3

Hi Al-Khattab,

Thanks for reaching out!

One option is to train three separate models and run them simultaneously. However, depending on the models, the runtime or memory consumption may grow too large for real-time deployment on Jetson Nano. This would be a good starting point.

If you have data on-hand (or can collect it) you could also try training a “hybrid” model. In this instance, each model uses the same backbone “feature extractor” and only trains the last layer or two for that task specifically. From my experience, this can work, but in some instances it can be tricky to balance each task (ie: to ensure the accuracy in one task is not biased over another).

Please let me know if this helps or you have any additional questions.

Best,
John

4

Hi john
thanks for the reply
yes I tried running 2 neural networks and as you said the performance is too low for real time deployment
I collected data and train a neural network for each task using transfer learning.

It is the first time I hear about the “hybrid” model I will have to do some research on that

I’m still new to this field and only have basic knowledge with pytorch.

can you please give me a heads up on what to look for or how can I build such model

thanks for your help.

5

Hi Al-Khattab,

No problem! I’m not sure if “Hybrid” is the correct term but the concept in this instance is pretty simple. I’ll base it assuming you’ve done the JetBot road following / collision avoidance examples.

Suppose we have two models resnet18_road_follow, resnet18_collision_avoidance. Each is defined as follows

model = torchvision.models.resnet18(pretrained=True)
model.fc = torch.nn.Linear(512, 2)

Each takes an image at 224x224 resolution and has 2 outputs: For collision avoidance these are [prob blocked, prob free]and for road following [x, y] of the target point.

The only difference is

  • The loss function we use during training: For collision,we use cross entropy, for road following we use mean squared error
  • The data provided during training: For collision we provide the class label (blocked, free), for road following we provide the target point x, y coordinates.

The architecture is identical, and a bulk of the model is likely used to compute natural image features which are shared for each task. We could then potentiall just create one model like this

model = torchvision.models.resnet18(pretrained=True)
model.fc = torch.nn.Linear(512, 4)

The number of outputs of the final layer is now 4. We can get the respective task values

collision_out = output[..., 0:2]
road_out = output[..., 2:4]

And compute a combined loss

collision_loss = ... # code to compute cross entropy for collision task
road_loss = ... # code to compute x, y mean squared error
loss = a * collision_loss + (1.0 - a) * road_loss

Where a is just some scalar variable like 0.5 to weight the relative importance of each task.

I’m not sure how well this would work, you may encounter some difficulty balancing the importance each task, but it’s something to try. You could also try some variants like

  • Holding the backbone weights constant
  • Replacing more than just the final layer

etc.

Please let me know if this helps or you have any questions.

Best,
John

6

Hi john
Thank you very much for the informative explanation
it is much easier to understand now
I will try to apply what you have explained and let you know what happens

best,
Al-khattab

7

Glad to hear it is helpful. Good luck, and look forward to hearing how it goes!

Please feel free to reach out if you run into issues.

Best,
John

8

Hi John,
I have set the torch size to 4 like you proposed. When I now run model.load_state_dict(torch.load(‘best_steering_model_xy.pth’)) it gives me a runtime error. Also I don’t quite understand, what I should write instead of the three dots to get the respective task values.
Please help us out

Thank you,

David

9

Where to download best_steering_model_xy.pth now ???