How to combine Road Following and Object Detection?

Autonomous Machines Jetson & Embedded Systems Jetson Nano
#1

Hi,

Currently I have Jetbot (Jetson nano robot based).

I would like to make a road follower robot like what the example has done. But I also want to make the robot will stop for a while (for example 5 seconds) at a stop sign(it could be anything, for example a cross sign on the road) and then continue to follow the road. So I will have a small circle track, and on that track there is a stop sign on the track.

Therefore I still can not imagine how to do that? Do I need to run 2 programs simultaneously (road follower and object detection)? But does it make a conflict since the road follower tell the robot to run but the object detection tell the robot to stop?

any idea how to implement this?

#2

Hi,

It looks like the detector is an interrupt signal for the road follower robot.

A suggestion is :
If the detector find a sign, the robot get freeze for several seconds without receiving the instruction from road follower net.
After the freeze period, the robot can restart to get the instruction from road follower net again.

Suppose you can define similar rules or finite-state machine for your use case.
Thanks.

#3

Great! thanks for your answer. I am still new in machine learning and jetson nano. How do I make the object detection part? When I look at the code of the object following example, there is no code for training, it just purely use the pre-training model. Do I need to make it from the scratch?

I also still confuse about why we use two predefined model in the object following example? In order to achieve my goal which part I have to follow?

#4

Hi,

Sorry for the late update.

You will need to collect a dataset based on your use case first.
After that, you can train a detector with the instructions shared in this tutorial:
https://github.com/dusty-nv/jetson-inference/blob/master/docs/detectnet-training.md

The model used in jetbot is for obstacle detection and you need another model for the sign.
If you don’t need to take care of obstacle avoidance, a sign detector should be enough for you.

Thanks.

#5

hi do you slove it? I want to do the same, but has some bugs when use two models, could you share you code?

#6

I built a object detection using tensorflow. You find more the information using “Custom Object Detection Tensorflow” keyword. I failed to optimize it on Jetson Nano, but it is still work.

So when the robot detected an object, i made the robot to stop for about 5 seconds.

#7

My code is like the object following(use ssd_mobile and road_following model) , but when i load all two module, the jetbot turn round , i guess it is because of the low memory, what’s about your code?

#8

I repeat it with combine road following and Collision Avoidance, it’s just make two models, but i slow down the process speed of camera image. I don’t known if it is the performance of nano or the camera

#9

It’s likely your performance reduction is because of GPU utilization, but you can measure this for sure with the NVIDIA performance tools. Beats guessing :-)

The system architecture for a system that uses multiple models will generally need to be different. Generally, you will want to modify the software such that it has one input for each sensors, then splits those streams to all the models that need it, then have a bottom tier which reads the output from all the models and decides how to actually control the underlying hardware.

You can do this by modifying the various sample programs, and making them coordinate correctly, as long as you understand the software development parts involved.

To create some kind of basic framework for systems that look like this, was basically why ROS was invented. If you’re going to go far into this direction, it may make sense to port the models you need to ROS and use that as your framework.