How to combine object detection and road following

Autonomous Machines Jetson & Embedded Systems Jetson Nano
1

Hi, everyone!
Now i have a task, let the jetbot do road following while recognizing traffic signs, for example turn left, turn right, speed limit, no speed limit, stop and traffic lights. But i don’t know how to do this?
Now i have a well-trained road following model and a self-trained detection model using SSD follow this: jetson-inference/pytorch-collect-detection.md at master · dusty-nv/jetson-inference · GitHub
As Combine Jetbot Road Following and Collision Avoidance Tasks shows that road following can combine with collison avoidence, but there’s only two types of detection, free and blocked.
For my task i have more than 5 traffic signs to detect and the let jetbot follow the corresponding rules.
So what should i do now? road following + collision avoidence with many types or road following with object detection? Or should i create a new jupyter notebook to combine those two parts, but i’m not very sure about the logic inside.

3

Hi,

You can try to extend the script with your own use case.

For example, the detection value can be extracted in the following line.
Then you can add the corresponding handler based on your requirement.

https://github.com/abuelgasimsaadeldin/Jetbot-Road-Following-and-Collision-Avoidance/blob/main/combine_scripts/RoadFollowing%2BCollisionAvoidance.ipynb

#Collision Avoidance model:
prob_blocked = float(F.softmax(model_trt_collision(image_preproc), dim=1).flatten()[0])

Thanks.

4

Thanks for your suggestion.
I will try it later.
But i have a question, thats to say, by the object detection we can use camera capture tool to label objects and add boxes around them. But for collision avoidance we can only label the images without adding boxes. So i dont know if there will be a problem when we use collision avoidance, because there might be more than one traffic sign in one image.(In my opinion these signs could be distinguished by different distances, but how?)

5

Hi,

Do you mean you can only do the classification rather than detection in collision avoidance?
If you want to detect ‘each’ sign, you will need a detector for it.

Thanks.

6

Hi, @AastaLLL and @dusty_nv !
after training i get model.pth, which i wanna use in roadfollowing + object detection. but first i must define the model(e.g.
model = torchvision.models.resnet18(pretrained=False)
) to load the model.path. when i load the model.path:
model.load_state_dict(torch.load(‘model.pth’)),
it obviously doesnt match e.g. resnet18.
i wanna find the right neural network under package torchvision.models, but after checking the official site von package torchvision.models, there is no SSD-mobilenetV1. So how can i do to load the trained model here to be combined with RoadFollowing?

7

Hi,

Would you mind sharing more information about the issue you met?
Do you want to load the model trained by jetson-inference with PyTorch API?

Thanks.

8

Hi @MaxMarth, you might find this project helpful:

JETBOT ROAD FOLLOWING AND COLLISION AVOIDANCE TASKS

Combine optimized Road Following and Collision Avoidance models to enable Jetbot to move freely around the track and also avoid collisions with obstacles at the same time. […] Combination of Road Following and Collision Avoidance models to allow the Jetbot to follow a specific path on the track and at the same time also be able to avoid collisions with obstacles that come on it’s way in real-time by bringing the Jetbot into a complete halt!

https://developer.nvidia.com/embedded/community/jetson-projects#jetbot_road_follow_collision_avoid

9

Hi @dusty_nv @AastaLLL !
actually i tried to use the method mentioned in this link. But it can not satisfy the requirements. So i discuss with my teammates and we think the only way is to use the methond mentioned in jetsen-inference, which can train the ssd-model with our own dataset.
As i mentioned above, in jetbot i can not direct load the trained .onnx file. So we try to convert it so that it can be used in jetbot. After few attemps we try to convert .onnx file into .engine. And as shown in the example code in Object Following in jetbot we can add it with road following.
Because of different visions of software packages we are still trying to make some breakthrough.
And we tried to change the file detection.py in jetson-inference/examples/detectnet and use it into jetbot. But always the error: No module named ‘jetson’. I wonder if i need to move or clone again the whole jetson-inference folder under the folder of jetbot.
Further we will try to use the method mentioned by Dusty, which suggests us to rewrite object_detection.py in jetbot/jetbot to use jetson.inference.detectNet. Hopefully it can success.

10

Hi Max, if you are using the jetbot containers, my guess is that jetson.inference is not installed into those containers.

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.