People Avoidance and Following in Isaac Sim Using Isaac ROS U-NET Segmentation

Autonomous Machines Robotics - Isaac Isaac ROS
, , , ,
1

1280×720 41.8 KB

People Avoidance and Following in Isaac Sim Using Isaac ROS U-NET Segmentation

Overview

Isaac ROS Image Segmentation includes ROS packages designed for semantic image segmentation. These packages enable pixel-level classification of input images by performing GPU-accelerated inference with deep neural network models. Each pixel in the input image is classified into predefined categories, allowing perception systems to determine the spatial distribution of each class within a 2D image. This output can also be integrated with depth information to create a 3D scene understanding.

DNN Model Overview of PeopleSemSegNet for Autonomous Mobile Robots (AMRs)

The PeopleSemSegNet model is designed for autonomous mobile robots (AMRs) to detect and segment human figures within images. This model provides a high level of accuracy in identifying one or more people in various environments, making it highly suitable for commercial applications.

Key Features

  1. Human Detection and Segmentation:
  • PeopleSemSegNet is trained to detect and segment “person” objects in images. It generates a semantic segmentation mask that outlines all detected individuals with high precision, allowing the AMR to differentiate people from other objects in the scene.
  1. Semantic Segmentation Mask:
  • The model outputs a detailed segmentation mask where each pixel is classified as either belonging to a person or the background. This mask is essential for tasks that require precise localization and interaction with people.
  1. Applications in AMRs:

PeopleSemSegNet enhances the capabilities of AMRs by enabling functionalities such as:

  • People Avoidance: Allowing the robot to navigate around detected people to avoid collisions.
  • People Following: Directing the robot to follow a specific person or interact with them based on the segmentation mask.
  • Human-Robot Interaction: Facilitating natural and safe interactions with humans in various environments, such as offices, hospitals, or public spaces.
  1. Integration with ROS:
  • In an Isaac ROS environment, the model’s output is typically published on a topic such as /unet/colored_segmentation_mask. This integration allows other ROS nodes to subscribe to this topic, process the segmentation data, and make decisions based on the detected human figures.

Why People Avoidance and Following are Important:

Discuss use cases like warehouse robots, autonomous delivery systems, and social robots.

  1. Warehouse Robots:
  • People Avoidance: In a warehouse setting, robots often operate in dynamic environments with human workers moving around. People avoidance is crucial to prevent collisions, ensuring the safety of both the robots and the workers. Robots equipped with people avoidance capabilities can navigate around obstacles and adjust their paths to avoid interfering with human activities.
  • People Following: Some warehouse robots may be designed to follow human operators, such as inventory pickers or supervisors. By following these individuals, robots can assist with carrying tools, restocking items, or even providing real-time inventory updates, enhancing productivity and streamlining operations.
  1. Autonomous Delivery Systems:
  • People Avoidance: Delivery robots operating in public spaces or residential areas must be able to navigate around pedestrians to avoid accidents and ensure safe delivery. People avoidance technology allows these robots to detect and avoid individuals, minimizing the risk of collisions and improving overall service quality.
  • People Following: In some scenarios, autonomous delivery robots may need to follow a specific person, such as a delivery recipient, to complete a transaction or hand over a package. This capability is particularly useful in environments where the recipient is moving or when the delivery process requires personal interaction.

figure 1

In Figure 1, the Isaac Sim simulation demonstrates the integration of Isaac ROS Unet image segmentation for people following. When the topic /unet/colored_segmentation_mask is activated, it provides segmentation information where people are highlighted in red. Using this segmentation data, the robot is able to identify and track individuals within the warehouse environment, allowing it to follow humans effectively.

Key Isaac ROS Packages for People Avoidance and Following:

1. isaac_ros_image_pipeline :

  • This package provides a suite of tools for image processing and segmentation. It enables the robot to visually recognize and interpret its surroundings, which is crucial for tasks such as people avoidance and following. By processing camera feeds and segmenting the images, this package helps the robot identify and track objects or individuals within its environment.
  1. isaac_ros_unet :
  • The isaac_ros_unet package leverages U-Net-based segmentation models to perform semantic segmentation. U-Net is a deep learning architecture specifically designed for image segmentation tasks, allowing the robot to distinguish and segment different objects in an image. When used for people detection, the U-Net model can accurately identify and outline individuals by assigning a specific class label (e.g., people) to each pixel. This capability enables the robot to follow or avoid people based on the segmented output, where individuals are highlighted distinctly (e.g., in red).

4. isaac_ros_dnn_image_encoder :

This package is designed for encoding input images to facilitate efficient processing by deep neural network models. The isaac_ros_dnn_image_encoder prepares and transforms camera feeds into a format suitable for deep learning algorithms, such as those used in isaac_ros_unet. This preprocessing step ensures that the segmentation models receive high-quality and appropriately formatted input, leading to more accurate and reliable detection and segmentation of people or other objects.

1455×909 97.2 KB

figure 2

In Figure 2 & 3, the RQT graph illustrates the interaction between key Isaac ROS packages involved in people avoidance and following:

  • isaac_ros_image_pipeline: This package is responsible for processing and segmenting image data from the robot’s sensors. It prepares the image feed for further analysis and segmentation.
  • isaac_ros_unet: The U-Net-based segmentation model is used to perform semantic segmentation on the processed images. This model identifies and classifies objects, such as people, within the images based on the segmentation mask provided.
  • isaac_ros_dnn_image_encoder: This package encodes the input images to make them compatible with deep neural network models. It ensures that the images are formatted correctly for accurate and efficient processing by the isaac_ros_unet model.

1455×909 109 KB

figure 3

figure 4

In Figure 4 , the Isaac Sim simulation demonstrates the integration of Isaac ROS Unet image segmentation for people avoidance. When the topic /unet/colored_segmentation_mask is activated, it provides segmentation information with people highlighted in red. Using this segmentation data, the robot can detect the presence of individuals within the warehouse environment and adjust its path to avoid collisions, ensuring safe and efficient navigation around humans.

Advantages of Using the isaac_ros_unet Image Segmentation Package with the PeopleSemSegNet AMR Model.

  1. Accurate People Detection:

The isaac_ros_unet package uses the U-Net model, which is highly accurate for identifying people in images. This ensures the robot can reliably detect individuals.

  1. Real-Time Processing:

The package allows the robot to process and segment images in real time. This is crucial for quickly reacting to changes in a busy environment like a warehouse.

  1. Enhanced Safety:

By accurately detecting people, the robot can avoid collisions and navigate safely around individuals, improving safety for both the robot and people.

  1. Better Efficiency:

With precise people detection, the robot can plan its movements more effectively, leading to faster and more efficient operations in environments like warehouses.

Conclusion

The integration of the isaac_ros_unet image segmentation package with the PeopleSemSegNet model significantly enhances the capabilities of autonomous mobile robots (AMRs) in dynamic environments. By leveraging high-accuracy semantic segmentation, real-time processing, and seamless ROS integration, robots can effectively detect and avoid people, ensuring safe and efficient navigation. The flexibility and customization options of the isaac_ros_unet package make it a valuable tool for adapting to various operational needs and environments. Overall, this technology advances the safety and operational efficiency of AMRs, paving the way for smarter and more reliable robotics solutions in settings such as warehouses and other complex environments.

2

Thanks for sharing such a detailed post. It will greatly benefit the community!

Raffaello