Rviz error with Isaac Sim

Hello,
I’ve been try to visualize my robots in Rviz along with Isaac Sim. I’ve been running into an error where when I set the position of the robot, it goes to the right place in Isaac Sim but not in Rviz. It also makes Rviz very buggy and I can’t use Nav Goal without the robot completely bugging out. When setting the position I use
“…
rand_loc_gf = Gf.Vec3f(x, y, 0)
prim.GetAttribute(“xformOp:translate”).Set(rand_loc_gf)
…”

where x, y are coordinates in meters from the origin. The robot seems to go to the right coordinates at first but the robots random moves after a second to a different position. If anyone has into this before I’d appreciate any insight!


This is the Rviz Map. This robot should be inside the environment as shown below.

The particle swarm around it is right but the position on the map is not.

Are you running some localization algorithm (like amcl) to estimate robot pose?
Or is your robot pose in Rviz based on ground truth (odom) from Isaac Sim?
Another thing to check might be your map data. Does you map have zero offsets w.r.t to the world in Isaac Sim? How did you generate your map? Did you do it following the tutorial ROS Navigation — Isaac Sim Documentation?

Hi,
I generated the map based on that Isaac Sim tutorial. My launch file for RViz is almost exactly the same. My robot pose is based on the ground truth from Isaac Sim.
Thanks for the help!

Do you mind providing a minimal setup so that I can try to replicate it on my side? Thanks

USER_HOME = os.environ['HOME']
REPO_PATH = USER_HOME + "/.local/share/ov/pkg/isaac-sim-4.0.0/frontiers_rl"
ENVS_PATH = REPO_PATH + "/environments"

robots = ["carter1","carter2","carter3"]         #TODO: make this configurable not hardcoded

class Subscriber(Node):
    def __init__(self):
       
        
        print("init minimal subscriber")
        super().__init__('minimal_subscriber')

        self.clear_costmap_local_srvs = dict()
        self.clear_costmap_global_srvs = dict()

        for robot in robots:
            self.clear_costmap_global_srvs[robot] = self.create_client(
                ClearEntireCostmap, robot+'/global_costmap/clear_entirely_global_costmap'
            )
            self.clear_costmap_local_srvs[robot] = self.create_client(
                ClearEntireCostmap, robot+'/local_costmap/clear_entirely_local_costmap'
            )
        self.ros_sub = self.create_subscription(Bool, "reset", self.reset_issac_sim_callback, 10)
        self.reset_env = False

        
    def reset_issac_sim_callback(self,msg):
        print(f"self.reset_env: {self.reset_env}")
        self.reset_env = msg.data
        print(f"self.reset_env changed: {self.reset_env}")

    def set_reset_val(self,val):
        self.reset_env = val
        print(f"reset_env set to {self.reset_env}")
        
    def clearGlobalCostmap(self):
        """Clear global costmaps."""
        for robot in robots:
            srv = self.clear_costmap_global_srvs[robot]
            while not srv.wait_for_service(timeout_sec=0.001):
                print(f'Clear global costmaps service for robot {robot} not available, waiting...')
            req = ClearEntireCostmap.Request()
            future = srv.call_async(req)
            rclpy.spin_until_future_complete(self, future)
        return
    
    def clearLocalCostmap(self):
        """Clear local costmaps."""
        for robot in robots:
            srv = self.clear_costmap_local_srvs[robot]
            while not srv.wait_for_service(timeout_sec=0.001):
                print(f'Clear local costmaps service for robot {robot} not available, waiting...')
            req = ClearEntireCostmap.Request()
            future = srv.call_async(req)
            rclpy.spin_until_future_complete(self, future)
        return
    
    
class RandomPositionNode(Node):
    def __init__(self):
        super().__init__('random_position_node')
        self.costmap_subscriptions = []
        self.costmaps = dict()
        
        for robot in robots:
            topic = '/' + robot + '/global_costmap/costmap'
            self.get_logger().info(f"Subscribing to {topic}")
            self.costmap_subscriptions.append(self.create_subscription(
                OccupancyGrid,
                topic,  
                # lambda msg: self.costmap_callback(msg, robot),
                partial(self.costmap_callback, robot=robot),
                10
            ))
            self.costmaps[robot] = None
        
    def costmap_callback(self, msg, robot):
        """
        Callback function to update the costmap once it's received.
        """
        self.costmaps[robot] = msg
        print(f"costmap: {robot}")
        self.get_logger().info(f"msg: {msg.header}")
        
        
    
    def get_costmap(self,robot):
        """Returns the latest received costmap."""
        return self.costmaps[robot]
     
    
    def get_costmaps(self):
        """Returns the dict latest received costmap."""
        return self.costmaps

def random_pos(bot, randomization_node):
    """
    Generates a random valid position within the costmap.
    """
    rclpy.spin_once(randomization_node, timeout_sec=0.001)
    costmap = randomization_node.get_costmap(bot)
    print("checking costmaps")
    
    if costmap == None:
        print("No Costmap :(")
        return None
  
    # Costmap properties
    width = costmap.info.width
    height = costmap.info.height
    resolution = costmap.info.resolution
    origin = costmap.info.origin
    
    # Generate random coordinates in the grid
    x_index = random.randint(0, (width - 1))
    y_index = random.randint(0, (height - 1))
    
    # Convert grid coordinates to real-world position
    x = origin.position.x + x_index * resolution
    y = origin.position.y + y_index * resolution
    
    # Check costmap data to ensure it's valid (free space)
    costmap_index = y_index * width + x_index

    if costmap.data[costmap_index] == 0:
        print(f"x: {x},y: {y}")
        return (x, y, 0)
        
    
    return random_pos(bot,randomization_node)
  
def reset_pos2(prims,randomization_node):
    i = 0
    for prim in prims:
        pos = random_pos(robots[i],randomization_node)
        if pos != None:
            x,y,z = pos
            print(f"loc of {robots[i]}: {x,y}")
            rand_loc_gf = Gf.Vec3f(x, y, 0)
            prim.GetAttribute("xformOp:translate").Set(rand_loc_gf)
            i += 1


parser = argparse.ArgumentParser()
parser.add_argument(
    "--environment",
    type=str,
    choices=["hospital", "office"],
    default="hospital",
    help="Choice of navigation environment.",
)

args, _ = parser.parse_known_args()

simulation_app.update()


print("Running custom multi-robot scenario")
HOSPITAL_USD_PATH = ENVS_PATH + "/hospital/hospital.usd"
OFFICE_USD_PATH = ENVS_PATH + "/office/office.usd"

num = 1
# num = random.randint(1, 2)

# if num == 1:
#     usd_path = HOSPITAL_USD_PATH
# elif num == 2:
#     usd_path = OFFICE_USD_PATH

if args.environment == "hospital":
    usd_path = HOSPITAL_USD_PATH
elif args.environment == "office":
    usd_path = OFFICE_USD_PATH


omni.usd.get_context().open_stage(usd_path, None)
world = World(stage_units_in_meters=1.0, physics_prim_path="/physicsScene", backend="numpy")
stage = omni.usd.get_context().get_stage()


#robots
robot = stage.GetPrimAtPath("/World/Nova_Carter_ROS_1")
robot1 = stage.GetPrimAtPath("/World/Nova_Carter_ROS_2")
robot2 = stage.GetPrimAtPath("/World/Nova_Carter_ROS_3")
prims = [robot,robot1, robot2]

world.reset()


rclpy.init()
minimal_subscriber = Subscriber()
random_pos_node = RandomPositionNode()

# Wait two frames so that stage starts loading
simulation_app.update()
simulation_app.update()

print("Loading stage...")
from omni.isaac.core.utils.stage import is_stage_loading

while is_stage_loading():
    simulation_app.update()
print("Loading Complete")

simulation_context = SimulationContext(stage_units_in_meters=1.0)

simulation_app.update()
simulation_context.play()
simulation_app.update()

sys.path.append(REPO_PATH + "/src/launch")
from carter_navigation import *

def start(launch_description: LaunchDescription):
    stop_event = multiprocessing.Event()
    process = multiprocessing.Process(target=run_process, args=(stop_event, launch_description), daemon=True)
    process.start()

def shutdown():
    stop_event.set()
    process.join()

def run_process(stop_event, launch_description):
    loop = asyncio.get_event_loop()
    launch_service = LaunchService()
    launch_service.include_launch_description(launch_description)
    launch_task = loop.create_task(launch_service.run_async())
    loop.run_until_complete(loop.run_in_executor(None, stop_event.wait))
    if not launch_task.done():
        asyncio.ensure_future(launch_service.shutdown(), loop=loop)
        loop.run_until_complete(launch_task)


launch_description = generate_launch_description(num)
start(launch_description)


while simulation_app.is_running():
    simulation_app.update()
    if world.is_playing():
        if rclpy.ok():
            rclpy.spin_once(minimal_subscriber, timeout_sec=0.001)
        if minimal_subscriber.reset_env:
            print("reset enviorment")
            minimal_subscriber.set_reset_val(False)
            minimal_subscriber.clearGlobalCostmap()
            minimal_subscriber.clearLocalCostmap()
            reset_pos2(prims,random_pos_node)
         

minimal_subscriber.destroy_node()
rclpy.shutdown()

simulation_context.stop()
simulation_app.close()

Here is the code I have. Sorry it’s long I wasn’t exactly sure what to trim down. The two nodes are for resetting the environment and getting the cost map. I’m trying to reset the position by checking whether the point is in the costmap. This is most of my code except for my launch file but that’s very similar to the one given in the example so I don’t think there should by anything wrong with it.

Hello!
I made a few additions to my code trying to fix the problem. I noticed that if I put a 2D pose estimate on the robot before send a nav goal the robot seems to navigate correctly. However when I tried sending the position to the initalpose topic, Rviz seemed to be one randomization behind Isaac Sim. I added markers to the code to show where on the rviz map the robots should be.

class RandomPositionNode(Node):
    def __init__(self):
        super().__init__('random_position_node')
        self.costmap_subscriptions = []
        self.costmaps = dict()
        
        self.pose_publisher = dict()
        
        for robot in robots:
            topic_costmap = '/' + robot + '/global_costmap/costmap'
            self.get_logger().info(f"Subscribing to {topic_costmap}")
            self.costmap_subscriptions.append(self.create_subscription(
                OccupancyGrid,
                topic_costmap,  
                # lambda msg: self.costmap_callback(msg, robot),
                partial(self.costmap_callback, robot=robot),
                10
            ))
            self.costmaps[robot] = None
            
            topic_pose = '/' + robot + '/initialpose'
            self.pose_publisher[robot] = self.create_publisher(PoseWithCovarianceStamped, topic_pose, 10)
            
            
    def publish_pose(self,x,y,bot):
        msg = PoseWithCovarianceStamped()
        msg.header.frame_id = "map"
        msg.header.stamp.sec = 1741657725
        
        msg.pose.pose.position.x = x
        msg.pose.pose.position.y = y
        msg.pose.pose.position.z = 0.0
        
        msg.pose.pose.orientation.x = 0.0
        msg.pose.pose.orientation.y = 0.0
        msg.pose.pose.orientation.z = 0.0
        msg.pose.pose.orientation.w = 1.0
        
        msg.pose.covariance = [
            0.25,0.0,0.0,0.0,0.0,0.0,
            0.0,0.25,0.0,0.0,0.0,0.0,
            0.0,0.0,0.0,0.0,0.0,0.0,
            0.0,0.0,0.0,0.0,0.0,0.0,
            0.0,0.0,0.0,0.0,0.0,0.0,
            0.0,0.0,0.0,0.0,0.0,0.0685389
        ]
        
        self.pose_publisher[bot].publish(msg)
        self.get_logger().info('publishing for "%s": "%s", "%s"' % (bot, msg.pose.pose.position.x, msg.pose.pose.position.y))
        
    def make_marker(self,x,y,bot):
            self.marker_pub = self.create_publisher(Marker, f"{bot}/Marker", 10)
            marker = Marker()
            marker.header.frame_id = "/map"
            # set shape, Arrow: 0; Cube: 1 ; Sphere: 2 ; Cylinder: 3
            marker.type = 2
            marker.id = 0

            # Set the scale of the marker
            marker.scale.x = 1.0
            marker.scale.y = 1.0
            marker.scale.z = 1.0

            # Set the color
            if bot == "carter1":
                marker.color.r = 0.0
                marker.color.g = 1.0
                marker.color.b = 0.0
                marker.color.a = 1.0
                
            if bot == "carter2":
                marker.color.r = 1.0
                marker.color.g = 0.0
                marker.color.b = 0.0
                marker.color.a = 1.0
                
            if bot == "carter3":
                marker.color.r = 0.0
                marker.color.g = 0.0
                marker.color.b = 1.0
                marker.color.a = 1.0

            # Set the pose of the marker
            marker.pose.position.x = x
            marker.pose.position.y = y
            marker.pose.position.z = 0.0
            marker.pose.orientation.x = 0.0
            marker.pose.orientation.y = 0.0
            marker.pose.orientation.z = 0.0
            marker.pose.orientation.w = 1.0

            self.marker_pub.publish(marker)


def reset_pos2(prims,randomization_node):
    i=0
    
    for prim in prims:
        pos = random_pos(robots[i],randomization_node)
        if pos != None:
            x,y,z = pos
            print(f"loc of {robots[i]}: {x,y}")
            rand_loc_gf = Gf.Vec3f(x, y, 0)
            randomization_node.make_marker(x,y,robots[i])
            prim.GetAttribute("xformOp:translate").Set(rand_loc_gf)
            time.sleep(1)
            rclpy.spin_once(randomization_node, timeout_sec=1.0)
            randomization_node.publish_pose(x,y,robots[i])
        i+= 1

I added my updates randomposition node and reset_pos2 function

Here’s a screenshot of the robot with the marker in Isaac Sim and Rviz. The green marker is where the robot is in isaac sim. As you can see the robot is in the wrong place in rviz but the particle swarm is right. If I randomize the robot again the robot in Rviz will go to the green marker but the marker and robot and Isaac Sim will go to a new position.

Hey @eminor2001! I am not able to launch the script you shared. It looks like it requires a repository frontiers_rl. What is that? Could you please share it as well? Could you please provide detailed steps to replicate the issue?

I noticed that you are using Isaac Sim 4.0.0. Have you tried with Isaac Sim 4.5.0?

According to your script, the random pose is generated based on costmap. What is the frame of the costmap?

Other things that you can check is the TF tree. You can run ros2 run tf2_tools view_frames in terminal to save a TF tree of your system.

You can also run ros2 run tf2_ros tf2_echo [source_frame] [target_frame] to check the TF between the map and the robot pose. Check if the TF aligns with the robot pose.

Hello,
I’ve checked and it doesn’t work with Isaac sim 4.5.0. I changed the file path to work with the sample environment instead of the one in my “frontier_rl” folder. The environments are identical except for one object so it should run the same. You should be able to run this code with the my code with multiple_robot_carter_navigation_hospital.launch.py and get the same results. Here is my updated code.

import argparse
import sys
import os
import omni
import carb
import numpy as np
import rclpy
from rclpy.node import Node
from std_msgs.msg import Bool
from nav2_msgs.srv import ClearEntireCostmap
import random
from launch import LaunchService
import asyncio
import multiprocessing
from functools import partial

from nav_msgs.msg import OccupancyGrid
import time
import carb
from isaacsim import SimulationApp

CONFIG = {"renderer": "RayTracedLighting", "headless": False}

# Example ROS2 bridge sample demonstrating the manual loading of Multiple Robot Navigation scenario
simulation_app = SimulationApp(CONFIG)
import omni
from omni.isaac.core import SimulationContext, World
from omni.isaac.core.utils import extensions, nucleus, prims, rotations, stage, viewports
from omni.isaac.core.utils.extensions import enable_extension
from pxr import Sdf
from omni.isaac.core.utils.prims import define_prim, get_prim_at_path
from omni.isaac.core.utils.stage import add_reference_to_stage, get_current_stage
from omni.isaac.nucleus import get_assets_root_path
from omni.isaac.core.utils.extensions import enable_extension
from pxr import Gf, PhysxSchema, UsdGeom, UsdLux, UsdPhysics

# enable ROS2 bridge extension
enable_extension("omni.isaac.ros2_bridge")


robots = ["carter1","carter2","carter3"]         #TODO: make this configurable not hardcoded

class Subscriber(Node):
    def __init__(self):
       
        
        print("init minimal subscriber")
        super().__init__('minimal_subscriber')

        self.clear_costmap_local_srvs = dict()
        self.clear_costmap_global_srvs = dict()

        for robot in robots:
            self.clear_costmap_global_srvs[robot] = self.create_client(
                ClearEntireCostmap, robot+'/global_costmap/clear_entirely_global_costmap'
            )
            self.clear_costmap_local_srvs[robot] = self.create_client(
                ClearEntireCostmap, robot+'/local_costmap/clear_entirely_local_costmap'
            )
        self.ros_sub = self.create_subscription(Bool, "reset", self.reset_issac_sim_callback, 10)
        self.reset_env = False

        
    def reset_issac_sim_callback(self,msg):
        print(f"self.reset_env: {self.reset_env}")
        self.reset_env = msg.data
        print(f"self.reset_env changed: {self.reset_env}")

    def set_reset_val(self,val):
        self.reset_env = val
        print(f"reset_env set to {self.reset_env}")
        
    def clearGlobalCostmap(self):
        """Clear global costmaps."""
        for robot in robots:
            srv = self.clear_costmap_global_srvs[robot]
            while not srv.wait_for_service(timeout_sec=0.001):
                print(f'Clear global costmaps service for robot {robot} not available, waiting...')
            req = ClearEntireCostmap.Request()
            future = srv.call_async(req)
            rclpy.spin_until_future_complete(self, future)
        return
    
    def clearLocalCostmap(self):
        """Clear local costmaps."""
        for robot in robots:
            srv = self.clear_costmap_local_srvs[robot]
            while not srv.wait_for_service(timeout_sec=0.001):
                print(f'Clear local costmaps service for robot {robot} not available, waiting...')
            req = ClearEntireCostmap.Request()
            future = srv.call_async(req)
            rclpy.spin_until_future_complete(self, future)
        return
    
    
class RandomPositionNode(Node):
    def __init__(self):
        super().__init__('random_position_node')
        self.costmap_subscriptions = []
        self.costmaps = dict()
        
        for robot in robots:
            topic = '/' + robot + '/global_costmap/costmap'
            self.get_logger().info(f"Subscribing to {topic}")
            self.costmap_subscriptions.append(self.create_subscription(
                OccupancyGrid,
                topic,  
                # lambda msg: self.costmap_callback(msg, robot),
                partial(self.costmap_callback, robot=robot),
                10
            ))
            self.costmaps[robot] = None
        
    def costmap_callback(self, msg, robot):
        """
        Callback function to update the costmap once it's received.
        """
        self.costmaps[robot] = msg
        print(f"costmap: {robot}")
        self.get_logger().info(f"msg: {msg.header}")
        
        
    
    def get_costmap(self,robot):
        """Returns the latest received costmap."""
        return self.costmaps[robot]
     
    
    def get_costmaps(self):
        """Returns the dict latest received costmap."""
        return self.costmaps

def random_pos(bot, randomization_node):
    """
    Generates a random valid position within the costmap.
    """
    rclpy.spin_once(randomization_node, timeout_sec=1.0)
    costmap = randomization_node.get_costmap(bot)
    print(f"{bot}: Checking costmaps")
    
    if costmap == None:
        print(f"{bot}: No Costmap :(")
        return None
  
    # Costmap properties
    width = costmap.info.width
    height = costmap.info.height
    resolution = costmap.info.resolution
    origin = costmap.info.origin
    
    # Generate random coordinates in the grid
    x_index = random.randint(0, (width - 1))
    y_index = random.randint(0, (height - 1))
    
    # Convert grid coordinates to real-world position
    x = origin.position.x + x_index * resolution
    y = origin.position.y + y_index * resolution
    
    # Check costmap data to ensure it's valid (free space)
    costmap_index = y_index * width + x_index

    if costmap.data[costmap_index] == 0:
        print(f"x: {x},y: {y}")
        return (x, y, 0)
    
    return random_pos(bot,randomization_node)
  
def reset_pos2(prims,randomization_node):
    i=0
    for prim in prims:
        robot = robots[i]
        pos = random_pos(robot,randomization_node)
        if pos != None:
            x,y,z = pos
            print(f"loc of {robot}: {x,y}")
            rand_loc_gf = Gf.Vec3f(x, y, 0)
            prim.GetAttribute("xformOp:translate").Set(rand_loc_gf)
        i+=1


parser = argparse.ArgumentParser()
parser.add_argument(
    "--environment",
    type=str,
    choices=["hospital", "office"],
    default="hospital",
    help="Choice of navigation environment.",
)

args, _ = parser.parse_known_args()

simulation_app.update()


print("Running custom multi-robot scenario")


HOSPITAL_USD_PATH = "/Isaac/Samples/ROS2/Scenario/multiple_robot_carter_hospital_navigation.usd"
OFFICE_USD_PATH = "/Isaac/Samples/ROS2/Scenario/multiple_robot_carter_office_navigation.usd"

num = 1
# num = random.randint(1, 2)

# if num == 1:
#     usd_path = HOSPITAL_USD_PATH
# elif num == 2:
#     usd_path = OFFICE_USD_PATH

if args.environment == "hospital":
    usd_path = HOSPITAL_USD_PATH
elif args.environment == "office":
    usd_path = OFFICE_USD_PATH


omni.usd.get_context().open_stage(usd_path, None)
world = World(stage_units_in_meters=1.0, physics_prim_path="/physicsScene", backend="numpy")
stage = omni.usd.get_context().get_stage()


#robots
robot = stage.GetPrimAtPath("/World/Nova_Carter_ROS_1")
robot1 = stage.GetPrimAtPath("/World/Nova_Carter_ROS_2")
robot2 = stage.GetPrimAtPath("/World/Nova_Carter_ROS_3")
prims = [robot,robot1, robot2]

world.reset()


rclpy.init()
minimal_subscriber = Subscriber()
random_pos_node = RandomPositionNode()

# Wait two frames so that stage starts loading
simulation_app.update()
simulation_app.update()

print("Loading stage...")
from omni.isaac.core.utils.stage import is_stage_loading

while is_stage_loading():
    simulation_app.update()
print("Loading Complete")

simulation_context = SimulationContext(stage_units_in_meters=1.0)

simulation_app.update()
simulation_context.play()
simulation_app.update()



while simulation_app.is_running():
    simulation_app.update()
    if world.is_playing():
        if rclpy.ok():
            rclpy.spin_once(minimal_subscriber, timeout_sec=0.001)
        if minimal_subscriber.reset_env:
            print("reset enviorment")
            minimal_subscriber.set_reset_val(False)
            minimal_subscriber.clearGlobalCostmap()
            minimal_subscriber.clearLocalCostmap()
            reset_pos2(prims,random_pos_node)
         

minimal_subscriber.destroy_node()
rclpy.shutdown()

simulation_context.stop()
simulation_app.close()

Hi @eminor2001! Could you also please provide the node that reads in map data and publishes global costmap (if this is how you set it up)? I believe there should also be a node that publishes TFs.

Hi,
Isaac_Forum.zip (15.4 KB)

Above is all the code I’ve written. The subscriber node clears the costmap when I send a reset signal and RandomPositionNode reads the costmap data to find a position in the costmap. I’m not sure about a node that published TFs. I’ve attached zip folder with all the code, launch file, hospital enviorment, and a readme

Thanks @eminor2001! I am able to run your script and observe the issue.

It looks like you are running amcl for each carter robot to localize them and it seems that amcl is not converging properly.
Rviz is visualizing the robot pose based on amcl result.

If you check the robot position from Isaac Sim and amcl_pose, they are the same when you first bring up the scene. But they would be different when you reset the robot poses.

Take carter1 for example. Here is when I just bring up the scene. The Isaac Sim pose aligns with amcl pose.

This is after I reset pose. The Isaac Sim pose doesn’t align with amcl pose

I think you might want to dig into amcl node. Here is something to check

  • When resetting the robot pose, what is the default covariance? Is it too big?
  • What triggers re-initialization of amcl node and if amcl node receives your random pose? You might want to check here
  • Does you script send random pose to re-initialize amcl? I might overlooked something but I don’t see where you send random pose to amcl

Hello,
I apparently wasn’t sending a pose to amcl so that’s definitely part of the issue. I added one but it didn’t seem to fix the problem. I also tried messing with the covariance and tried resetting both the amcl node and the nav2 stack but that didn’t work either. I’m not sure why the pose doesn’t align but I’m going to keep tried to debug that. I added the amcl publisher to my position node and but the new file below.
robot2.py.zip (3.6 KB)