Isaac sim in headless mode python script help

fairy new to isaac sim,

I have been able to view 2d lidar data in rviz using the ros_bridge what i want to do is to visualize 3d lidar points in rviz.

what i am trying to do is to create a python script that reades the 3d data from omni isaac lidar and make a custom ros publisher

i can run the python code in the isaac sim code editor but as the isaac sim is running in an aws instance it can only be runned in headless mode.
so not able to use the menu to open or save python scripts and as the clipboard is not supported yet its been really tedious to write code and the code getting erased if isaac sim crashes.

i think there is a really simple workaround but not able to find anything helpful on the internet.

would really appreciate any help.
Thansk.

Hi

An alternative to run your code is to develop it as a custom Extension. One of the advantages of this approach is that any modification of the code will be automatically reloaded by the Extension Discovery system. Also, the code will persist, even to possible Isaac Sim crashes.

Read the documentation about extension here.

This post details a little example of a custom extension.

1 Like

Thanks for your help.

Following your answer, I was able to run my python script as an extension and was able to read lidar data from isaac simulation.
just one more thing i am not able to do “import rospy” in my script
as i understand isaac-sim is using its own python installation
i have tried to add ros to python path using sys.path(“path to ros installlation”) but still not able to get it to work.

Thansk.

Hi,
I’m glad to hear about that :)

To use rospy you need to add some specifics python2.7 paths to sys.path…
Fortunately, there is a python package that made all the hard work for you

pyros-setup: “Toolsuite for running ROS environments directly from python code, without any specific requirements outside of usual python”

  1. You need to install it for Isaac Sim’s python

    /isaac-sim/_build/target-deps/kit_sdk_release/_build/target-deps/python/bin/python3 -m pip install pyros-setup
    

    Note: maybe you will need to install rospkg

    /isaac-sim/_build/target-deps/kit_sdk_release/_build/target-deps/python/bin/python3 -m pip install rospkg
    
  2. Then, you can import rospy on your code as follow:

    import pyros_setup
    pyros_setup.configurable_import().configure().activate() # mysetup.cfg from pyros-setup
    
    import rospy
    import std_msgs.msg
    import sensor_msgs.msg
    import geometry_msgs.msg
    
1 Like

You are really a life saver.
Thanks a lot…

one more thing (again) when i can get the lidar data using omni.isaac.lidar: LIDAR Simulation — Omniverse Robotics 2020.2 documentation

i want to convert it to PoinCloud2 format of ros
is there a built in function in omni that can accomplish this? or can you give me some pointer.

Once again really gratefully for all your help

1 Like

Hi

As far as I know, there is not any clear exposed omni’s built-in function to generate PointCloud2 messages… However, I am just an Isaac Sim’s user…

I hope those external links can be useful to you

1 Like

Hey,

really sorry to bother you once again.
following your advice i was able to send data using ros message and view it rviz.
the problem i am having right now is that the point cloud that is been shown is not accurate.

i am using the following code to get x,y,z coordinates using depth, zenith and azimuth data from the isaac lidar. But as you can see in the attached screen shots it is rotated 90 degree and also curved along the sides.

    depth = np.array(lidarInterface.get_depth_data(lidarPath), dtype=np.float32)
    zenith = np.array(lidarInterface.get_zenith_data(lidarPath), dtype=np.float32)
    azimuth = np.array(lidarInterface.get_azimuth_data(lidarPath), dtype=np.float32)


    theta = zenith
    phi = azimuth[:, np.newaxis]

    x_points = depth * np.sin(theta) * np.cos(phi)
    x_points = x_points.flatten()

    y_points = depth * np.sin(theta) * np.sin(phi)
    y_points = y_points.flatten()

    z_points = depth * np.cos(theta)
    z_points = z_points.flatten()

    cloud_points = np.stack([x_points, y_points, z_points], axis=1)


can you please help me with this, where might i have gone wrong.?

Once again really grateful for all your help.

Hi @wakasu

It seems to be the zenith data provided by lidar is respect to XY-plane and not Z-axis, Then, taking this into account you can use the next equations to generate the 3D point cloud

x = depth * sin(π/2 - zenith) * cos(azimuth)
y = depth * sin(π/2 - zenith) * sin(azimuth)
z = depth * cos(π/2 - zenith)

Also, you can verify them using the next code that generate the data shown in the figure

from omni.isaac.lidar import _lidar
from omni.add_on.visualizer import _visualizer
import numpy as np

# lidar interface
lidarInterface = _lidar.acquire_lidar_interface()
lidarPath = "/World/Lidar"

# get lidar data
depth = lidarInterface.get_depth_data(lidarPath) / 65535
zenith = np.pi / 2 - lidarInterface.get_zenith_data(lidarPath)
azimuth = lidarInterface.get_azimuth_data(lidarPath)[:,np.newaxis]

# compute x,y,z
x = (depth * np.sin(zenith) * np.cos(azimuth)).flatten()
y = (depth * np.sin(zenith) * np.sin(azimuth)).flatten()
z = (depth * np.cos(zenith)).flatten()

# get valid x,y,z (where lidar line traces interact with the objects)
indexes = np.where(depth.flatten() < 1)[0]

x = x[indexes]
y = y[indexes]
z = z[indexes]

# show point cloud
fxy = _visualizer.figure("XY")
fyz = _visualizer.figure("YZ") 
fxz = _visualizer.figure("XZ")

fxy.scatter(x, y, aspect="equal")
fyz.scatter(y, z, aspect="equal")
fxz.scatter(x, z, aspect="equal")

_visualizer.scatter3d(x, y, z)

1 Like

Thank again.

this worked and am able to get proper point cloud and visualize in rviz.
Nvidia should really put this in documentation or something, @toni.sm you have really been great help.

just one thing i don’t seem to have omni.add_on.visualizer in my installation
by the name of it, i think its an addon? how can i get it

Hi,

omni.add_on.visualizer is an extension I created to generate and show graphic data in an easy way… You can find it on GitHub (Any feedback will be welcome):

Also, I recently posted it here:

1 Like

Wow, great work 👍👍
this will be really helpful.

Thanks alot