How to get point cloud, render and view it?

Autonomous Machines Robotics - Isaac Isaac Gym
#1

Hi,
Thank you for your work on Issac Gym. It’s impressive and excellent.

I want to ask questions about point clouds. I have noticed some APIs that are helpful to get point cloud, but can you explain more detailed steps? Are there any relevant examples?
In addition, how to render and view the point cloud in the simulation environment after obtaining it.

Thank you very much.

#2

We have an internal example of converting depth cam data to point clouds. Would this be useful? It is quite an old example and doesn’t leverage the new tensor API or full GPU acceleration, which could be a big benefit. I can still dig it up for you if you think it would be useful.

#3

This is what that example looks like. It creates point clouds from multiple depth cameras and renders them in an external viewer (pptk):

depth_cam_to_point_cloud
depth_cam_to_point_cloud1674×930 356 KB

#4

Hello is there a way , we can use inbuilt point cloud provided in ZED 2 SDK also spatial mapping module , which comes with ZED 2 , can a custom codelet be made in C++ or Python ? are the message API in Isaac SDk sufficient to display to Nvidia WebSight . We are trying to display the spatial Imaging data from Zed 2 to Isaac SDK , using a custom C++ Codelet .

#5

@lwawrzyniak
Could you extend on how the example is implemented, please?
Is there source code on github?

#6

Hi @Ish,

Note that this is for Isaac Gym only, which is not yet integrated with Isaac Sim or Isaac SDK - that will happen some time next year. Isaac Gym is intended as a preview of APIs for end to end GPU reinforcement learning.

Take care,
-Gav

#7

@_av and @AshWang

A GitHub GIST with this example is here: depth_cams_to_point_cloud.py · GitHub

Note that there are some issues with pptk that require some workarounds for libz described here: pip install pptk doesn't work on ubuntu 18.04/kubuntu 18.04 · Issue #3 · heremaps/pptk · GitHub / pptk.view() hangs without showing debug/info/error information · Issue #4 · heremaps/pptk · GitHub

Take care,
-Gav

#8

@gstate
Thank you very much.
trying to settle pptk on jetson with Jetson Nano に pptk をインストールする - Qiita
gym wont work on Jetson, wil it? Host PC only?

#9

Yes, Gym is currently available only for Linux x64 systems.

Take care,
-Gav

#10

This looks good, I can try it, thanks!
But, How will using the new tensor API affect the simulation results?

#11

@gstate
Could you extend how to run the example, please?
After getting pptk to work I still seem missing something PVD

python3 depth_cams_to_point_cloud.py 
Importing module 'gym_36' (/home/nvidia/dev/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_36.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/nvidia/dev/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
Not connected to PVD
Segmentation fault

hm, gym examples won’t work anymore terminating with the same error :/
Probably I shall post it as a separate issue?

#12

If your Gym examples worked before you altered your conda environment with the libz symlink needed for pptk, you may want to delete and recreate your conda environment. Sounds like something else went wrong.

For troubleshooting in general, make sure that you have a working vulkan environment by running vulkaninfo - that’s often the most common cause for unusual issues.

Take care,
-Gav

#13

@gstate
Thank you.
Running the vulkaninfo revealed broken GPU driver, thus the cause got determined.
reinstalling the cuda seems resolved the issue.
the depth pointcloud example works by now.
Thank you very much

#15

I also had the error: Not connected to PVD but I was able to resolve it by moving the pptk import statement below the isaacgym import.