Hi NVIDIA,
I am extending your point cloud code to a scenario where I have multiple environments running in parallel. Even though I used the exact same set up for each environment, the generated point clouds are not the same.
To be exact, the point clouds of the first env and second environments are 2 meters apart (see image below for visualization in open3D and pptk). I also printed out the points matrices and confirmed that they’re different.
I realized that this line of your code prints out different matrices for different environments. vinv = np.linalg.inv(np.matrix(gym.get_camera_view_matrix(sim, env, cam_handles[c]))) Shouldn’t the camera view matrix be the same for all environments, given that the setups are the same? I believe that this is the main cause of the issue
Please see below for my code which creates two separate point clouds for two environments. Please advise me on a good way to generate point clouds in multiple environments.
test_point_cloud_multiple_scences.py (8.8 KB)
Thanks a lot,
Bradley
2 Likes
Thanks for the report @yuan.truyenbao, I’ll take a look into this issue.
Hi @vmakoviychuk,
Thanks for your help! Did you have a chance to look into this issue and come up with a solution to fix it?
Have a great day,
Bradley
Hi, I’m responding to this thread because I have a similar issue. View matrices for cameras in different environments are different even when the environments are set up similarly. I think this may be because the view matrix returned by Gym.get_camera_view_matrix is in global space instead of env space.
Will this be fixed? I imagine for all use cases it would be better for the view matrix to be in env space. It would be best if the coordinate space could be configured with a parameter, and at very least the documentation should explain why the view matrices can be different in different environments.
Thank you.
I have same exact problem when using get_camera_view_matrix. To reproduce this problem one can look at multiple_camera_envs.py in isaac_gym python examples. Running that code we get:
Added cam0 with handle 0 in env 0 | View matrix:
[[ 1. 0. -0. 0. ]
[-0. 1. -0. 0. ]
[ 0. 0. 1. 0. ]
[-1.38 -1. 0. 1. ]]
Added cam0 with handle 0 in env 1 | View matrix:
[[ 1. 0. -0. 0. ]
[-0. 1. -0. 0. ]
[ 0. 0. 1. 0. ]
[-3.38 -1. 0. 1. ]]
Added cam1 with handle 1 in env 0 | View matrix:
[[ 1. 0. -0. 0. ]
[-0. 1. -0. 0. ]
[ 0. 0. 1. 0. ]
[-0.5 -9. -0.8 1. ]]
Added cam1 with handle 1 in env 1 | View matrix:
[[ 1. 0. -0. 0. ]
[-0. 1. -0. 0. ]
[ 0. 0. 1. 0. ]
[-2.5 -9. -0.8 1. ]]
View matrix of cam0 with handle 0 in env 0:
[[ 1. 0. -0. 0. ]
[-0. 1. -0. 0. ]
[ 0. 0. 1. 0. ]
[-1.38 -1. 0. 1. ]]
View matrix of cam1 with handle 1 in env 0:
[[ 1. 0. -0. 0. ]
[-0. 1. -0. 0. ]
[ 0. 0. 1. 0. ]
[-0.5 -9. -0.8 1. ]]
View matrix of cam0 with handle 0 in env 1:
[[ 1. 0. -0. 0. ]
[-0. 1. -0. 0. ]
[ 0. 0. 1. 0. ]
[-3.38 -1. 0. 1. ]]
View matrix of cam1 with handle 1 in env 1:
[[ 1. 0. -0. 0. ]
[-0. 1. -0. 0. ]
[ 0. 0. 1. 0. ]
[-2.5 -9. -0.8 1. ]]
Compare view matrix of the second camera (cam1) in the first and second env (env0 and env1). They are not the same. Is there a fix or work around for this? I have a custom setup and my env are exactly the same in terms of the cameras. But I get different view matrices from different envs. Now, one thing to point out here is that the first env is always the correct one. I used the same code for point cloud to get point clouds from cameras. Which would results in a correct point cloud for only the first objects:
mesh and its extracted point cloud from first env:
everything is good but look at the second env. This second env has exact same cameras and object but just because it is in the second place in the list when iterating through all the envs in the sim, its view matrix gets messed up
Noticed in Preview 2 Release, it is mentioned that the view matrix calculation is corrected. I have Preview 4, so the version should not be the source of the problem for me.
OK, here is the solution: as @NoahCGreen mentioned, get_camera_view_matrix() returns the view matrix in global coordinate not the env coordinate. So we need to find the transformation between the global space and each env which can be found like:
env_position = self.gym.get_env_origin(env_handle)
Create homogeneous transformation assuming the orientation of the global and env coordinate system are the same and only the position is different:
env_to_global = np.identity(4)
env_to_global[:3,3] = np.array([env_position.x, env_position.y, env_position.z])
Use this transformation to convert the points from global space to env space before appending them at line 210:
p2_env = np.linalg.inv(env_to_global) @ p2
And things works correctly. This is for z-up simulation
