Hey, I’m trying to create a simulation model for synthetic data generation in Isaac Sim 2022.2.0. There are a lot of boxes on a pallet with randomized positioning, lighting and materials. I’m getting the corners and the center points of the top from the 3d bounding box annotator in a custom writer with the following code which works for a single bounding box.
import omni.syntheticdata as sd def world_to_image_pinhole(world_points: np.ndarray, camera_params: dict) -> np.ndarray: # Project corners to image space (assumes pinhole camera model) proj_mat = camera_params["cameraProjection"].reshape(4, 4) view_mat = camera_params["cameraViewTransform"].reshape(4, 4) view_proj_mat = np.dot(view_mat, proj_mat) world_points_homo = np.pad(world_points, ((0, 0), (0, 1)), constant_values=1.0) tf_points = np.dot(world_points_homo, view_proj_mat) tf_points = tf_points / (tf_points[..., -1:]) return 0.5 * (tf_points[..., :2] + 1) def write(self, data: dict): render_product = [k for k in data.keys() if k.startswith("rp_")] bbox3ds = data["bounding_box_3d"]["data"] corners_3d = sd.get_bbox_3d_corners(bbox3ds) corners_3d = corners_3d.reshape(-1,3) points = [corners_3d.tolist().pop(i) for i in range(4,8)] points.append(center_point(*points)) corners_2d = world_to_image_pinhole(points, data["camera_params"]) # contains top corners and center point corners_2d *= np.array([[self.width, self.height]])
Is there a way to also get the visibility/occlusion of these points to see if they are hidden behind other prims? I already tried different annotators (point cloud, occlusion, bounding box 2d), but they show all points/objects in the scene. I would need like a list with all the points that are visible to the camera or check in some way if the points are hidden/visible. Is that possible?
Thanks for any help!