Invalid rotations from offline_pose_generation.py

I am observing an error in rotation when running the offline pose demo script on objects that aren’t from YCB-Video.

The tutorial I am following is here:
https://docs.omniverse.nvidia.com/app_isaacsim/app_isaacsim/tutorial_replicator_offline_pose_estimation.html#

By visualizing the object axis and bounding cuboid, I can see that the pose of the cracker box object is valid:

However, when I try this on a simple cube object, I observe the following:

The translation appears correct, but not the rotation.

My workflow is to export to .usda from blender. Here are the .blend and .usda files:
cube.blend (929.9 KB)
cube.usda (8.6 KB)

I’m testing on a cube with the XYZ faces colored red, green, and blue, respectively, to help with debugging.

Here are my scripts for data generation and visualization:
offline_pose_generation_aja9675.zip (372.6 KB)
I only made a few changes to offline_pose_generation.py for the new object path and config file.

Reproducing is very easy:
export ISAAC_ROOT=/isaac_sim-2022.2.0

Generate cracker box images:
$ISAAC_ROOT/python.sh ./offline_pose_generation.py --num_mesh 0 --num_dome 25 --writer ycbvideo --output_folder ./test_data/cracker_box
Visualize results:
./check_pose.py cracker_box

Generate cube images:
$ISAAC_ROOT/python.sh ./offline_pose_cube.py --num_mesh 0 --num_dome 25 --writer ycbvideo --output_folder ./test_data/cube
Visualize results:
./check_pose.py cube

I also observe the error when using a cube that was created in IsaacSim, so I do not believe it is an issue with the Blender->usd export.

@ahaidu I’m really blocked on this. Any help would be much appreciated.

Observing the incorrect result on another Omniverse provided USD asset:
omniverse://localhost/NVIDIA/Assets/Isaac/2022.2.0/Isaac/Props/Rubiks_Cube/rubiks_cube.usd

Ex.

It seems that only the YCB-V objects work.

Hi @WendyGram, could you help me out by forwarding this to the dev team? I’m working on a demo for GTC but really stuck on this issue.

Hi @aja9675, could you try running the script with --writer dope to see if the issue is recreated?
I tested this with the DOPE writer out of curiosity and was not able to recreate the issue. Wondering if this is something wrong with the YCB writer specifically.

Hi @andrew_g, thanks for looking into this.

Interestingly, the projected vertices, called “projected_cuboid”, in the DOPE ground truth are correct, but the rotation is still not:


Note how the green dots (projected vertices), line up on the cube, but the axis do not.

That said, I found the error in this case. The rotation in the DOPE ground truth is reported as “quaternion_xyzw” (scalar last format). However, it appears it’s actually in scalar first format (wxyz). If I reorder the elements, I can observe a good result from the DOPE writer:

I’m trying to apply this same fix to the ycbvideo writer, but haven’t had any luck yet. This may be because we get a 3x3 rotation matrix instead of a quaternion from the ycbvideo writer. Converting this to a quaternion and then re-ordering is not working.

I found that I can hack OgnPose.py and ycb_video_writer.py to make it more like the DOPE writer and get closer:

Here is my patch:
rotation_hack.patch (4.3 KB)

Interestingly, there is now a slight translation error that is not consistent. In the previous image, it was in the -X (red) direction. A different sample shows it in the +X direction:

Depth and semantic information is information is important for my application, so I need to use the YCB writer. I can’t just use the DOPE writer and fix the quaternion.

The remaining translation error in this case seems to be caused by the optical center.

Changing
C_X: 637.642
C_Y: 367.56
to
C_X: 640
C_Y: 360
fixes the error.

It appears when we create a camera with rep.create.camera(), there is no way to specify the optical center for a pinhole camera model. We can only do it for a fisheye camera model. Since this is different than the main issue I’m facing, I will create a separate issue for this.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.