Custom writer with built-in GT and camera pose

Hi there,

We want to make a custom writer for our scene, including built-in GT such as rgb/bbox_2d_tight/depth/semantic_segmentation as well as the camera 6D pose.

We tested the code for obtaining camera pose in Script Editor and also generated some GT with built-in BasicWriter in GUI. They both work well.

Now we want to wrap it up together as an end-to-end standalone python script. We are wondering what would be the best way to do that? “Best”, say, the most efficient way for generating GT (Render as RTX Path-tracing) & camera pose in large-scale in-door scene with multiple assets as well as some randomization.

Considering the I/O, it seems that omni.replicator.core.BackendDispatch would be a good choice? Our machine is RTX A6000*2.

Could you give some suggestions or advices? Thanks!

Best,
Rosy

Hi there,

to avoid reading the 6d pose from stage and feeding it into the writer you could try using the cameraViewTransform (camera to world transformation matrix) data from the CameraParameters annotator directly in the writer.

Here are some example questions using this parameter:

Let me know if you need further assistance

Best

@ahaidu How can I use the ObjectronWriter in IsaacSim 2023.1.1 enterprise to feed the data to NVIDIA TAO CenterPose?

I was able to train the NVIDIA TAO CenterPose on Google Objectron bike dataset.

@andrew_g

Thank you. I have access to this however it doesn’t write the AR_data into json and the validation phase in Nvidia TAO centerpose requires it. Are you able to add AR_data into annotations too?

Hi @mona.jalal ,

For training, you don’t need access to the AR_data field. The AR_data field is used in the evaluation from objectron but there are some ways to get around this. For example, if you use your own 3D IOU functions, you don’t need AR_data.

We currently don’t have a timeline for adding this field into the annotation files.

1 Like

@andrew_g

If I don’t do the valuation phase during training, would that make any sense at all with NVIDIA TAO CenterPose?

Also, I was able to train (only 4.7K data points of multiple instances of the same object of interest in each frame) but the inference results are not as good. Is that because validation phase was done on bike data (or set to be done every 140 epochs when also my number of epochs was as well 140?) or just because my dataset was small? I am having some problem with visibility threshold (even though I have set it to 0.6 it still sometimes creates frames with 0.1 or much less visibility of object of interest) and have tried multiple resolutions and was not able to fix it. I am in talk with NVIDIA enterprise team about this and once it is fixed I will create a 20K dataset.

Also, could you please help me how to change the the 3D IoU function that is part of NVIDIA TAO CenterPose? I am doing the training using a docker and run this command

print("For multi-GPU, change num_gpus in train.yaml based on your machine or pass --gpus to the cli.")
# If you face out of memory issue, you may reduce the batch size in the spec file by passing dataset. batch_size=2
!tao model centerpose train \
          -e $SPECS_DIR/train.yaml \
          results_dir=$RESULTS_DIR/

Hi Mona, the model performance is not affecting by the validation set. It would be great to try a better training dataset. What do you mean by “creates frames with 0.1”? The number shown on the top left of the object is the dimension/scale, instead of score.

1 Like

@jianhey Enterprise Isaac Sim is helping me with visibility threshold issue. Thank you.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.