[KeyError: 'AR_data'] Training CenterPose with custom synthetic dataset from Omniverse Replicator

[thanhly]

Hi,

I am trying to train the CenterPose model on a custom dataset using this provided example TAO CenterPose notebook (tao_tutorials/notebooks/tao_launcher_starter_kit/centerpose/centerpose.ipynb at main · NVIDIA/tao_tutorials · GitHub).

Here are the steps I did before training:

1. I created my custom dataset using the sample given file ./python.sh standalone_examples/replicator/pose_generation/pose_generation.py (Pose Estimation Synthetic Data Generation — Omniverse IsaacSim latest documentation). I have my own .usd files of the assets. This step was successful.
2. I made sure the dataset structure followed this reference: Data Annotation Format - NVIDIA Docs
3. I set up the TAO toolkit following this article: TAO Toolkit Quick Start Guide - NVIDIA Docs
4. I was able to run the notebook to train a model using Google Objectron dataset for the cereal box. This was successful.
5. I tried to train the model using the same notebook, but on my custom dataset created in step 1. However, there was an error: KeyError: 'AR_data'.

Questions:

1. How can I annotate AR_data to the .json files when I create my custom datasets using pose_generation.py? So far, I have not found a way to annotate AR_data, please advise.
2. OR, is there a way I can modify the notebook such that AR_data is not required?
3. OR, is there another away I can train CenterPose using my custom dataset?

This is an example of my dataset:

Corresponding .json file:

{
  "camera_data": {
    "intrinsics": {
      "fx": 3199.9999539287774,
      "fy": 3199.9999539287774,
      "cx": 960.0,
      "cy": 600.0
    },
    "camera_view_matrix": [
      [
        1.0,
        0.0,
        0.0,
        0.0
      ],
      [
        0.0,
        -1.0,
        -0.0,
        0.0
      ],
      [
        0.0,
        0.0,
        -1.0,
        0.0
      ],
      [
        -0.0,
        0.0,
        0.0,
        1.0
      ]
    ],
    "camera_projection_matrix": [
      [
        3.33333,
        0.0,
        0.0,
        0.0
      ],
      [
        0.0,
        5.33333,
        0.0,
        0.0
      ],
      [
        0.0,
        0.0,
        0.0,
        -1.0
      ],
      [
        0.0,
        0.0,
        0.01,
        0.0
      ]
    ],
    "width": 1920,
    "height": 1200
  },
  "keypoint_order": [
    "Center",
    "LDB",
    "LDF",
    "LUB",
    "LUF",
    "RDB",
    "RDF",
    "RUB",
    "RUF"
  ],
  "objects": [
    {
      "label": "fork_7500",
      "prim_path": "/World/Fork_7500_0/Fork_7500",
      "visibility": 0.997,
      "location": [
        -3.3173322677612305,
        1.0951167345046997,
        18.979755401611328
      ],
      "quaternion_xyzw": [
        -0.4627035697245575,
        0.5088046860137205,
        0.3824751094322646,
        0.6170380772038369
      ],
      "scale": [
        2.6722641475656896,
        3.3445917221702697,
        0.6275341926171762
      ],
      "keypoints_3d": [
        [
          -1.8345400139741974,
          -0.3419354947783642,
          -19.2678387661581
        ],
        [
          -3.1706720877570422,
          -2.014231355863499,
          -19.581605862466688
        ],
        [
          -3.859232924098995,
          0.3991668015520579,
          -19.445766987961576
        ],
        [
          -0.022099927041445255,
          -1.081748639719478,
          -20.18871318550983
        ],
        [
          -0.7106607633833981,
          1.3316495176960792,
          -20.052874311004715
        ],
        [
          -2.9584192645649967,
          -2.0155205072528077,
          -18.482803221311485
        ],
        [
          -3.6469801009069496,
          0.39787765016274923,
          -18.346964346806374
        ],
        [
          0.19015289615060027,
          -1.0830377911087863,
          -19.089910544354627
        ],
        [
          -0.4984079401913526,
          1.3303603663067705,
          -18.95407166984951
        ]
      ],
      "projected_cuboid": [
        [
          655,
          657
        ],
        [
          442,
          929
        ],
        [
          325,
          534
        ],
        [
          956,
          771
        ],
        [
          847,
          387
        ],
        [
          448,
          949
        ],
        [
          324,
          531
        ],
        [
          992,
          782
        ],
        [
          876,
          375
        ]
      ],
      "truncation_ratio": 0.0
    }
  ]
}

Thank you.

[pcallender]

Moving this to the isaacsim forum, where I think this can be answered better.

[phennings]

Could you please post the full text of the error you get on stdout? or the line of code of this exception? Thanks!

[thanhly]

Hi,
I basically had the same problem as these users: AR_data for custom synthetic data made with Isaac Sim Replicator · Issue #34 · NVIDIA-ISAAC-ROS/isaac_ros_pose_estimation · GitHub

I’m trying to reproduce the error outputs. However, in the meantime, please take a look at the issue on the link provided.

Thanks!

[phennings]

The AR_data field does not affect your training. However, you will have to set it to any number, as in the post you mentioned. It’s a known issue.