Having trouble getting the class id from Cocowriter in the Replicator

Omniverse Synthetic Data Generation (SDG)
1

Hi,

I am trying to generate the synthetic data to coco format by using the following script in the script editor.

It works fine with basicwriter and kittiwriter, but the annotations that cocowriter generates are all shown as others with id 201.

The script is modified from the tutorial script and the attachment is the output annotation json.
Does any one know how to figure it out?

Refer to the replicator api.

Thank you in advanced.

import omni.replicator.core as rep
import os

with rep.new_layer():

    camera = rep.create.camera(position=(0, -10, 1),rotation=(0,0,-90))

    render_product = rep.create.render_product(camera, (1024, 1024))

    torus = rep.create.torus(semantics=[('class', 'torus')] , position=(0, -2 , 1))

    sphere = rep.create.sphere(semantics=[('class', 'sphere')], position=(0, 1, 1))

    cube = rep.create.cube(semantics=[('class', 'cube')],  position=(1, -2 , 1) )

    with rep.trigger.on_frame(max_execs=20, rt_subframes=2):
        with rep.create.group([torus, sphere, cube]):
            rep.modify.pose(
                position=rep.distribution.uniform((-1, -1, -1), (2, 2, 2)),
                scale=rep.distribution.uniform(0.1, 2))

    # Initialize and attach writer 
    writer = rep.WriterRegistry.get("CocoWriter")

    print("***********************************************")
    output_directory = os.getcwd() + "/omni.replicator_out/coco_output"
    print("Final output base directory:", output_directory)
    coco_categories = {
        torus: {
            'name': 'torus',
            'id': 0,
            'supercategory': 'geometry'
        },
        sphere: {
            'name': 'sphere',
            'id': 1,
            'supercategory': 'geometry'
        },
        cube: {
            'name': 'cube',
            'id': 2,
            'supercategory': 'geometry'
        }
    }
    print(f'coco_categories: {coco_categories}')

    writer.initialize(
        output_dir=output_directory,
        rgb=True,
        bounding_box_2d_tight=True,
        semantic_types=["class"],
        coco_categories=coco_categories
    )

    writer.attach([render_product])

    # rep.orchestrator.preview()
    rep.orchestrator.run()

coco_annotations_arejdbfc.txt (18.8 KB)

2

Hi!
Did you manage to figure it out? I’m running into the same problem right now.
Thanks in advance!

3

No. I think they are not taking care of this issue soon even they are advertising this feature which is disappointing.
What I am doing now is to use Kittiwriter then write my own converter.
The hardest part is to deal with the segmentation since Kitti is using mask however COCO is using contour. This you might want to use some OpenCV functions to solve. Or if anyone has better idea please let me know.

4

@wy.chien , @kai.schlauersbach , I must apologize first for both the delay in getting back to you an in the issue you’ve encountered. The issue is one of documentation, which in this case is incomplete and not as clear as it could have been. The key to each field in the coco_categories dictionary should be a string representing the semantic value to match to. It’s also necessary to specify a color field in addition to name and id. We’ll update the API docs accordingly. Thank you so much for bringing this to our attention. Here’s the corrected script below, and please do let us know if you run into any further issues.

import omni.replicator.core as rep
import os

rep.settings.set_stage_up_axis("Z")
rep.settings.set_stage_meters_per_unit(1.0)

with rep.new_layer():

    camera = rep.create.camera(position=(0, -10, 1),rotation=(0,0,-90))

    render_product = rep.create.render_product(camera, (1024, 1024))

    torus = rep.create.torus(semantics=[('class', 'torus')] , position=(0, -2 , 1))

    sphere = rep.create.sphere(semantics=[('class', 'sphere')], position=(0, 1, 1))

    cube = rep.create.cube(semantics=[('class', 'cube')],  position=(1, -2 , 1) )

    with rep.trigger.on_frame(max_execs=20, rt_subframes=2):
        with rep.create.group([torus, sphere, cube]):
            rep.modify.pose(
                position=rep.distribution.uniform((-1, -1, -1), (2, 2, 2)),
                scale=rep.distribution.uniform(0.1, 2))

    # Initialize and attach writer 
    writer = rep.WriterRegistry.get("CocoWriter")

    print("***********************************************")
    output_directory = os.getcwd() + "/omni.replicator_out/coco_output"
    print("Final output base directory:", output_directory)
    coco_categories = {
        "torus": {
            'name': 'torus',
            'id': 0,
            'supercategory': 'geometry',
            'color': (255, 0, 0),
        },
        "sphere": {
            'name': 'sphere',
            'id': 1,
            'supercategory': 'geometry',
            'color': (0, 255, 0),
        },
        "cube": {
            'name': 'cube',
            'id': 2,
            'supercategory': 'geometry',
            'color': (0, 0, 255),
        }
    }
    print(f'coco_categories: {coco_categories}')

    writer.initialize(
        output_dir=output_directory,
        coco_categories=coco_categories
    )

    writer.attach([render_product])

rep.orchestrator.run()
5

Hi @jlafleche , thank you for the update. Now it can export the panoptic segmentation along with the rgb image. But all the class are shown in the same color with count >1. It is actually semantic segmentation.

However, in the coco annotation json file, the segmentation field is still empty as the attachment shown.
coco_annotations_arejdbfc.txt (18.8 KB)

After checking the ~/.local/share/ov/pkg/isaac-sim-4.2.0/extscache/omni.replicator.core-1.11.20+106.1.0.lx64.r.cp310/omni/replicator/core/scripts/writers_default/coco.py, I found that the iscrowd is 0 and segmentation is empty both with TODO comments.

I want to clarify that the panoptic segmentation of CocoWriter is currently unavailable, right?