Bottles covering each other in replicator

[jonathan.meitzler]

I’m generating datasets of bottles for in order to train a program, however they keep overlapping each other and that can mess up the training. Is there a way to measure the amount of overlap and if it’s over say 50%, the bottle’s position is randomized again or hidden entirely?
Here’s the code I have:
import omni.replicator.core as rep

with rep.new_layer():
camera = rep.create.camera(position=(0, 250, 1300), focal_length = 18.14756)
render_product = rep.create.render_product(camera, (1920, 1200))

def dome_lights():
    lights = rep.create.light(
        light_type="Dome",
        rotation= (270,0,0),
        texture=rep.distribution.choice([
            '/home/ubuntu/Documents/AIM/EXR/belfast_sunset_4k.hdr',
            '/home/ubuntu/Documents/AIM/EXR/belfast_sunset_puresky_4k.hdr', 
            '/home/ubuntu/Documents/AIM/EXR/castel_st_angelo_roof_4k.hdr',
            '/home/ubuntu/Documents/AIM/EXR/fouriesburg_mountain_cloudy_4k.hdr',
            '/home/ubuntu/Documents/AIM/EXR/garden_nook_4k.hdr',
            '/home/ubuntu/Documents/AIM/EXR/golf_course_sunrise_4k.hdr',
            '/home/ubuntu/Documents/AIM/EXR/neon_photostudio_4k.hdr',
            '/home/ubuntu/Documents/AIM/EXR/outdoor_workshop_4k.hdr',
            '/home/ubuntu/Documents/AIM/EXR/pretville_cinema_4k.hdr',
            '/home/ubuntu/Documents/AIM/EXR/pretville_street_4k.hdr', 
            '/home/ubuntu/Documents/AIM/EXR/rainforest_trail_4k.hdr', 
            '/home/ubuntu/Documents/AIM/EXR/sandsloot_4k.hdr', 
            '/home/ubuntu/Documents/AIM/EXR/snowy_field_4k.hdr', 
            '/home/ubuntu/Documents/AIM/EXR/studio_small_09_4k.hdr',
            '/home/ubuntu/Documents/AIM/EXR/wide_street_01_4k.hdr',
            '/home/ubuntu/Documents/AIM/EXR/workshop_4k.hdr'
            ])
        )
    return lights.node
rep.randomizer.register(dome_lights)

def distant_light():
    sun = rep.create.light(
        light_type = "Distant",
        position = rep.distribution.uniform((-1500, -2000, -2000), (1500, 2000, 2000)), 
        intensity = rep.distribution.uniform(4000, 10000), 
        look_at = (0, 0, 0)
        )
    return sun.node
rep.randomizer.register(distant_light)

def move_shapes():
    shapes = rep.get.prims(semantics=[
        ('class', 'BlueMonster'), ('class', 'C4'), ('class', 'GreenMonster'), ('class', 'WhiteMonster'), 
        ('class', 'PurpleMonster'), ('class', 'OrangeMonster'), ('class', 'PinkMonster')])
    with shapes:
        rep.modify.pose(
            position=rep.distribution.uniform((-700, -150, -4000), (700, 400, 0)), 
            rotation=rep.distribution.uniform((0, 0, 0), (360, 360, 360))
        )   
    return shapes.node
rep.randomizer.register(move_shapes)

with rep.trigger.on_frame(num_frames=3):
    rep.randomizer.move_shapes()
    rep.randomizer.dome_lights()
    rep.randomizer.distant_light()

writer = rep.WriterRegistry.get("BasicWriter")
writer.initialize(
    output_dir="occlusion", 
    rgb=True, 
    bounding_box_2d_loose=True, 
    instance_segmentation=True, 
    bounding_box_3d=True,
    occlusion=True
)
writer.attach([render_product])

[mati-nvidia]

Hi @jonathan.meitzler. Let me check with the Replicator team.

[hclever]

Hi Jonathan,

Thanks for the inquiry.

I believe the solution will be to make use of our scatter nodes, that have built-in functionality to check for collisions and prevent them by using rejecting sampling. We have two nodes for this - scatter_2d and scatter_3d. The former is useful for cases such as randomizing the position of objects on a flat surface, while the latter can be used to randomize objects in a 3D hull (e.g. you have a cube representing a fish tank and you want to put fish in random locations in it).

Here is some sample code for the former - randomizing a couple basic shapes on a plane –

import omni.replicator.core as rep

with rep.new_layer():
    sphere = rep.create.sphere(semantics=[('class', 'sphere')], position=(100, 100, 100))
    cube = rep.create.cube(semantics=[('class', 'cube')],  position=(200, 200 , 100) )
    plane = rep.create.plane(scale=10, visible=False)

    def get_shapes():
        shapes = rep.get.prims(semantics=[('class', 'cube'), ('class', 'sphere')])
        with shapes:
            rep.randomizer.scatter_2d(plane, check_for_collisions=True)
        return shapes.node

    rep.randomizer.register(get_shapes)


    # Setup randomization
    with rep.trigger.on_frame(num_frames=30):
        rep.randomizer.get_shapes()

And here is another snippet for randomizing small spheres inside of a large 3D cube:

import omni.replicator.core as rep
 
with rep.new_layer():
    cube1 = rep.create.cube(semantics=[('class', 'cube')],  position=(125, 125 , 75), visible=False)

    def randomize_spheres():
        spheres = rep.create.sphere(scale=0.15, count=50)

        with spheres:
            rep.randomizer.scatter_3d(volume_prims=[cube1], check_for_collisions=True)
        return spheres.node

    rep.randomizer.register(randomize_spheres)

    with rep.trigger.on_frame(interval=10, num_frames=1):
        rep.randomizer.randomize_spheres()

Importantly, the scatter functions are sent a check_for_collisions=True flag.

Let me know if this is sufficient to get you going again!

-Henry

[jonathan.meitzler]

Hello Henry, thanks a lot for the suggestion. I tried testing it but everytime I use it in my code, the program crashes and I’m not sure why. The samples you sent work fine so something might be wrong with the code I’m using. Here’s the log and code:
kit_20221117_144121.log (1.1 MB)
bottlesVoid_VolumePrim.py (3.6 KB)

[hclever]

Hi Jonathan,

I simplified your code a bit since I don’t have the assets and it runs on my machine without issue. Can you try it? Code below-

import omni.replicator.core as rep

with rep.new_layer():
    camera = rep.create.camera(position=(0, 0, 1300), focal_length = 18.14756)
    render_product = rep.create.render_product(camera, (1920, 1200))

    theCone = rep.create.cone(semantics=[('class', 'cone')], position=(0, 0,-500), rotation=(90,0,0), scale=(39, 35.3, 23), visible=True)
    sphere1 = rep.create.sphere(semantics=[('class', 'sphere1')], visible=True)
    sphere2 = rep.create.sphere(semantics=[('class', 'sphere2')], visible=True)
    sphere3 = rep.create.sphere(semantics=[('class', 'sphere3')], visible=True)
    sphere4 = rep.create.sphere(semantics=[('class', 'sphere4')], visible=True)

    def distant_light():
        sun = rep.create.light(
            light_type = "Distant",
            position = rep.distribution.uniform((-3000, -3000, -3000), (3000, 3000, 3000)), 
            intensity = rep.distribution.uniform(3000, 7000), 
            look_at = (0, 0, 0)
            )
        return sun.node
    rep.randomizer.register(distant_light)

    def move_shapes():
        shapes = rep.get.prims(semantics=[
            ('class', 'sphere1'), ('class', 'sphere2'), ('class', 'sphere3'), ('class', 'sphere4')])
        with shapes:
            rep.randomizer.scatter_3d(volume_prims=[theCone], check_for_collisions=True)
        return shapes.node
    rep.randomizer.register(move_shapes)

    def rotate_shapes():
        shapes = rep.get.prims(semantics=[
            ('class', 'sphere1'), ('class', 'sphere2'), ('class', 'sphere3'), ('class', 'sphere4')])
        with shapes:
            rep.modify.pose(rotation = rep.distribution.uniform((0, 0, 0), (360, 360, 360)))
        return shapes.node
    rep.randomizer.register(rotate_shapes)

    with rep.trigger.on_time(interval=1, num=10):
        rep.randomizer.move_shapes()
        rep.randomizer.rotate_shapes()
        rep.randomizer.distant_light()

    writer = rep.WriterRegistry.get("BasicWriter")
    writer.initialize(
        output_dir="/home/hclever/data/bottle_crash", 
        rgb=True, 
        bounding_box_2d_loose=True, 
        instance_segmentation=True
    )
    writer.attach([render_product])

[jonathan.meitzler]

So I managed to fix the crashing problem, I applied the semantic to the blue symbol instead of it’s xform subcategory. Although something else is happening. When the collision check is True, the items have a habit of going outside the prim as this images shows:

The fact that some bottles are off camera isn’t a problem, what troubles me is that only a tiny fragment of a bottle could show up on one of the edges of the camera and mess with what we’re trying to train.

[jonathan.meitzler]

Maybe if a certain percentage of a bottle is off camera the visibility is toggled off like:
sphereBBox = rep.get_bbox(sphere)
if sphereBBox < 45% seen:
then visibility=false

It’s very rough but I think you get the idea

[hclever]

Hm - I’m not able to reproduce this issue with the bottles being so far outside of the cone on my end using spheres. Let me check on our internal side.

However, I definitely have some spheres that are right on the edge of the cone and partially visible. I believe this is because it is able to sample a point on the conical hull but it does not ensure the extents of the shape are entirely inside. It makes sense that is an issue, but we don’t have functionality to check if it’s say 45% seen, as far as I know. So as a workaround I would suggest making the cone you sample from just a bit smaller – so if you introduce a tolerance roughly the size of one of the bottles and slightly shrink the cone, it should fix the problem.

[hclever]

Does the same thing happen when you use my code for the spheres? or do they only go “way outside” the cone extents with your bottles?

[jonathan.meitzler]

There are moments where the spheres go outside the cone, but they never leave the screen or go as far as some of the bottles do. Is it because the bottles are bigger?

[hclever]

That is likely the case. I believe it samples the “center” of the bottle, however that is defined kinematically, to be on or inside of the conical hull. So I believe you would have to measure the furthest distance from that center to another point on the bottle and call that “max_overhang” or something, and shrink the cone by that much on each side.