5.1: Incorrect BBoxes in BasicWriter output

Omniverse Synthetic Data Generation (SDG)
1

Continuation of: Incorrect BBoxes in BasicWriter output

The problem is back, even with poolling of all object types.

I am wondering if there is a limit for the assets that we have reached?

We have N spawners (1 for each object type) that spawn 50 objects in UNDERGROUND and just scatter samples for each frame.

Recently after adding 3 new spawners the problems reappeared.

Attaching example output.

repro.zip (25.6 MB)

2

cc @jlafleche @pcallender

3

@jlafleche @pcallender after over a year of trying to find a resolution, with a lot of help from the team (you both especially), and remaining faithful to the technology running many tests and burning 100s of hours per month we have finally ….

… found a solution!

In general, we have stumbled upon the incorrect semantic data in outputs when:

  • The USD loading/unloading was happening anywhere in on_trigger. Instead, each SDG job should generate its population at start and then just scatter those objects. Minimal changes are possible, like randomizing tints. No animation changes, no accessory changes, no loading data during the steps.
    Lesson: split the generation into separate little jobs and randomize population at start.
  • Recent discovery: Turns out if you randomize visibility of lamps (on/off) in on_trigger that will also break the outputs from the BasicWriter. We stumbled upon this solution completely randomly but we have proven for it to be the cause of the issue. We were misled by the possiblitiy that our worlds were the problem and we have been debugging it forever, while in fact it was the light points all along.
    Lesson: randomize the population of lights at start.

Today we seem to have a fully working SDG and our 3d team is picking up the pace fulfilling the backlog that has been accumulated over from the original bug report.

We are now moving onto also randomizing camera attributes but there is a pattern emerging…

4

That is incredibly useful insight! When visibility of a prim is toggled, it effectively gets destroyed from the renderer’s side even if it remains in the USD scene graph. I think digging in this direction we might be able to finally work on isolating the issue. Thank you so much for sticking with us and sharing your findings!!