Synthetic data generation via code: instanceSegmentation - ValueError


I’m trying to perform synthetic data generation through code, but when trying to retrieve the instance segmentation like so:

gt = sd_helper.get_groundtruth(["instanceSegmentation"], viewport)

I receive this error:

2022-10-03 14:48:09 [1,362,298ms] [Error] [omni.ui.python] ValueError: zero-size array to reduction operation maximum which has no identity

  /home/isaac/.local/share/ov/pkg/isaac_sim-2022.1.1/kit/extscore/omni.kit.pip_archive/pip_prebundle/numpy/core/ _amax
  /home/isaac/.local/share/ov/pkg/isaac_sim-2022.1.1/kit/extscore/omni.syntheticdata/omni/syntheticdata/scripts/ get_instance_segmentation
  /home/isaac/.local/share/ov/pkg/isaac_sim-2022.1.1/exts/omni.isaac.synthetic_utils/omni/isaac/synthetic_utils/ get_groundtruth
  /home/isaac/Documents/Extensions/dataset-generator/exts/omni.dataset.generator/omni/dataset/generator/ capture_demo_image
  /home/isaac/Documents/Extensions/dataset-generator/exts/omni.dataset.generator/omni/dataset/generator/ <lambda>

For context:

  • A part of the scene is semantically annotated using the add_update_semantics function from omni.isaac.core.utils.semantics.
  • The code is activated using an Extension.
  • is the python file that contains the code and capture_demo_image is the function that is called when capturing the image.
  • Other types of synthetic data generation (e.g. depth and semantic segmentation do work).
  • Through the “Synthetic Data Recorder” widget of Isaac Sim the instance segmentation does work.
  • I was planning on saving the captured image via code as you can see below, but it already gets stuck at the get_groundtruth function.
instance_seg = gt["instanceSegmentation"][0]
image_ins = visualize.colorize_segmentation(instance_seg)
im_ins = Image.fromarray(image_ins.astype(np.uint8))"{path}/output/Demo/instance_seg.png")

How do I fix this?

Update: I now noticed that when I first try to capture instance segmentation while there is no instance present to be captured in the scene, I receive the following error:

2022-10-06 11:16:52 [924,536ms] [Warning] [omni.syntheticdata.plugin] SdRenderVarToRawArray missing valid input renderVar InstanceSegmentationSD

The weird thing is that if I now run the exact same code as described in my main post, suddenly it does work. So in a way I’ve found the solution to my problem, but I do not understand why it is happening and whether there is something I should add to my code which can prevent this bizarre occurence from happening.

(In general it looks like an instance segmentation initialization problem that is out of my power)
(Update 2: this trick does not always work…)

Hi there,

synthetic_utils is planned to be deprecated and to be fully replaced by Replicator. Would it be possible to use the Replicator API to access the data? Here are some examples and tutorials 1, 2, 3.


Hi Andrei,

Good to know, thank you! I have indeed already tried to work with Replicator but only with the rgb example, I will also look into the other synthetic data methods.
The only thing that I noticed is that when I open multiple viewports at the same time through code (using the same method as in one of the links that you provided), Isaac Sim crashes, while it’s not a problem when I open them separately. Is this a known bug or is my GPU (NVIDIA GeForce RTX 2080 SUPER/PCIe/SSE2) simply not strong enough? (The error when the program crashes: VK_ERROR_OUT_OF_DEVICE_MEMORY)

What I generally noticed is that spawning them one at a time goes well up until 3 cameras, with 4 they start glitching to the other possible cameras, and with 5 the whole program crashes. Doing only 2 already at the same time also crashes the program.


Can you see if using small resolutions or simple scenes (e.g. 2 boxes) causes a crash as well?

A smaller resolution with the same scene causes the crash as well but less quickly, the smaller scene with 2 cubes does work with 5 viewports! I turned on the GPU memory statistics and 5 viewports use about 3GB in the simple scene already (with 1280x720 resolution). Simplifying my scene does allow me to open 5 viewports while all viewable at the same time, but then I still only have 1 GB of GPU memory left (so luckily it does the job (woohoo!), but it’s not ideal).

Are there more ways to avoid the high GPU memory usage? And is there a way to perform synthetic data generation without having to open all viewports belonging to each camera? For example, I’ve already tried to pause the simulation, then (through code) switch the active camera in a single viewport (to keep the GPU memory usage low) and capture an image, however, in simulation the viewport doesn’t change camera until the code has finished running, resulting into all the same images. I did it this way since in my scene, 5 images should be captured at the exact same time from different points of view, and since it’s a moving scene, pausing it seemed to be the most reliable way of capturing the exact same moment with all cameras, however, it’s clearly not resulting into the results that I wanted.

The offline pose estimation tutorial manually steps through the simulation. You can use the tutorial as an example to build your script, basically moving the camera around between paused steps and taking snapshots. Using a custom writer or by accessing the data directly you can then keep track in which position the camera is to label your data accordingly.

Thanks for the tip! I’ll look into it :)

@rosalie_va Your concern is clearly related to what is discussed here Reducing GPU memory usage for multi-camera uses - #8 by ltorabi

I am presenting some of my observations concerning the GPU memory usage.

@ahaidu In my case (several scenes with a robot arm having a camera attached to the end effector, see the last response in the topic cited above) the only solution would be to wait for each robot to pause its displacement and capture an image using only one camera for every robots by moving it around them ? It might slow things down a lot if images should be taken while the robotic arm is moving (which is not wished in my case at the moment)

1 Like

Will replicator support multiple cameras?

Yes replicator support multiple cameras (render products). See this example.

after creating viewports, can you close them? or recreate render product? without opening new viewports

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.