Cannot generate more than 500 images with rep.trigger.on_frame(num_frames=1000)

Omniverse Synthetic Data Generation (SDG)
1

Hi,
I am using Replicator to generate synthetic data with custom usd files and randomizing them. When I tried to generate 1000 images with num_frames=1000 , only 503 got generated every time. I tried re running the code for 5 times and same happened. I can generate exact number of images if num_frames<=500. I am not sure what the issue is.

Any help is appreciated.

Thanks

2

@sai.puneeth.reddy.gottam interesting, any luck using frame ranges between 500 and 1000? does the who process stops when it hits 500 or does it keep going just not saving out the images? lastly, what’s your machine spec, perhaps providing the latest log file could facilitate the troubleshooting.

4

I have not yet tried with frames between 500-1000. I will try that too. I am using 2 GPUs NVIDIA GeForce RTX 4090 24GB.
The data generation does not stop, but skip to the next step in the loop. I run it headless.
The script looks like:
def main(step:int):
…
…
with rep.trigger.on_frame(num_frames=1000)
…
…
writer.attach(render_product)

rep.orchestrator.run()

for i in range(0,5):
main(step=i)

I ran it for 5 steps with different prims in each step.
Update: I ran it again today morning with 500 frames, and now it generated 336 images in every step. I don’t why it happened.

5

Hi @sai.puneeth.reddy.gottam Can you provide a log from one of the sessions where it did not output the correct number of frames?