The weird thing is that if I now run the exact same code as described in my main post, suddenly it does work. So in a way I’ve found the solution to my problem, but I do not understand why it is happening and whether there is something I should add to my code which can prevent this bizarre occurence from happening.
(In general it looks like an instance segmentation initialization problem that is out of my power)
(Update 2: this trick does not always work…)
Good to know, thank you! I have indeed already tried to work with Replicator but only with the rgb example, I will also look into the other synthetic data methods.
The only thing that I noticed is that when I open multiple viewports at the same time through code (using the same method as in one of the links that you provided), Isaac Sim crashes, while it’s not a problem when I open them separately. Is this a known bug or is my GPU (NVIDIA GeForce RTX 2080 SUPER/PCIe/SSE2) simply not strong enough? (The error when the program crashes: VK_ERROR_OUT_OF_DEVICE_MEMORY)
What I generally noticed is that spawning them one at a time goes well up until 3 cameras, with 4 they start glitching to the other possible cameras, and with 5 the whole program crashes. Doing only 2 already at the same time also crashes the program.
A smaller resolution with the same scene causes the crash as well but less quickly, the smaller scene with 2 cubes does work with 5 viewports! I turned on the GPU memory statistics and 5 viewports use about 3GB in the simple scene already (with 1280x720 resolution). Simplifying my scene does allow me to open 5 viewports while all viewable at the same time, but then I still only have 1 GB of GPU memory left (so luckily it does the job (woohoo!), but it’s not ideal).
Are there more ways to avoid the high GPU memory usage? And is there a way to perform synthetic data generation without having to open all viewports belonging to each camera? For example, I’ve already tried to pause the simulation, then (through code) switch the active camera in a single viewport (to keep the GPU memory usage low) and capture an image, however, in simulation the viewport doesn’t change camera until the code has finished running, resulting into all the same images. I did it this way since in my scene, 5 images should be captured at the exact same time from different points of view, and since it’s a moving scene, pausing it seemed to be the most reliable way of capturing the exact same moment with all cameras, however, it’s clearly not resulting into the results that I wanted.
The offline pose estimation tutorial manually steps through the simulation. You can use the tutorial as an example to build your script, basically moving the camera around between paused steps and taking snapshots. Using a custom writer or by accessing the data directly you can then keep track in which position the camera is to label your data accordingly.
I am presenting some of my observations concerning the GPU memory usage.
@ahaidu In my case (several scenes with a robot arm having a camera attached to the end effector, see the last response in the topic cited above) the only solution would be to wait for each robot to pause its displacement and capture an image using only one camera for every robots by moving it around them ? It might slow things down a lot if images should be taken while the robotic arm is moving (which is not wished in my case at the moment)