I’m trying to use Isaac SIM for dataset generation for the Neural Radiance Field training pipeline. However, for a fixed camera setting, it returns different images. The standard deviation for each pixel across 100 images rendered from the sim gives an image as in the attachment. The script I’m using resembles the one here.
Funny enough, when I ran COLMAP on these images, it gave better results during training NeRF than the one taken directly from the sim aligned with Procrustes analysis.
I appreciate any solution and help at this point. I run Isaac SIM with the following command:
docker run --gpus all -e "ACCEPT_EULA=Y" --rm --network=host --entrypoint bash -it -e "OMNI_USER=admin" -e "OMNI_PASS=admin" --name isaac -v ~/Projects/isaac-dataset/:/isaac-dataset nvcr.io/nvidia/isaac-sim:2021.2.1
An update: The issue can be solved by using path-traced rendering, turning on “eco mode,” or anti-aliasing in real-time rendering. The question remains: why does that shaking happen in the first place?
The same thing happened to me, Everything was fine before, but about a week ago the screen started to shake, and without any settings on my part. I tried reinstalling the GPU driver and Isaac Sim, but neither solved the problem.
I’ve rerendered the whole dataset and rerun COLMAP - I still obtain worse results when using camera parameters directly from the sim compared to the results from COLMAP. I’ve run Procrustes analysis, and some cameras are still misaligned. Interestingly, the euclidean distance between flattened orientation matrices in both sets is periodic (as if there was some issue with rounding errors in the sim). I upload the plot of errors below. Y-axis = euclidean distance, X-axis = timestamp of a frame in the animation.
Ok, the issue with periodicity was by my side, ouch…
I extended omni.isaac.core.tasks.BaseTask to create a class with the running sim. The only place I could find to place a specific camera setting and capture the renders with omni.isaac.synthetic_utils.SyntheticDataHelper was in the pre_step function. Apparently, setting the camera does not cause the change in camera in the simulator yet, pre_step has to be executed first by the sim directly. So overall, all the cameras were shifted by 1. I capture the renders by every second call to pre_step now and it works perfectly.
I would love NVidia to fill the docs so some stuff can be obvious how they’re executed! Also, if there’s a pre_step function, then there should be post_step.
