Photorealism in Omniverse

Hello, we are using omniverse to generate synthetic images for traning. at the moment, despite of training with tens of thousands of synthetic images, we are not seeing improvements with the baseline (even though the base line is not that great). I suspect this is because of the domain gap which at this point could only be the photorealism.

The images do look pretty “toyish” even though we only by the best of assets. I am looking for help/documentation on how to modify the render settings to look more photo-realistic.

any would would be greatly appriciated,
Thank you

personally, achieving photorealism is a combination of things. while using the best assets is a great start, you may find relying on great assets and render settings may not be enough to get the result you are after. other factors such as scene lighting, asset shader/texture work, post process/volumetrics, composition/camera settings, and placement of assets, etc. can all attribute to the overall look. and just as you know, there is no set in stone formula to follow.

that said, you can probably poke around example datasets shipped with OV, such as the Da Vinci’s workshop to see what settings they used. but you should also keep in mind it may take time and require more than one artist to really finesse and push to photorealistiic level. if inclined, perhaps you can mention a bit more about your scene so the mods/devs can offer further guidance.

Hi,

Thank you for the link. I’m not sure why but the da Vinci’s environment does not look so great on my computer.

The render setting are local,
image

so it would make sence for the usd to render differently on my Omniverse. What render setting should I be looking at to close on the visual gap between my local da Vinci’s and the creative team’s da Vinci’s?

here are some examples of the images I am generating at the moment:
20240703_183030_4dd37a43-7c51-4723-8626-796b14966c25_leftImg8bit
20240703_183105_929b8ce3-d2a4-4059-a350-eae74f33a095_leftImg8bit
20240703_183130_8b105a7e-5d98-4cd4-95c3-769cc7aae6c5_leftImg8bit
20240703_183406_0b92a748-398d-4587-a2c3-169f62533a40_leftImg8bit
20240703_183911_84ec891b-edbd-49d3-9953-d9784199248b_leftImg8bit

For the lighting I am using sunstudy.
We feel this images are far from photorealism and it’s creating a large domain gap.

Thank you

Please, can someone point us a direction on how to achive photorealism in omniverse?

Maybe it would be easier to help me if I focus specifically on our domain gap and not photorealism.

These are real images we captured with our camera:
20240307_091408_769_0dea9a59dc6311ee8743a0cdf337b3d7_leftImg8bit

20240307_092044_821_f9f7b568dc6311ee995fa0cdf337b3d7_leftImg8bit
20240614_133918_544_7fd889672a5311ef8c54a0cdf3370f3f_leftImg8bit
20240616_080756_281_89d45ea22bb711efa58ea0cdf337b3d7_leftImg8bit

and I need to minimize the gap between the images generated with the omniverse camera and these reall images.

Thank you

more reall images:

20240616_105612_990_0bf803222bcf11efa41ba0cdf336e761_leftImg8bit
20240616_112023_039_6c3e0b902bd211efb74ea0cdf336e761_leftImg8bit
20240618_181333_744_79818eba2d9e11ef9d92a0cdf3397957_leftImg8bit
20240618_183103_716_eb5960112da011ef99baa0cdf3397957_leftImg8bit

Hi,
Comparing the sim vs real:
Generally speaking, the real photos have more details, and are far more contrasted.
You can note saturated colors, over-exposed areas in clouds and even on the ground where the sun hits.

In the sim, the horizon line ends the world “abruptly”, while in reality we can see hills, trees and other details, also, real terrain is irregular and in the sim it is flat, resulting in an even-colored green plane of grass.

In the real world, the area close the the camera is very rich in detail, and we can see the grass in composed of more elements like leafs, small pebbles, etc

Also, real photos have a poor image quality.

These are just few things that come to mind.

While what I pointed is doable - It is easier said then done, and may require a vfx/environment artist expert to join the project for consulting and maybe some assistance.

@lbenhorin has made several good points above.

seeing the actual photos does help paint the picture of what you are trying to achieve. real life has a lot more imperfections and variations in general compared to our 3d counterparts. not only applicable to objects in the scene, texture, but also imperfection from the camera. take for example, by introducing new grass patches (or use the same clump of grass) with slightly different color and height could help make it more random and introduce those variations:

image

another observation regarding the imperfections of the camera such as depth of field, exposure (it would seem the real captures get overexposed during day due to the aperture/fstop), but also obvious vignetting around the frame, which can be easily achieved with the post process setting and chromatic aberration (cyan/magenta fringing) could help make the images match a bit more to the reference. Also, have you tried using HDR/skydome to help produce less constant/even lighting that the sunsky produces in OV? there are resources like HDRIs • Poly Haven where you can try out a few different types of lighting scenario instead of pure clear sky that may create more believable result.

just my 2c

1 Like

Hi guys, thank you for your input.

@lbenhorin we will at some point have to bring in an artist.

@Simplychenable Thank you, will the HDR/skydome work with sunstudy?

More in general, I am trying to understand the render settings in omniverse.

This is how the Da Vinci’s workshop project looks in my Isaac sim.

this is how it should look:

please, can you help me understand what i should do in the render settings in omniverse to achive the latter?

Thank you

You are running it REALTIME mode and not PATHTRACING mode. Try switching to that. Secondly, as mentioned above, Isaac Sim, like the USD Composer template, is capable of incredible photo-realism but you have have experience with Lighting, exposure, color composition, camera work, post effects etc.

  1. Set it to path tracing mode
  2. Expand this arrow to actually get all of your rendering settings for camera. Exposure for one

Isaac Sim is really about simulation rather than photo real rendering. If you want to push photo-realism you should go download the latest USD Composer Kit app template from here. GitHub - NVIDIA-Omniverse/kit-app-template: Omniverse Kit App Template