Imposing Consistent Light in "3d conditioning" blueprint. (NVIDIA Omniverse Blueprint: 3D Conditioning for Precise Visual Generative AI)

AI & Data Science NVIDIA Blueprints
,
1

Hello.

This concept of 3D conditioning for precise visual image generation is game changing, particularly in the field of generating marketing contents with brand-approved 3D digital twins/assets of the hero assets. However I feel like there is room for improvements regarding this workflow to be actually useful in an enterprise marketing scene.

So I have two questions regarding the blueprint for “3d conditioning” (aka NVIDIA Omniverse Blueprint: 3D Conditioning for Precise Visual Generative AI)

interactive demo: 3D Conditioning for Precise Visual Generative AI Blueprint by NVIDIA | NVIDIA NIM
git url: GitHub - NVIDIA-Omniverse-blueprints/3d-conditioning: Enhance and modify high-quality compositions using real-time rendering and generative AI output without affecting a hero product asset.
NVIDIA tech blog (for context): How Generative AI Fuels Personalized, Brand-Accurate Visuals With OpenUSD

1. How to impose consistent light on the “hero asset” and the rest of the objects and background?

  • According to the comfyui workflow, the rgb image of the “hero asset” is masked, untouched, and then later merged with the generated background (along with the other objects in the scene).
  • This resulting image is very unnatural since the generated background and the “hero asset” has different lighting.
  • Is there a way to impose consistent lighting on both the “hero asset” and its surroundings?

2. How to generate a “clean contour” of the “hero asset”?

  • Below is the sample image generated using the blueprint, and as you can see the “hero asset” is unaltered as intended, however a “jagged contour” of the “hero asset” is also generated around the masked area as you can see in highlighted area of 2nd image below.
  • Is there a way or a workaround to prevent this from happening? Any help would be much appreciated!

image
image1024×1024 74.1 KB

image
image622×785 96.8 KB

2

@Simplychenable im really sorry and I know this is wild for cold calling and tagging you on my post. but I have been following your posts and comments on this forum and you struck me as a person with expertise in this area (if you are not, my apologies and you can just simply ignore this post).
So if you have the time, it would be greatly appreciated if you could help me out with this problem or direct me to some sources/materials that can help me tackle this problem? thanks!

3

@jaeyeun i’ll be frank, you have grossly overestimated my knowledge on the subject matter; there is no doubt lots more people knows more than i do. that said, i have been keen on seeing developments with 3d conditioning’s approach using OV/OpenUSD. it’s one of the topics that i’d like to dig into deeper myself.

regarding 3d conditioning or blueprints, i can’t say i have done much work with it. so i’ll defer your questions to the NVIDIA devs. i do agree, however, on your observations in which combination of certain inputs (in the interactive demo) create certain level of uncanny valley in terms of integration of key subject(s) and generated background. i observed similar visual discrepancies as you have already noted (fringe edges around the machine and window panes, lighting and shadow mismatch, etc), so i’ll also be interested in the dev’s in-depth responses as well!

1 Like