Holdout objects cut out the volume and do not take into consideration parts of volume, that are in front of the object that should not be cut out.
When clearing the holdout property, we can clearly see smoke, that is in front of the geometry, that hasd been cut out.
@egertkanep is it safe to assume you are using the latest Composer version?
I am using 2023.2.5
Yes I have to say this is a very “VFX” style workflow. This is not something Composer is designed to support. The “matte” object you are describing is not designed this way. It is a simple matte through to the background alpha. It is not designed as a vfx hold out workflow. And if it does not interact with vdbs correctly that is not something will be addressing. RTX is designed as a photo realistic simulation rendering engine. But we focus on complete scene rendering. Not advanced composition. We do not try to support it.
Indeed I was exploring Omniverse in terms of using it for some VFX. Thanks for clarifying that that is how the holdout is supposed to work. There are some dirty workarounds though :D.
Well I am happy to help and glad you are exploring it, but please understand we are focusing on physically accurate simulation. Vfx is not our focus.