How to properly use visibility masks?

We’re trying to use the new visibility mask API to selectively rtTrace specific scene groups but it does not seem to have an effect.

To reproduce the issue, we modified the optixSphere project from the Optix 6 samples:

optixSphere.cpp line 243:

+ RT_CHECK_ERROR( rtGeometryGroupSetVisibilityMask(geometrygroup, 2) ); line 72:

~ rtTrace(top_object, ray, prd, 4);

So the group that contains the sphere instance has visibility mask 2 and we trace visibility mask 4.
With this change, I expected the sphere not to be intersected since there’s no overlap between the two bitmasks, but it’s still rendered.

Is there something wrong with the way we use visibility masks?


Works for me on a Quadro RTX 6000 with 419.17 drivers. This might be GPU specific.

Please always list the following system configuration information when asking about OptiX isuses:
OS version, installed GPU(s), VRAM amount, display driver version, OptiX major.minor.micro version, CUDA toolkit version used to generate the input PTX, host compiler version.

Hi Detlef,

We found that we have the issue with 2 different configurations, and we have 2 other configurations that work.

Those do not work:

GTX 1080 Ti (x2) with 11 Gb RAM each
Windows 7
Display driver 419.67
Optix 6.0.0
Cuda 10.0

Quadro GV100 (x2) with 32 Gb RAM each
CentOS Linux 7.6.1810
Display driver 418.43
Optix 6.0.0 ?
Cuda 10.0 ?

However, those work:

RTX 2080 Ti with 11 Gb RAM
Windows 10
Display driver 419.35
Optix 6.0.0
Cuda 10.0

RTX 2080 Ti with 11 Gb RAM
CentOS Linux 7.6.1810
Display driver 418.43
Optix 6.0.0
Cuda 10.0

So basically it works on RTX GPUs and it doesn’t work with other GPUs, which are paired.

Thanks, that’s what I was looking for. I’ll file a bug report against OptiX. (=> That already existed.)