Instancing issues with Optix Prime


I’m using Optix Prime with instancing and masking. The triangles are uploaded in the following manner:

RTPmodel mdl;
        CHK_PRIME_INST(rtpModelCreate(*context, &mdl));

        CHK_PRIME_INST(rtpBufferDescSetRange(vtxDesc, 0, vertsPtr.size() / 3));
        CHK_PRIME_INST(rtpBufferDescSetStride(vtxDesc, sizeof(float) * 3));

        std::vector<int4> indices;
        indices.resize(vertsPtr.size() / 9);

        for (size_t triIdx = 0; triIdx < vertsPtr.size() / 9; ++triIdx) {
            indices[triIdx] = make_int4(3 * (int) triIdx,
                                        3 * (int) triIdx + 1,
                                        3 * (int) triIdx + 2,

        CHK_PRIME_INST(rtpBufferDescSetRange(indicesDesc, 0, vertsPtr.size() / 9));
        CHK_PRIME_INST(rtpBufferDescSetStride(indicesDesc, 0));

        CHK_PRIME_INST(rtpModelSetTriangles(mdl, indicesDesc, vtxDesc));
        int useT = 1;
                       RTP_BUILDER_PARAM_USE_CALLER_TRIANGLES, sizeof(int),
        CHK_PRIME_INST(rtpModelUpdate(mdl, RTP_MODEL_HINT_ASYNC));

My code has two paths, one for supplying an optional mask, and one for assuming all facets are unmasked and simply supplying affine transformation matrices. The issue I’m currently seeing occurs when there is no mask (though I haven’t yet tested with one). When there is no mask, the rays are constructed on the GPU and the descriptors are built in the following way:

        rtpBufferDescSetRange(raysDesc, 0, rays.count());

Where Ray::format is of type RTP_BUFFER_TYPE_CUDA_LINEAR and pinned. The Ray format is: RTP_BUFFER_FORMAT_RAY_ORIGIN_TMIN_DIRECTION_TMAX.

Where things go awry is when I use the non-instancing version of Optix with model specified as one giant model instead of multiple sub-models, with rays of the same format, but the models are built without the special builder parameter (USE_CALLER_TRIANGLES). When I do this query (I’m using the CLOSEST query in both cases), it intersects ~500 different triangles out of the 16000 it intersects.

Besides the differences I’ve named above, the only other real difference is that the model is uploaded piecewise in a different order when using instancing (which initially led me to believe I had an indexing problem for my submodel to model translation, but as far as I can tell this is not the case). Perhaps uploading the vertices in different order and with instancing constructs a different enough acceleration structure that the intersections are different/wrong?

Any help would be appreciated. The code is pretty much a direct adaptation of the samples but with some organizational data structures to be used in conjunction with Matlab (it’s a mex module). And please let me know if I’m using some incompatible or otherwise invalid combination of descriptors and parameters for the API.

Hi Adam, looks like you never got a reply to this, sorry. Just to make sure I read your description correctly, it sounds like some triangles that you think should be intersected are being skipped, in a particular Prime setup: masked geometry, all triangles in a single Model, and not passing USE_CALLER_TRIANGLES. Is that accurate?

Debugging ideas:
Does this problem depend on the size of the scene? Can you figure out which triangles are being skipped, then shrink the scene down to just a few of those triangles to see if they’re still skipped?

When you switch to instancing mode, try placing an identify xform over the single model, rather than splitting into submodels, just to remove any submodel to model translation/ordering difference.

I messed with the primeMasking example in the SDK and it worked ok with your setup, where the geometry has a mask and the rays do not. I had to disable the mask update in the latter part of the sample, which would only be supported for “caller” triangles.