Hello,

I’m using Optix Prime with instancing and masking. The triangles are uploaded in the following manner:

```
RTPmodel mdl;
CHK_PRIME_INST(rtpModelCreate(*context, &mdl));
CHK_PRIME_INST(rtpBufferDescCreate(
*context,
RTP_BUFFER_FORMAT_VERTEX_FLOAT3,
bufferType,
&vertsPtr[0],
&vtxDesc));
CHK_PRIME_INST(rtpBufferDescSetRange(vtxDesc, 0, vertsPtr.size() / 3));
CHK_PRIME_INST(rtpBufferDescSetStride(vtxDesc, sizeof(float) * 3));
std::vector<int4> indices;
indices.resize(vertsPtr.size() / 9);
for (size_t triIdx = 0; triIdx < vertsPtr.size() / 9; ++triIdx) {
indices[triIdx] = make_int4(3 * (int) triIdx,
3 * (int) triIdx + 1,
3 * (int) triIdx + 2,
0);
}
CHK_PRIME_INST(rtpBufferDescCreate(
*context,
RTP_BUFFER_FORMAT_INDICES_INT3_MASK_INT,
RTP_BUFFER_TYPE_HOST,
&indices[0],
&indicesDesc));
CHK_PRIME_INST(rtpBufferDescSetRange(indicesDesc, 0, vertsPtr.size() / 9));
CHK_PRIME_INST(rtpBufferDescSetStride(indicesDesc, 0));
CHK_PRIME_INST(rtpModelSetTriangles(mdl, indicesDesc, vtxDesc));
int useT = 1;
CHK_PRIME_INST(rtpModelSetBuilderParameter(mdl,
RTP_BUILDER_PARAM_USE_CALLER_TRIANGLES, sizeof(int),
&useT));
CHK_PRIME_INST(rtpModelUpdate(mdl, RTP_MODEL_HINT_ASYNC));
```

My code has two paths, one for supplying an optional mask, and one for assuming all facets are unmasked and simply supplying affine transformation matrices. The issue I’m currently seeing occurs when there is no mask (though I haven’t yet tested with one). When there is no mask, the rays are constructed on the GPU and the descriptors are built in the following way:

```
CHK_PRIME_INST(rtpBufferDescCreate(
*context,
Ray::format,
rays.type(),
rays.ptr(),
&raysDesc));
rtpBufferDescSetRange(raysDesc, 0, rays.count());
```

Where Ray::format is of type RTP_BUFFER_TYPE_CUDA_LINEAR and pinned. The Ray format is: RTP_BUFFER_FORMAT_RAY_ORIGIN_TMIN_DIRECTION_TMAX.

Where things go awry is when I use the non-instancing version of Optix with model specified as one giant model instead of multiple sub-models, with rays of the same format, but the models are built without the special builder parameter (USE_CALLER_TRIANGLES). When I do this query (I’m using the CLOSEST query in both cases), it intersects ~500 different triangles out of the 16000 it intersects.

Besides the differences I’ve named above, the only other real difference is that the model is uploaded piecewise in a different order when using instancing (which initially led me to believe I had an indexing problem for my submodel to model translation, but as far as I can tell this is not the case). Perhaps uploading the vertices in different order and with instancing constructs a different enough acceleration structure that the intersections are different/wrong?

Any help would be appreciated. The code is pretty much a direct adaptation of the samples but with some organizational data structures to be used in conjunction with Matlab (it’s a mex module). And please let me know if I’m using some incompatible or otherwise invalid combination of descriptors and parameters for the API.