Should I abandon GAS buildInput.triangleArray.sbtIndexOffsetBuffer if I use IAS?
If you have any GAS with build inputs which use more than one SBT record, you must use an sbtIndexOffsetBuffer for such build inputs to be able to specify which SBT record to use per primitive.
If there is only one SBT record per build input there is no use for an sbtIndexOffsetBuffer.
I’m doing the latter in my OptiX examples: https://forums.developer.nvidia.com/t/optix-advanced-samples-on-github/48410/4
Here’s a related post which explains the SBT indexing for that case: https://forums.developer.nvidia.com/t/sbt-problem-when-using-multiple-gas-objects/108824/2
Should I modify my codes to map one SUB-MESH to one IAS OptixInstance::sbtOffset?
That shouldn’t be required. Sounds more like you didn’t set the instance SBT index correctly. It needs to take the number of SBT records per underlying GAS of the previous instances into account.
Please have a look at how the SBT index is calculated exactly in this formula:
https://raytracing-docs.nvidia.com/optix7/guide/index.html#shader_binding_table#shader-binding-tables-for-geometric-acceleration-structures
Then check the basic GAS SBT record indexing for the simple and your multiple SBT record case.
Then compare your implementation with the example in the OptiX 7 programming guide which shows a case of instancing a GAS with multiple build inputs of which one has two SBT records:
https://raytracing-docs.nvidia.com/optix7/guide/index.html#shader_binding_table#example-sbt-for-a-scene
Specifically look at the SBT instance offset described in the table below that scene graph image.
Note that the second instance sbtIndex starts at 6 because that is the number of SBT records used by the first instance already.
If that’s not it, more information about how you setup your SBT exactly would be required.