You don’t have vertices on a GeometryGroup. What you can get are the vertex attributes from one of the GeometryInstance’s Geometry underneath that. If that is a 1:1 relationship to your GeometryGroup, then yes.
Depending on what you need there are multiple ways to do that.
For example, if you only require the vertex positions of the primitive you hit, you could write them as additional attributes inside your intersection program. With GeometryTriangles in OptiX 6.0.0 that would go into the attribute program.
Mind that these are in object space. You’d need to transform them in world space via the rtTransfrom*() functions (recommended) or by rtGetTransform(object_to_world) and an own matrix multiplication. This seems to be your main point of the question.
Looks like this for normals. Same with rtTransformPoint() for positions and rtTransformVector() for tangents or bitangets.
More flexible and most likely generally faster than above method is to store bindless buffer IDs of the vertex attribute buffer and (optional) index buffer at your GeometryInstance. (At the GeometryInstance because any_hit and closest_hit programs do not have access to the Geometry scope directly. See program variable scoping here: http://raytracing-docs.nvidia.com/optix_6.0/guide/index.html#programs#program-variable-scoping )
Then you can access both inside the any_hit and closest_hit programs, fetch whatever primitive index you want and retrieve all vertex attributes you need, and again transform them to world space.
Be careful, if you need these attributes to do calculations against the rtCurrentRay, mind that it is in different coordinate spaces in these program domains. Inside the closest_hit program it’s in world space, in the any_hit program it’s in object space.
Even more generally, you can store bindless buffer IDs of the vertex attributes and index buffers of your Geometry in any context global data structure and access them from any of the program domains at will.
You just need to have a way to identify the geometry, e.g. like with an index variable defined at the GeometryInstance scope. But you only have the current transformation available inside the any_hit and closest_hit programs.
For example, I’m using that last method to implement arbitrary mesh lights. A global buffer contains these bindless buffer IDs (and some other data including the current transformation) inside a light definition structure per light in the scene, which means I can pick any light in the scene and have their geometry data available for sampling.
(Note that OptiX 5 supports a super-set of the GPUs supported in OptiX 4. Both support Kepler GPUs.)