Well, hard to say what went wrong there. Have never seen this error myself before. Maybe using CUDA shaders from different examples clashed somewhere. The SDK examples are written to run individually, not to be able to copy-and-paste code together.
I would start pruning things down until it works, for example, only keeping the static parallelogram, then add the other branch with the transform, with a different parallelogram.
Then the other way round, starting with the mesh and getting that to work alone.
If it’s clear that all sub-trees work individually but fail when used together, check all CUDA code of the shaders if there isn’t anything wrong which matches the error description.
If you’re using MSVS 2013 already and you want to keep using OptiX 3.x, I would recommend to update to OptiX 3.9.1 which received a lot of improvements over 3.8.0.
See if the problem persists with that and if yes, maybe also update to CUDA 7.5 which is supported by OptiX 3.9.1.
In general I don’t recommend to begin learning OptiX by using the OBJ loader. That’s doing quite some things under the hood and it’s better to understand each individual step to build an OptiX scene graph first. My recommendations how to start leaning OptiX can be found here:
Use 64-bit build targets for OptiX applications and their CUDA to PTX translation.