OptixHello embeded in new application run in Release but not in Debug mode

It looks like something is undefined in Debug mode, while it is defined and works well in Release mode.

It’s not possible to analyze that without the complete source code.
Usually it’s the other way round with uninitialized data.

Some guesses:

Which optixHello source code are you looking at?
OptiX SDK 7.7. 0 does not define optixModuleCreateFromPTX anymore. It has been renamed to optixModuleCreate because it can take PTX and OptiX IR input.
https://raytracing-docs.nvidia.com/optix7/api/optix__host_8h.html#ad270324129ed5f291b80b128daf8edea

First of all please make sure that you always null all OptiX structures you use in your code.
Things like this: https://github.com/NVIDIA/OptiX_Apps/blob/master/apps/MDL_renderer/src/Device.cpp#L347
That is required in case the structures change in the future and add new fields.
Usually zero is the default for OptiX field members.

You might have compiled the OptiX device code with debug information but didn’t set up the OptixModuleCompile and OptixPipelineCompilation options to use debug information.

To get more informatin please set a logger callback on the OptixDeviceContextOptions, set it to level 4 and enable the validationMode.
https://github.com/NVIDIA/OptiX_Apps/blob/master/apps/MDL_renderer/src/Device.cpp#L281
https://raytracing-docs.nvidia.com/optix7/guide/index.html#context#validation-mode

Note that the OptiX SDK examples are building the OptiX device code with debug information for the debug target. The resulting runtime performance is really, really slow.
I would not recommend doing that until really required. I usually translate the OptiX device code as release with line info for all targets which results in fast runtime in debug targets which helps as long as you only need to debug host code.