Optix 7.7 breaking changes wrt. Optix 7.6

tobias46 · July 28, 2023, 9:17am

Hi,
I have a codebase based of an Optix SDK sample that worked fine in Optix 7.3, needed little mending for 7.6, but does not work anymore in Optix 7.7. I checked the release notes and couldn’t find any indications for breaking changes. The Optix SDK examples all work fine though.

The specific issues I encounter are:

During runtime time the following call, which is a sort of sanity check throws: “invalid Bitcode” upon optixModuleCreate

    OptixModuleCompileOptions module_compile_options2 = {};
#if !defined( NDEBUG )
module_compile_options2.optLevel = OPTIX_COMPILE_OPTIMIZATION_LEVEL_0;
module_compile_options2.debugLevel = OPTIX_COMPILE_DEBUG_LEVEL_FULL;
endif
OptixPipelineCompileOptions pipeline_compile_options2 = {};

    OPTIX_CHECK_LOG(optixModuleCreate(
        state.context,
        &module_compile_options2,
        &pipeline_compile_options2,
        ptx.c_str(),
        ptx.size(),
        log,
        &sizeof_log,
        moduleptr));

When using the correct options (either the set that worked in Optix 7.6 or the adapted versions for Optix 7.7 with new fields added, see below):

OPTIX_CHECK_LOG(optixModuleCreate(
state.context,
&module_compile_options,
&state.renderpipe.options,
ptx.c_str(),
ptx.size(),
log,
&sizeof_log,
moduleptr));

I obtain:

Caught exception: OPTIX_ERROR_INVALID_VALUE: Optix call ‘optixModuleCreate( state.context, &module_compile_options, &state.renderpipe.options, ptx.c_str(), ptx.size(), log, &sizeof_log, moduleptr)’ failed: …\program_manager.cpp:72)
Log:
╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠╠…

OptixModuleCompileOptions module_compile_options = {
100, // maxRegisterCount
OPTIX_COMPILE_OPTIMIZATION_DEFAULT, // optLevel
//OPTIX_COMPILE_DEBUG_LEVEL_LINEINFO // debugLevel // Optix 7.3
DEBUG_LEVEL_LINE // debugLevel // Optix 7.6+
, // Below only for Optix 7.7
nullptr, // OptixModuleCompileBoundValueEntry → ignored
0, // numBoundValues → ignored
0, // numPayloadTypes → has to be 0, because “OptixPipelineCompileOptions::numPayloadValues” is not zero
nullptr // OptixPayloadType* → ignored because OptixPipelineCompileOptions are used
};

state.renderpipe.options = {
    false,                                                  // usesMotionBlur
    OPTIX_TRAVERSABLE_GRAPH_FLAG_ALLOW_ANY,                 // traversableGraphFlags
    4,     // numPayloadValues 
    3,    // numAttributeValues 
    OPTIX_EXCEPTION_FLAG_NONE,                              // exceptionFlags
    "params"                                                // pipelineLaunchParamsVariableName
    ,                                                       //      Below only for Optix 7.7
    OPTIX_PRIMITIVE_TYPE_CUSTOM,                            // usesPrimitiveTypeFlags
    static_cast<int>(false)                                 // allowOpacityMicromaps
};

My build system uses Cuda 12.2, Optix 7.7, Visual Studio 2022, Windows 10 64bit. It doesn’t matter if I build using NVRTC or not. It seems the signature of the compile options changed somewhere, but I verified that all new fields are ignored. I use custom type primitives as specified in OptixBuildInput.type: OPTIX_BUILD_INPUT_TYPE_CUSTOM_PRIMITIVES.

Do you have any idea what the reason could be, given that the code works in 7.6 but not 7.7?

Best
Tobias

droettger · July 28, 2023, 10:40am

What is your display driver version and installed GPU?

I had no issues with updating my examples from OptiX SDK 7.6 to 7.7 when adjusting the few API changes.
(Commit diff here https://github.com/NVIDIA/OptiX_Apps/commit/fd0d413ad6a47a3d9f5a5b98a66f56039f4b062d )

The only thing which looks wrong inside your code excerpts is the OptixModuleCompileOptions debugLevel DEBUG_LEVEL_LINE but that’s probably a typo. That OptixCompileDebugLevel define doesn’t exist in OptiX.

If you default initialized all structures of the OptiX API, there should normally be no issues when migrating between OptiX versions.

I ran into one issue in early R535 drivers with optixModuleCreate where the given input source code size was not adhered to but required a null terminated string for the PTX input, but you’re using a c_str() which should have that.
(I’m loading data as binary into a std::vector<char> because that works with both PTX and OptiX IR input.)
That should be fixed in current R535 drivers. That’s why the driver version is required for bug reports.

Are you only using custom primitives? Otherwise usesPrimitiveTypeFlags should have more bits set.

Is there a difference between Release and Debug builds of your application?
Asking because of the different optLevel and debugLevel settings.

Is there any additional information when enabling OptiX validation mode?
https://github.com/NVIDIA/OptiX_Apps/blob/master/apps/MDL_renderer/src/Device.cpp#L286

We would need some more information to be able to tell what is going wrong in your case.
If you get some “invalid bitcode” error message, that would be from deep inside the compiler chain and something which would only be possible to investigate when having a reproducer with the actual input sources in case there is something unexpected inside that.

Another thing to test might be different CUDA Toolkits. I’m still using CUDA 12.1.

Other than that, I would not set the maxRegisterCount to only 100. That’s from some bad example code and could affect performance. Instead set it to OPTIX_COMPILE_DEFAULT_MAX_REGISTER_COUNT and only change it when you analyzed the performance of your OptiX kernels with NSight Compute.
usesMotionBlur is also just an int, not a bool, and should be 0 or 1, or using the same cast as you do for allowOpacityMicromaps.
Are you calculating and setting the OptiX stack size explicitly? (Always recommended, and strictly required when using callables.)

tobias46 · July 28, 2023, 1:02pm

Dear droettger,

thank you for the fast response.

I had no issues with updating my examples from OptiX SDK 7.6 to 7.7 when adjusting the few API changes.
(Commit diff here Updated build system to support OptiX SDK 7.7.0. · NVIDIA/OptiX_Apps@fd0d413 · GitHub )

I see that there are no changes to the OptixModuleCompileOptions or OptixPipelineCompileOptions or OptixContext, so that eliminates all “unknowns” from the optixModuleCreate call. Unless, there is a problem from before. I also verified that OptixPipelineLinkOptions has no debugLevel set.

The only thing which looks wrong inside your code excerpts is the OptixModuleCompileOptions debugLevel DEBUG_LEVEL_LINE but that’s probably a typo. That OptixCompileDebugLevel define doesn’t exist in OptiX.

My apologies, I forgot to mention, that I set for convenience:

constexpr auto DEBUG_LEVEL_FULL = (DEBUG_MODE) ? OPTIX_COMPILE_DEBUG_LEVEL_FULL : OPTIX_COMPILE_DEBUG_LEVEL_NONE;
constexpr auto DEBUG_LEVEL_LINE = (DEBUG_MODE) ? OPTIX_COMPILE_DEBUG_LEVEL_MINIMAL : OPTIX_COMPILE_DEBUG_LEVEL_NONE; // Optix 7.6+

If you default initialized all structures of the OptiX API, there should normally be no issues when migrating between OptiX versions.

Aside of the context variable, that is what the sanity check was for.

I ran into one issue in early R535 drivers with optixModuleCreate where the given input source code size was not adhered to but required a null terminated string for the PTX input, but you’re using a c_str() which should have that.
(I’m loading data as binary into a std::vector because that works with both PTX and OptiX IR input.)
That should be fixed in current R535 drivers. That’s why the driver version is required for bug reports.

I am using std::string and std::string::c_str, so it should be nullterminated indeed, although I feel the error might be something alike. I am using driver V. 536.25 on a RTX 4080 mobile.

Are you only using custom primitives? Otherwise usesPrimitiveTypeFlags should have more bits set.

I am using Triangles and custom primitives. If that would have been the issue, why didn’t it show up in OptiX 7.6? I corrected it and nothing changed.

Is there a difference between Release and Debug builds of your application?
Asking because of the different optLevel and debugLevel settings.

The main difference between Debug and Release build’s are the Optix specific build/output flags. With Debug mode I obtained the output of the former post. In Release build, there is an empty log.

Is there any additional information when enabling OptiX validation mode?
https://github.com/NVIDIA/OptiX_Apps/blob/master/apps/MDL_renderer/src/Device.cpp#L286
We would need some more information to be able to tell what is going wrong in your case.
If you get some “invalid bitcode” error message, that would be from deep inside the compiler chain and something which would only be possible to investigate when having a reproducer with the actual input sources in case there is something unexpected inside that.

I will test this and report back.

Another thing to test might be different CUDA Toolkits. I’m still using CUDA 12.1.

Ok, this will be the next escalation step, if the validation information doesn’t turn up anything.

Other than that, I would not set the maxRegisterCount to only 100. That’s from some bad example code and could affect performance. Instead set it to OPTIX_COMPILE_DEFAULT_MAX_REGISTER_COUNT and only change it when you analyzed the performance of your OptiX kernels with NSight Compute.

Ok. Thank you. I was not sure where this value came from, if a colleague or just a leftover piece of code. I will analyze it once it’s running again.

usesMotionBlur is also just an int, not a bool, and should be 0 or 1, or using the same cast as you do for allowOpacityMicromaps.

Ah right. Thank you.

Are you calculating and setting the OptiX stack size explicitly? (Always recommended, and strictly required when using callables.)

I need to double check in the code and get back to you on that. I didn’t find the respective lines immediately.

droettger · July 28, 2023, 1:26pm

The main difference between Debug and Release build’s are the Optix specific build/output flags. With Debug mode I obtained the output of the former post. In Release build, there is an empty log.

OK, so you’re saying this only fails in debug mode and the release version works without issues?

If the error only happens for debug targets, are you compiling your PTX with device code debug information ? (NVCC options -G)

Next would be trying to isolate if it’s due to debug information inside the PTX code or due to the OptixModuleCompileOptions

module_compile_options2.optLevel = OPTIX_COMPILE_OPTIMIZATION_LEVEL_0;
module_compile_options2.debugLevel = OPTIX_COMPILE_DEBUG_LEVEL_FULL;

There had been issues with debug information inside PTX code in the past and maybe CUDA 12.2 generates something OptiX doesn’t understand.

I think there is a warning when using debug information without the full debug module compile option.
I normally compile without device code debug information for performance reasons.

So if you have debug information inside device code, check first with that disabled.

When that doesn’t help, change the debugLevel away from OPTIX_COMPILE_DEBUG_LEVEL_FULL to the minimal or none setting.

If one of this changed the behavior, we’d need a minimal and complete reproducer to fix that compiler issue.
Since this happens during the optixModuleCreate() already that would only require the failing input source.

I am using Triangles and custom primitives. If that would have been the issue, why didn’t it show up in OptiX 7.6? I corrected it and nothing changed.

Hmm, the default value 0 is automatically allowing built-in triangles and custom primitives.
The minimum number of intersection attribute registers is 2 anyway, so built-in triangles would always have enough. Not sure if that changes anything.
Well, I would recommend to indicate exactly what primitives are allowed inside a pipeline, even if just for good code style. For built-in curves and spheres that is required anyway and when only using built-in triangles it’s actually a performance recommendation inside the OptiX Programming Guide to only set the triangle bit instead of the default 0 setting.

(Sorry if the previous posts vanished shortly. I pressed the wrong button.)

tobias46 · July 28, 2023, 7:02pm

The main difference between Debug and Release build’s are the Optix specific build/output flags. With Debug mode I obtained the output of the former post. In Release build, there is an empty log.

OK, so you’re saying this only fails in debug mode and the release version works without issues?
If the error only happens for debug targets, are you compiling your PTX with device code debug information ? (NVCC options -G)

No, it crashes either way. In Debug mode the log is just filled with garbage, while in Release mode the error log is empty, but the error is still thrown by OptixModuleCreate:

Caught exception: OPTIX_ERROR_INVALID_VALUE: Optix call ‘optixModuleCreate( state.context, &module_compile_options, &state.renderpipe.options, ptx.c_str(), ptx.size(), log, &sizeof_log, moduleptr)’ failed: …\program_manager.cpp:72)
Log:

I will still get back to you on the Optix Stack size and validation mode information. I hope to be able to do this by next Monday/Tuesday. Have a nice weekend.

droettger · July 31, 2023, 7:06am

The OptiX stack size is something which happens later on. That was just something which came to mind on pipeline errors during the optixLaunch. You’re not reaching that if the problem happens during module compilation already.

If you could provide the input PTX code which crashes along with the display driver version, installed GPU(s) and the complete OptixModuleCompileOptions, we could check if we can reproduce this inhouse.

tobias46 · July 31, 2023, 7:15am

Dear Detlef,

the validation mode didn’t turn up any more information. I also checked Cuda V12.1 (using the DDriver from V12.2, i.e. 536.25) and it didn’t work either.

The OptiX stack size is something which happens later on. That was just something which came to mind on pipeline errors during the optixLaunch. You’re not reaching that if the problem happens during module compilation already.

I see. That appears logical. But either way I noticed that we didn’t configure the Stack Size manually so it’s left at its default. We will change this, too.

For now, I will try to get you a minimal working example, and if that fails provide you the PTX file. How would you like me to provide you the files? If you could provide me an e-mail address via PM, I can share the files with you there.

The setup is either way:
Windows 10 Pro Workstation
Cuda 12.2
Nvidia Driver 536.25
1x RTX 4080 Mobile
1x iGPU from i9-13900HX (Driver Version: 31.0.101.3887)
I tested it with and without Optimius, i.e. with adaptive GPU or forced discrete GPU (via UEFI Firmware). No change, so it’s unrelated to this.

Best
Tobias

droettger · July 31, 2023, 7:37am

Thanks for the additional information.

For now, I will try to get you a minimal working example, and if that fails provide you the PTX file. How would you like me to provide you the files? If you could provide me an e-mail address via PM, I can share the files with you there.

If the reproducer is smaller than 10 MB when packed into an archive, that could be sent to the OptiX-Help(at)nvidia.com email address you’ll also find on the bottom of the OptiX download site.
That would reach more OptiX developers in case someone is out of the office.
When using ZIP archives, please rename the extension to something like *.zi_, otherwise the email servers will block it.

The other way would be a private message on this forum with the reproducer as attachment but I don’t know the size limit of that.

If the reproducer is something bigger, that would need to be shared via some other mechanism. Google Drive and Microsoft Sharepoint should work. Most other file sharing sites won’t be accessible for us.

tobias46 · August 1, 2023, 8:29pm

I sent an e-mail now with the PTX and compile options as well as a runtime log from Optix 7.6. In the meantime, I could verify that neither compilation to OptixIR nor PTX nor NVRTC changes anything, but changing the SDK from 7.7 to 7.6 I build against does - in 7.6 it works without problems again. I also verified all CMake build options/scripts to be set as in the OptixSDK.

I will also verify building under Linux to see, if that changes anything.
I am still working to carve out a minimal reproducer because the codebase is quite entangled.

tobias46 · August 2, 2023, 9:06pm

Dear Detlef,

thank you so much for noticing the mistake. In fact what happened is that in the Optix 7.7 version, I set usesPrimitiveTypeFlags to:

OPTIX_PRIMITIVE_TYPE_CUSTOM | OPTIX_PRIMITIVE_TYPE_TRIANGLE

instead of

OPTIX_PRIMITIVE_TYPE_FLAGS_CUSTOM | OPTIX_PRIMITIVE_TYPE_FLAGS_TRIANGLE, which actually should be just set to 0 as the source code in optix_types.h indicates.

Fixing this typo made the code compile and execute again.

Best
Tobias

system · August 16, 2023, 9:07pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.