Called rtIgnoreIntersection, but the mesh still has side effect

I’m trying to make some global illumination thing with path tracing, and I noticed a pretty annoying issue.

Basically I have models with transparent textures, and for those I added an anyHit program that checks the transparency of the texture, and calls rtIgnoreIntersection if it’s below a certain threshold.

What I noticed is that if there is a transparent mesh in front of another one, the color gets really distorted where the mesh is supposed to be transparent.

And just to make sure I didn’t mess something up, I created a material whose anyHit program is a single rtIgnoreIntersection call. Basically I made a computationally complex way of drawing nothing.
But I’m not drawing nothing, the mesh messes up the color of the one behind it.

Upon further testing, I swapped my material that calls rtTrace for one that simply samples a texture, and returns the color. Now the ghosting distortion effect disappeared.

So apparently rtIgnoreIntersection has a side effect when I recursively call rtTrace behind it?
Any idea how to solve this issue?
(I’m using optix 5.1.)

Did you provide a such a cutout opacity anyhit program for all ray types?

Please have a look at the OptiX Introduction examples.
GTC 2018 Presentation video, slides and github open source example links here:

I’m doing exactly that starting in example 07 which adds texture support and a cutout opacity material.

Yes, I’m only using one type of ray.

Hard to say what’s going wrong. I have not seen that effect and you have my example code to verify.
Could you provide a screenshot of the problem? (Hover over a posted message and use the paper clip icon in the top right to attach files.)

Such errors are normally missing initialization of all rtPayload fields to valid defaults in case there is none of the expected intersection events.

Maybe post the code of all your involved shaders. If confidentiality is a concern you can also send that as private message.

I don’t have access to my pc right now to show you a screen shot, but I can copy paste the code in.

#include "PerRayData.cuh"
#include <curand_kernel.h>

rtTextureSampler<uchar4, 2, cudaReadModeNormalizedFloat> diffuseTexture;
rtDeclareVariable(rtObject, top_object, , );
rtDeclareVariable(float, roughness, , );
rtDeclareVariable(float, isothropy, , );

rtDeclareVariable(float2, texcoord, attribute texcoord, );
rtDeclareVariable(float3, normal, attribute normal, );

rtDeclareVariable(optix::Ray, ray, rtCurrentRay, );
rtDeclareVariable(PerRayData, perRayData, rtPayload, );
rtDeclareVariable(float, t_hit, rtIntersectionDistance, );

rtBuffer<float, 1> rng;
rtDeclareVariable(uint, rngSize, , );
rtDeclareVariable(uint2, launch_index, rtLaunchIndex, );
rtDeclareVariable(uint2, launch_dim, rtLaunchDim, );

RT_PROGRAM void closestHit()
	if (perRayData.recursion > 0) {
		float3 color = make_float3(tex2D(diffuseTexture, texcoord.x, texcoord.y));

		uint index = (launch_index.x + launch_index.y*launch_dim.x + perRayData.recursion) % rngSize;
		float a = rng[index];
		float b = rng[index + 1];
		float c = rng[index + 2]*4.f;

		float cosAlpha = sqrtf(a / (roughness - a*roughness + a));
		float sinAlpha = sqrtf(1 - cosAlpha*cosAlpha);
		float phi = M_PI_2f*sqrtf(isothropy*isothropy*b*b / (1 - b*b + b*b*isothropy*isothropy));
		if (c >= 1.f && c < 2) {
			phi = M_PIf - phi;
		} else if(c<3) {
			phi = M_PIf + phi;
		} else {
			phi = M_PIf * 2 - phi;

		float3 V_in = -ray.direction;
		float3 N = normalize(rtTransformNormal(RT_OBJECT_TO_WORLD, normal));
		if (dot(N, ray.direction) > 0) N = -N;
		float3 B = normalize(cross(V_in, N));
		float3 T = cross(N, B);
		float3 V_out = B * sinAlpha * cosf(phi) + T * sinAlpha * sinf(phi) + N * cosAlpha;
		float3 H = normalize(V_in + V_out);
		float3 H_proj = normalize(H - dot(H, N)*N);	

		float t = dot(H,N);
		float u = dot(V_in,H);
		float v_in = dot(V_in,N);
		float v_out = dot(V_out,N);
		float w = dot(H_proj, T); //ez az abran rosszul van

		float3 S = color + (1 - color)*powf(1 - u, 5);
		float Z = roughness / powf(1 + roughness*t*t - t*t, 2);
		float A = sqrtf(isothropy / (isothropy*isothropy - isothropy*isothropy*w*w + w*w));
		float G_in = v_in / (roughness - roughness*v_in + v_in);
		float G_out = v_out / (roughness - roughness*v_out + v_out);
		float G = G_in * G_out;
		float D = (Z*A*G + 1 - G) / (4 * M_PIf*v_in*v_out);
		float3 R = S*D;

		float3 start = ray.origin + t_hit * ray.direction;
		optix::Ray outray = optix::make_Ray(start, V_out, 0, 0.0001f, RT_DEFAULT_MAX);
		PerRayData prd;
		prd.recursion = perRayData.recursion - 1;
		rtTrace(top_object, outray, prd);

		float pdf = v_in*M_1_PIf*0.5f + Z*A / (8.f * M_PIf * dot(V_out, H));
		perRayData.result = fminf(prd.result*R/pdf*v_in, make_float3(1,1,1));
	} else {
		perRayData.result = make_float3(0,0,0);
#define NOMINMAX
#include <optix.h>
#include <optixu/optixu_math_namespace.h>

using namespace optix;

rtDeclareVariable(unsigned int, MAX_RECURSION, , );

struct PerRayData {
	float3 result;
	int recursion;
#include "PerRayData.cuh"

rtDeclareVariable(uint2, launch_index, rtLaunchIndex,  );
rtDeclareVariable(uint2, launch_dim, rtLaunchDim, );
rtBuffer<float4, 2>   result_buffer;

rtDeclareVariable(float3, eye,  ,   );
rtDeclareVariable(float3, lookat, , );
rtDeclareVariable(float3, up, , );
rtDeclareVariable(float3, right, , );

rtDeclareVariable(rtObject, top_object, , );
rtDeclareVariable(float, frameCount, , );

rtBuffer<float, 1> rng;
rtDeclareVariable(uint, rngSize, , );

RT_PROGRAM void camera()
	uint index = (launch_index.x + launch_index.y*launch_dim.x) % rngSize;
	float2 d = (make_float2(launch_index)+make_float2(rng[index], rng[index + 1])) / make_float2(launch_dim) * 2.f - 1.f;
	float aspectRatio = (float)launch_dim.x / (float)launch_dim.y;
	float3 dir = normalize(lookat + right * d.x * aspectRatio + up * d.y);

	optix::Ray ray = optix::make_Ray(eye, dir, 0, 0.0001f, RT_DEFAULT_MAX);
	PerRayData prd;
	prd.recursion = MAX_RECURSION;
	rtTrace(top_object, ray, prd);
	result_buffer[launch_index] = make_float4((make_float3(result_buffer[launch_index]) * frameCount + prd.result) / (frameCount + 1.f), 0);

The any hit program for the other material may as well be a rtignoreintersection.

I just realized there are a few useless includes left in… Oh well.

1.) Make sure you have an exception program set at all times.
From your renderer architecture the image corruption could well be a stack overflow exception.
The stack overflow exception is enabled by default.
Example code (the important part is the super-magenta debug color in the case of an exception):

2.) If you can change the MAX_RECURSION at runtime, you would also need to make sure your Context’s setStackSize() is sufficiently large.
Note that in OptiX 6.0.0 that function doesn’t take effect anymore in the RTX execution strategy and you must use setMaxTraceDepth() and setMaxCallableProgramDepth() instead.

3.) Please try to always set your PerRayData prd.result to some defined color before each rtTrace call.
You do not reach the else-clause of the recursion check inside the closest hit program containing
perRayData.result = make_float3(0,0,0); if there is a miss. (That else clause can go away then.)
If the corruption looks like small screen aligned rectangles, that is an indication of a missing rtPayload initialization.

4.) If you implement a path tracer, change the algorithm from recursive to iterative! This is much more efficient and less memory consuming.
Please work through my OptiX Introduction presentation and example code and you have an elegant and flexible foundation for a global illumination path tracer.

5.) Instead of rtTextureSampler inside device code, please use bindless texture IDs. They are just integers and allow to check for the presence of a texture by comparing against RT_TEXTURE_ID_NULL and also don’t have any limits for the number of simultaneously used textures inside a scene.
Most importantly, only they allow to use all texture targets (cubemaps, layered) and texture fetching calls via the rtTex*(id, coord, …) intrinsices defined inside optix_device.h.

6.) There is no need to put an epsilon on the t_min of the primary ray, 0.0f will do.
That epsilon to avoid self-intersection is scene size dependent. Your constant value will not work in all cases.

  1. MAX_RECURSION can be changed runtime, in practice it’s 3.

  2. I have a miss program setup, so that shouldn’t be an issue? Besides, my code is buggy when there is intersection.

  3. That’s actually a great idea if I manage to get that working with openGL.

Okay, so stack overflow is definitely not an issue, added an error program to my cameras, and no error message.
At least not until I accidentally removed the exit clause. At least the error program is working.
Also increased the stack size by a factor of 16, but as expected that didn’t do anything.

Anyway, here is a picture:

Also I just noticed that this distortion issue only effects other meshes, but not the same mesh.
Now I’m really confused.


Please always try to keep the stack size as small as possible.

In that image, there is another invisible object in front of the hair primitives?

There was a very strange bug in OptiX 5.0 which had to do with any_hit programs calling bindless callable programs resulting in incorrect cutout opacity, but that should have been solved in OptiX 5.1.x.

Are your hairs non-triangle primitives with different bounding box and intersection programs?
I wrote my own hair intersection and shading routines and never encountered that issue.

If everything else I recommended is not helping, I’m unable to explain this effect with the given information.
I could only guess, like incorrect attributes generated by the intersection programs etc.

If this is a bug in OptiX 5.1.x, could you try with OptiX 6.0.0 instead?

I know, I know… It’s definitely not nice to waste vram, but I have 8GB of it, and currently ease of development has priority over hardware requirements. I’ll have to showcase it on a gt640 though, that’ll be awkward, if at all possible.

I don’t use bindless programs.

The hair is using the exact same geometry and material programs as everything else.
And it even works if there is no other geometry instance behind it. Like the hair and the neck are in the same buffer, and it works there. The envmap doesn’t get distorted either.

Time to install some optix.

Note that no variant of a GeForce GT640 will be supported by OptiX 6.0.0. That is a Kepler GPU at maximum and OptiX 6.0.0 only supports Maxwell and above.

If nothing helps, we would need to have a minimal reproducer in failing state along with the usual system configuration information below to be able to file a bug report.
OS version, installed GPU(s), display driver version, OptiX version (majro.minor.micro), CUDA compiler version used to generate the input PTX code, host compiler version.

It worked. Nice.
Thanks for the help.

Expected as much.
It’s a wonder really that those old PCs turn on at all.

Ok, that’s still a little scary.

Could have been an OptiX 5 or compiler issues. (The recommended development environment for OptiX 5 under Windows is MSVS 2015 and CUDA 9.0 not higher).

I didn’t have these problems with OptiX 5.1.0 when implementing cutout opacity in my OptiX Introduction examples. I would still recommend to use their architecture over your recursive approach.

I’m using CUDA 8, so maybe there is some issues with those 2 working together?

Well, I have been using CUDA 8.0 and 9.0 for quite some time with OptiX versions without huge issues, but I can’t prove the absence of errors.
I only switch compiler versions when I must or the newer version is really better.

That was my line of thinking too. If it ain’t broken don’t fix it.
But apparently it was indeed broken.