Rebuilding OpenGL Camera and Projection Parameters in OptiX

I wish you all a happy new year!

I tried to render a scene in OptiX with the same camera and projection parameters that are used in the OpenGL pass, to basically achieve rendered images from the same point of view. I converted code from the tutorial I try to follow to basically use glm instead of gdt to generate the ray direction used in the optixTrace-call:

const glm::vec2 screen(glm::vec2(ix + .5f, iy + .5f) / glm::vec2(optixLaunchParams.frame.size));

glm::vec3 ray_dir = glm::normalize(camera.direction + (screen.x - 0.5f) * camera.horizontal + (screen.y - 0.5f) * camera.vertical);

camera.horizontal and camera.vertical are computed as follows (field of view in radians), aspect is width/height):

launch_params.camera.horizontal = cam_projection.field_of_view * aspect * glm::normalize(shared_camera.get_right());
launch_params.camera.vertical = cam_projection.field_of_view * glm::normalize(shared_camera.get_up());

this produces actually quite similar images, but the further a certain pixel is located towards the border the more they ray traced image deviates from the rasterized image (ignore the colour, the issue is actually about deviating visible areas):

OpenGL Image
OptiX Image

So I thought, maybe this is due to the non-linear behaviour of perspective projection matrices and I tried to use the same matrices used in the OpenGL part which Sibaku already summarized. So I added projection and view matrices along with their inverted versions to the launch parameters and used them like this to generate the ray directions:

auto dims = optixGetLaunchDimensions();
int width = dims.x; 
int height = dims.y;

float x_screen = (ix + .5f) / width; 
float y_screen = (iy + .5f) / height;

float x_ndc = x_screen * 2.f - 1.f;
float y_ndc = y_screen * 2.f - 1.f;

glm::vec4 homogenious_ndc = glm::vec4(x_ndc, y_ndc, 1.f, 1.f);

glm::vec4 p_viewspace = optixLaunchParams.matrices.inverse_projection * homogenious_ndc;

glm::vec4 p_worldspace = optixLaunchParams.matrices.inverse_view* p_viewspace;

glm::vec3 ray_dir = glm::normalize(glm::vec3(p_worldspace.x, p_worldspace.y, p_worldspace.z));

This produced the following images, which appear to suffer from the same issues, however, if the value of the far clipping plane is large (third image) OpenGL and OptiX produce images with the same parameters.

Again the OpenGL Image
OptiX with new Code. Dragon is smaller

It somehow appears like the Vertical Field of View is different with OptiX, however when I set the far clipping plane to a value of ~50.000 the images appear to be the same

OpenGL Image with 50.000 fcp
OptiX image with 50.000 fcp

I thought I was familiar with how a projection matrix works, so this catches me pretty off-guard as I really don’t know why the far clipping plane should affect the vertical field of view (as it appears) in any way and consequently also the direction of the rays. Could somebody please explain? Or is there a more elegant way to achieve consistent camera and projection parameters between OptiX and OpenGL?

Hi @Gummel!

The most likely explanation is a bug somewhere in your camera projection setup. A clipping plane usually does not and should not affect the field of view at all. There’s not enough code here to see what the problem is, but with OptiX, the camera is 100% under your control, there is no hidden transform code running before you render like in OpenGL, so you should be able to track down why the clipping plane value is leaking into your matrix. First track forward exactly what happens to the clipping plane value line by line, and also track backward exactly where all your matrix values come from line by line. You should be able to verify and prove that the clipping plane value is independent of the matrix and cannot affect it. If you do that and become certain it shouldn’t be affecting your matrix, and you cannot see any explicit link, then first check for out-of-order parameter passing mistakes, and if that’s not it, then it might be time to look for accidental memory overwrite or out of bounds memory access somewhere. Good luck!


David.

Hello David,

thank you for getting back.

The matrices are the same matrices used in the OpenGL pass. Those are the lines that set the matrices for the OptiX pass:

launch_params.matrices.projection = glm::perspective(cam_projection.field_of_view, aspect, cam_projection.near_cp, cam_projection.far_cp);
launch_params.matrices.view = shared_camera.get_view_matrix_spheric();

launch_params.matrices.inverse_projection = glm::inverse(launch_params.matrices.projection);
launch_params.matrices.inverse_view = glm::inverse(launch_params.matrices.view);

So, the cam_projection values come directly from the GUI. I then use the inverted matrices to convert the ray directions into worldspace coordinates. As you have mentioned, the clipping plane values shouldn’t affect the field of view at all, which also was my understanding of a projection matrix. I mean, since glm::perspective takes values for near and far clipping plane, it sort of makes sense that it is somehow affected by those values, but I’m quite surprised that it is affecting the OptiX scene the way it does and doesn’t have an explanation for it, for I would have guessed that maybe ray directions pointing outside of the created frustum might just be limited a length that reaches the far clipping plane. I’m obviously using a wrong mental model.

Yes, glm::perspective does take near and far as inputs, and they do affect the projection matrix (I described it wrong) but still they should not affect the field of view if applied correctly, they should only affect how the resulting Z values are mapped. Having the field of view mismatch just means that the explanation is that part of the process of applying view parameters is different between your two implementations.

What is shared_camera.get_view_matrix_spheric()? That name sounds like something you might not want in this case. It might explain why in your first pair of images above, the horizon line is a straight line in the OpenGL image, and curved in the OptiX image?

I considered asking about that curved horizon line earlier and wasn’t sure I was seeing it correctly. Are you certain you are even using a linear view matrix on the OptiX side? What is the view class, and how does it generate rays? Between the potentially curved horizon and the call that implies something spherical, there seem to be multiple indicators that your OptiX view might not be a linear transform like you’re expecting. BTW I would think in theory your near and far values should only affect your projection matrix, and not your view matrix. So you could verify that your view is actually a matrix, and validate that the clipping planes don’t affect the values in the view matrix.


David

Hello David,

thanks for answering.

Maybe let’s start with the curved horizon line. In a different post here I found an explanation on how to implement a near and far clipping plane in OptiX, which suggested to simply pass the values for near and far clipping plane into the optixTrace-call as tmin and tmax values like this (cu_camera_position and cu_ray_dir are just glm::vec3 values converted into float3 values, as optixTrace didn’t want the glm::vec3):

optixTrace(optixLaunchParams.traversable, cu_camera_position, cu_ray_dir, optixLaunchParams.camera.near_cp, optixLaunchParams.camera.far_cp, 0.0f /* rayTime */, OptixVisibilityMask(255), OPTIX_RAY_FLAG_DISABLE_ANYHIT /*OPTIX_RAY_FLAG_NONE*/, SURFACE_RAY_TYPE /*SBT offset*/, RAY_TYPE_COUNT /*SBT stride*/, SURFACE_RAY_TYPE /*missSBTIndex*/, u0, u1);

So, the curvature there is caused by a fixed ray length, as the fare clipping plane value of 200 is only true for the center ray (the other rays further towards the screen border simply fall short), so I’m quite confident that an angle-dependent term for tmax would fix the curvature.

At this point I’m realizing that I might have skipped a bit of code that may have been helpful for others to gain a better understanding of what’s going on… sorry for that!

Regarding the shared_camera object: this used to be a simple camera class that only used Euclidean coordinates. I reused the class in a data generator that distributes light points in a hemisphere above a center point and the distributed points were computed in spherical coordinates and so I extended the class also to maintain its location in spherical coordinates besides the original location in Euclidean coordinates, so whenever either of the coordinates changes, everything else is kept consistent. The class also offers two different methods to get a view matrix: get_view_matrix() and get_view_matrix_spheric():

glm::mat4 cgbv::Camera::get_view_matrix()
{
	adjust_up();

	glm::vec3 forward_vec = get_forward(); // glm::normalize(target - position);
	glm::vec3 right_vec = get_right(); // glm::normalize(glm::cross(forward, up));
	glm::vec3 up_vec = get_up(); // glm::normalize(glm::cross(right ,forward));

	glm::mat4 m;

	m[0][0] = right_vec.x;		m[1][0] = right_vec.y;		m[2][0] = right_vec.z;		m[3][0] = 0.f;
	m[0][1] = up_vec.x;			m[1][1] = up_vec.y;			m[2][1] = up_vec.z;			m[3][1] = 0.f;
	m[0][2] = -forward_vec.x;	m[1][2] = -forward_vec.y;	m[2][2] = -forward_vec.z;	m[3][2] = 0.f;
	m[0][3] = 0.f;				m[1][3] = 0.f;				m[2][3] = 0.f;				m[3][3] = 1.f;

	m *= glm::translate(glm::mat4(1.f), -position);
	
	return m;
}


glm::mat4 cgbv::Camera::get_view_matrix_spheric()
{
	// https://stackoverflow.com/questions/40195569/arcball-camera-inverting-at-90-deg-azimuth
	
	float azimuth = spheric.x, elevation = spheric.y, distance = spheric.z;

	glm::mat4x4 view = glm::mat4(1);
	view = glm::translate(view, glm::vec3(0, 0, -distance));
	view = glm::rotate(view, glm::radians(elevation), glm::vec3(1.f, 0.f, 0.f));
	view = glm::rotate(view, glm::radians(-azimuth), glm::vec3(0.f, 1.f, 0.f));
	view = glm::translate(view, target);

	return view;
}

They both produce pretty much (except for rounding errors in the 6th or 7th decimal place) identical matrices. What actually surprises me a bit is that in the code in the first post, homogenious_ndc needs to be (x_ndc, y_ndc, **1.f**, 1.f) instead of (x_ndc, y_ndc, **-1.f**, 1.f), as shared_camera is basically looking from (0, 0, 30) (well, somewhat, as it’s a bit elevated, but it’s close enough for better imagination) towards (0, 0, 0). So I would have guessed that the ray direction would need to point somewhere into the negative z-direction, but if I do so, I seem to look in the wrong direction (so it had to be 1.f). Somehow, my intuition in the OpenGL world seems to fit quite well, but as soon as I try to apply it to OptiX, something appears to be slighly off.

So, the curvature there is caused by a fixed ray length, as the fare clipping plane value of 200 is only true for the center […] I’m quite confident that an angle-dependent term for tmax would fix the curvature.

Got it, I understand. Yes you could fix it with some math (you could use tan() call w/ angle, or probably simpler is compute the length to the point on the far plane, but anyway…). But remember, OpenGL solves this via the projection matrix and clip space. Maybe this is related to the overall question here of matching the views exactly. Using separate projection and view matrices like OpenGL does when you are ray tracing is generally awkward and unnecessary, usually with ray tracing you only need to specify the view, and the projection doesn’t need to be handled using a matrix. It might be easier to specify your view in terms of look-from, look-at, and field of view, and simply validate and debug your OptiX implementation to guarantee your field of view is working correctly and accurately. You should be able to match the OpenGL view exactly without needing to duplicate OpenGL’s viewing pipeline.

So you do have a linear view matrix. The view matrix code at first glance looks correct to me, and seems to be working based on your images. The view matrix does not (and should not) involve the field of view, nor clipping plane values. So this tends to implicate the projection matrix setup and/or ray generation. We already confirmed the far clip value shouldn’t have any effect on the field of view.

Maybe it could help to setup a reference scene and camera that will have a known view given a known field of view, and use it to help debug the camera? You could stick a unit cube one unit away from the camera and use a field of view of 90 degrees, for example. Perhaps this will make it easy to see which camera isn’t working correctly.

So I would have guessed that the ray direction would need to point somewhere into the negative z-direction, but if I do so, I seem to look in the wrong direction

Maybe this is confusion over which handedness you’re using for each phase? This might help: Is OpenGL coordinate system left-handed or right-handed? - Stack Overflow


David.

Hello David,

thank you for answering. I actually spent the afternoon and evening thinking about it. I’ve also checked the content of the projection matrices at far clipping plane values of 200, 3 000 and 5 000 000, which resulted in those values:

200

mat4                                                             inv_mat4
[       1.784022        0       0       0       ]                [      0.5605312       0       -0      0       ]
[       0       3.1715946       0       0       ]                [      0       0.3152988       0       -0      ]
[       0       0       -1.0100503      -1      ]                [      -0      0       -0      -0.49749997     ]
[       0       0       -2.0100503      0       ]                [      0       -0      -1      0.50249994      ]

3 000

mat4                                                             inv_mat4
[       1.784022        0       0       0       ]                [      0.5605312       0       -0      0       ]
[       0       3.1715946       0       0       ]                [      0       0.31529883      0       -0      ]
[       0       0       -1.0006669      -1      ]                [      -0      0       -0      -0.49983335     ]
[       0       0       -2.0006669      0       ]                [      0       -0      -1      0.50016665      ]

5 000 000

mat4                                                             inv_mat4
[       1.784022        0       0       0       ]                [      0.5605312       0       -0      0       ]
[       0       3.1715946       0       0       ]                [      0       0.3152988       0       -0      ]
[       0       0       -1.0000004      -1      ]                [      -0      0       -0      -0.49999988     ]
[       0       0       -2.0000005      0       ]                [      0       -0      -1      0.50000006      ]

So, the values are actually quite similar. Do you think, this could be due to rounding issues? I also had a look at the inverted-matrix approach with fixed values for tmin and tmax and I can’t really tell if it’s really an issue with the field of view or rather an issue with a misplaced camera.

I’ve also had a look at different implementations, which use W-, V- and U-vectors. However, those approaches never made clear what those vectors mean semantically? Is it correct to assume that W might be the forward-vector of my camera, U represents the right-vector and V represents the up-vector of my camera? If so, I assume that using them as normalized vectors might not necessarily incorporate the vertical field of view angle, which means I somehow have to incorporate that vertical field of view into the lengths of those vectors, right?

Notice how in your forward matrix, the 2x2 upper left corner is always identical, these control your field of view (they are the projection of your X & Y coordinates). Only the Z scale is changing, because that is affected by your clipping plane values, but that will not affect your field of view. In the the inverse matrix, you can see minor bit differences in the upper left 2x2, and yes that is due to rounding.

This might rule out your projection matrix and implicate something else, perhaps the code that turns the projection & view matrices into rays.

Yes, sometimes camera code uses a ‘basis’ or ‘frame’ of orthogonal vectors all located at the look_from point, and these basis vectors are often labeled U, V, W. It’s a good assumption that U==right, V==up, and W==look_at direction, and this is what the OptiX SDK samples do. It’s more efficient to factor the field of view into the scale of the U,V,W basis vectors than to handle FOV in the raygen program. That way, in raygen you only need to do a linear weighted sum of these vectors, where the U & V weights are your pixel index in normalized screen space. The W weight in this case would normally be a constant 1.0.

You could take a look at the OptiX SDK sample called optixPathTracer and study the camera setup. If you used that code as-is, you can match an OpenGL view exactly, even though it doesn’t have the concept of a projection matrix. The projection is defined by casting rays from the look_at toward the virtual image plane defined by U & V, the projection code is in raygen. The view is also just defined by the view parameters, i.e., the look_from, look_at, and fov, optixPathTracer doesn’t have a matrix for either concept. You can look at initCameraState() in optixPathTracer.cpp for the view, and __raygen__rg() in optixPathTracer.cu for the ray generation (handles both projection and view). These connect through our helper class called sutil::Camera.


David.

Hello David,

thanks for getting back.

I experimented a bit with different results and printed the computations for a certain ray (ix = dims.x / 2 and iy = 0) at different far clipping plane values and this is the result:

inv_projection (3000.00000):
[           0.56053         0.00000        -0.00000         0.00000     ]                [   0.00029]
[           0.00000         0.31530         0.00000        -0.00000     ]                [  -0.56050]
[          -0.00000         0.00000        -0.00000        -0.49983     ]                [  -0.87474]
[           0.00000        -0.00000        -1.00000         0.50017     ]                [   0.00033]
computed ray_dir:
[   0.00029]
[  -0.56050]
[  -0.87474]



inv_projection (1699.50000):
[           0.56053         0.00000        -0.00000         0.00000     ]                [   0.00029]
[           0.00000         0.31530         0.00000        -0.00000     ]                [  -0.55852]
[          -0.00000         0.00000        -0.00000        -0.49971     ]                [  -0.86734]
[           0.00000        -0.00000        -1.00000         0.50029     ]                [   0.00059]
computed ray_dir:
[   0.00029]
[  -0.55852]
[  -0.86734]



inv_projection (1012.20001):
[           0.56053         0.00000        -0.00000         0.00000     ]                [   0.00029]
[           0.00000         0.31530         0.00000        -0.00000     ]                [  -0.55542]
[          -0.00000         0.00000        -0.00000        -0.49951     ]                [  -0.85577]
[           0.00000        -0.00000        -1.00000         0.50049     ]                [   0.00099]
computed ray_dir:
[   0.00029]
[  -0.55542]
[  -0.85577]



inv_projection ( 741.50000):
[           0.56053         0.00000        -0.00000         0.00000     ]                [   0.00029]
[           0.00000         0.31530         0.00000        -0.00000     ]                [  -0.55262]
[          -0.00000         0.00000        -0.00000        -0.49933     ]                [  -0.84532]
[           0.00000        -0.00000        -1.00000         0.50067     ]                [   0.00135]
computed ray_dir:
[   0.00029]
[  -0.55262]
[  -0.84532]



inv_projection ( 470.70001):
[           0.56053         0.00000        -0.00000         0.00000     ]                [   0.00029]
[           0.00000         0.31530         0.00000        -0.00000     ]                [  -0.54660]
[          -0.00000         0.00000        -0.00000        -0.49894     ]                [  -0.82283]
[           0.00000        -0.00000        -1.00000         0.50106     ]                [   0.00212]
computed ray_dir:
[   0.00029]
[  -0.54660]
[  -0.82283]



inv_projection ( 304.10001):
[           0.56053         0.00000        -0.00000         0.00000     ]                [   0.00029]
[           0.00000         0.31530         0.00000        -0.00000     ]                [  -0.53756]
[          -0.00000         0.00000        -0.00000        -0.49836     ]                [  -0.78911]
[           0.00000        -0.00000        -1.00000         0.50164     ]                [   0.00329]
computed ray_dir:
[   0.00029]
[  -0.53756]
[  -0.78911]



inv_projection ( 200.00000):
[           0.56053         0.00000        -0.00000         0.00000     ]                [   0.00029]
[           0.00000         0.31530         0.00000        -0.00000     ]                [  -0.52427]
[          -0.00000         0.00000        -0.00000        -0.49750     ]                [  -0.73951]
[           0.00000        -0.00000        -1.00000         0.50250     ]                [   0.00500]
computed ray_dir:
[   0.00029]
[  -0.52427]
[  -0.73951]

Seems like the amount of deviation is enough to cause this particular problem, what do you think?

I’ll have a look at the implementation in the examples and may post again if I run in any trouble. Thank you.

I think it might be more helpful to look at your complete ray generation code, understand it line by line, and not worry about the matrix values too much. If you’d rather not post the whole thing, then the general goal should be to establish a reference test with a known field of view, and then debug the code properly - fully understand all the dependencies you’ve included, and then fully understand why the clipping value is affecting your field of view when it shouldn’t be. BTW, I would spend more time on small far clipping plane values. You’re only seeing minor discrepancies because your fov values are large, but I think we saw earlier that small values had a larger effect, which might make debugging a bit easier & more obvious, right? Again, remember that your OptiX camera is 100% under your control, so if there are any mysterious parts you included as dependencies, it’s time to look under the covers and understand exactly what they do.

In these new matrices, the Z values in the bottom-right 2x2 sub-matrix are not just rounding error, they appear to be a function of your inputs and matrix construction process. It could be that there’s a bug in the matrix construction (I have to admit I’m not looking at the values that carefully, and I’m not an expert on the OpenGL viewing pipeline), or it could be that there’s a bug or a conceptual error in turning the matrix into a ray. OpenGL’s perspective pipeline has the Z-divide (which is something I hope you’re not doing). OpenGL is built on the assumption that you’re going to apply the viewing pipeline transform matrix by all the vertices in your scene during rasterization. This is opposite of what you want when you do ray tracing. The OpenGL pipeline is also just extra complicated and confusing, IMO, and in my experience it’s really common for people to get stuck for a while trying to get it right and end up trying a bunch of random configurations and sign changes until it works. So generally speaking, I suspect trying to mimic OpenGL’s complexity on your OptiX side is making it harder than it needs to be. If you were starting fresh, I’d recommend using a simpler OptiX camera model, debugging it until your field of view behaves exactly as expected, and then matching the working OpenGL camera to it. Since you’re also there, maybe that’s seems like overkill, but the basic step of diving into why field of view appears to be a function of the far clipping plane value is something that should be relatively straightfoward to track down with a debugger and/or code analysis.


David.

Hello,

I finally found solution to produce identical images with OptiX using OpenGL information:

const int ix = optixGetLaunchIndex().x;
const int iy = optixGetLaunchIndex().y;

auto dims = optixGetLaunchDimensions();
int window_width = dims.x; 
int window_height = dims.y;

// X and Y in the projection plane through which the ray shall be shot
float x_screen = (static_cast<float>(ix) + .5f) / static_cast<float>(window_width); 
float y_screen = (static_cast<float>(iy) + .5f) / static_cast<float>(window_height);

// X and Y in normalized device coordinates
float x_ndc = x_screen * 2.f - 1.f;
float y_ndc = y_screen * 2.f - 1.f;

glm::vec4 homogenious_ndc = glm::vec4(x_ndc, y_ndc, 1.f, 1.f);

glm::vec4 p_viewspace = inverse_projection_matrix * homogenious_ndc;

// Transform into world space but get rid of disturbing w-factor
glm::vec4 p_worldspace = inverse_view_matrix * glm::vec4(p_viewspace.x, p_viewspace.y, p_viewspace.z, 0.f);

glm::vec4 ray_dir = glm::normalize(p_worldspace);
	
glm::vec3 origin = worldspace_Position_of_Camera;
	
float t_min = projection_near_clipping_plane;
float t_max = projection_far_clipping_plane;

// Values that are passed on to optixTrace(...)
float3 cu_ray_dir = make_float3(ray_dir.x, ray_dir.y, ray_dir.z);
float3 cu_camera_position = make_float3(origin.x, origin.y, origin.z);

Thank you again for your help!

Kind regards
Markus

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.