Streamed Frame Projection Issues with Unity CXR Plugin 0.1.0

[timgc]

We’ve begun integrating the CXR 4.0 Unity plugin (version 0.1.0) into a new Unity project, aiming to transition our native Android CloudXR client to Unity. While we’re excited about the potential benefits of a Unity client, we’re currently facing significant issues, particularly with the plugin’s basic example.

Our setup follows the official getting started guide, but we’re encountering inconsistent frame rendering on various Quest headsets (Quest 2, 3, and Pro). We’re using Unity 2022.2.12f1, noting a possible typo in the guide (recommending 2022.2.12fa). Also, the guide references the CXR 0.0.0 version, not 0.1.0, which might hint at compatibility issues or additional configuration needs.

You can review our implementation in our demo repo here. We’ve tested the CXR server both locally and remotely, encountering similar problems:

1. The app occasionally crashes on stream start, which we’ve temporarily bypassed with a delayed CXR connection start.
2. We’re also experiencing frame rendering issues when we do get the steam to work:

• Frames are often misaligned, primarily showing black space.
  

  
  
  –
  
  

• Frames appear reversed, as if in a mirror. [See a video of this here]

We are really looking forward to leveraging Unity’s features for our CloudXR client. However, these issues are currently blocking us from going down this path. We would be extremely grateful for any help or insight on these issues.

Thank you!

[dominique.sandoz]

Confirming this issue as well on the new released version 0.1.1 of the Unity Plugin:

Did you find a solution @timgc? Or is anyone at Nvidia looking into this…?

[timgc]

@dominique.sandoz I have not found a solution to this or heard anything from Nvidia on if they are looking into this. It’s reassuring to hear that another person has run into this issue, though I don’t see how this plugin is working for anyone. I would love to see an example of this working for someone!

[dominique.sandoz]

I had some luck by disabling Reprojection in the Cxru Reprojection Shader (_ReprojectionEnabled false). It gives other artifacts but at least the full field of view is covered, which I think does not matter because it is TERRIBLY SLOW, the latency between movement and update of the frame is unbelievable.

I was never more disappointed with something coming from Nvidia than with this.

[timgc]

Unfortunately, that is the behavior I am seeing as well when it does connect with the correct rendering. Unless an Nvidia dev can point out something I’m not doing correctly in my Unity setup, this plugin is not ready for primetime yet (as of 0.1.1). I’m really looking forward to being able to migrate my app to Unity, so hopefully the plugin will be updated soon to get the same level of performance that I get from the native implementation.

[dominique.sandoz]

also here a reaction or answer after 3 months is not too much to ask.

[AndreusNvidius]

Hello, folks,

I understand the frustration with our lack of communication. My sincere apologies.

This is now a tracked bug, and we are actively working to resolve it for a new release in the coming weeks.

[AndreusNvidius]

I both do and don’t hope this problem is this simple. Could you try the steps posted here and see if this resolves the issue?

Rendering Issue CloudXR Client based on Unity Sample

Alright, I may have a lead on this issue. @dominic.ermark , @xaver.freiherr-loeffelhol : Can you please check what the Tracking Origin Mode is set to? Hierarchy > Select XR Origin Inspector > XR Origin > Tracking Origin Mode > dropdown [image] If it is NOT set to “Floor”, could you try this with the reprojection shader enabled, and see if the issue recurs?

[sai.karthick]

Please try to reset the camera rotation for XR Rig. If you start a new scene or anytime you change XR Rig make sure to reset the camera viewport to its intended forward. That seemed to fix the issue as long as no recentering is done using Meta button recentering.

[zekri1]

Replace the ReprojectUV function in the reprojection shader with the following. It fixes the view issue, but performance is still bad.

float2 ReprojectUV(float2 uv,float4x4 tV,float4x4 tVS){
# if UNITY_UV_STARTS_AT_TOP
uv.y = 1.0-uv.y;
# endif

  //ray direction from screenspace uv's
  float3 rd = normalize(ComputeViewSpacePosition(float2(uv.x,uv.y), 0.9, UNITY_MATRIX_I_P));
  rd.z =- rd.z;
  rd = mul((float3x3)tVS, rd);// current view ray direction in world space

  //streamed frame ray direction
  float3 rds = mul(rd,(float3x3)tVS); 
  float3 ro = _WorldSpaceCameraPos;
  
  float4 pp = mul(UNITY_MATRIX_P,float4(normalize(rds.xyz),1));
  float2 uvp = (pp.xy / pp.w) * 0.5 + 0.5;
  return uvp;
}