Unity client (visual quality, positional offset and logging issues)

Gaming and Visualization Technologies XR – VR/AR/MR CloudXR (VR and AR Streaming)
1

Hi CloudXR team,

Our team has been working on integrating support for the Unity based CXR client, and we’ve run into a collection of issues that we were hoping you could shed some light on.

As a point of comparison, we have a functional native Android client based on the CXR 4.0 SDK, and none of the issues below are present, making us feel confident that these issues reside in the Unity SDK specifically. For reference, we are using v0.1.1 of the Unity SDK, and testing on a Quest 3.

The issues are listed below in order of importance for us:

1. Visual quality degradation
The visual quality for the Unity client is noticeably worse than in the Android client. It seems like the Unity client might be upscaling the resolution. Please see the attached videos for a comparison.
CXR-client comparisons.zip (39.1 MB)

2. Viewport offset
When connecting to a stream, the stream is being rendered in the direction where we had started the boundary calibration instead of the current HMD direction, requiring us to face the same direction where we were facing during boundary calibration and resetting the HMD to have it render in the correct orientation. If we don’t do this, there are very noticeable black areas visible in the viewport. This can also be seen in the attached videos.

I believe this is the same issue that others have described here:

3. Log spam
When the client cannot connect to the server, it is logging at a rate of about 190 log messages per milliseconds which is causing the log files to rapidly grow to the point of exhausting the storage on the device. The log message that is spammed is the following:
(I)[2024-02-01 23:56:19,296]=16:56:19=00000{533936572784} Wait For IO: wait was canceled by event: 0
(E)[2024-02-01 23:56:19,297]=16:56:19=00000{533936572784} UDP RTP Source: failed to receive data (Error: 0x80000013)

4. Crash when sleeping
We have observed that when taking off the HMD, it may go to sleep, when this happens occasionally the CXR Unity client will crash.

1 & 2 are critical for us in terms of delivering a workable solution to our customers, 3 & 4 are annoying but tolerable.

If there is any direction or guidance you can provide on any of these issues please let us know.

Cheers,
Chris

1 Like
2

Thank you so much Chris for all of this incredibly detailed and specific information. I will do what I can to make some of the issues more transparent.

  1. Visual quality. This is a general problem that does not have a perfect solution. In the “native” client, we are using the OVR API, which is extremely deprecated, and which allows us to get the device native physical resolution. OpenXR very specifically, and intentionally, does not expose a method to get this information, instead favoring the so-called “recommended” and “maximum” resolutions. On Quest 2, for instance, the “recommended” resolution is lower than physical, and the “maximum” is reported essentially as the GPU maximum texture size. Unity uses this recommended resolution to set up how it talks to the swapchain. The ultimate fix for this involves ensuring that Unity itself is sending higher-resolution textures to the compositor, and that the CloudXR layer is set up to use this higher resolution. A piece of information that would be useful is to check SteamVR’s settings, and see what resolution gets set for the native and Unity clients respectively. A simple thing to try would be to, in the Unity configuration, increase cxrLibraryConfig.cxrDeviceConfig.maxResFactor to 1.2 or 1.5, and see if this significantly improves the quality.
  2. Viewport offset. Your videos make the issue extremely clear, and it requires investigation. I am unsure how this misalignment occurs, but occur it does, and I am almost relieved, because the only explanation I had to offer prior to your videos and description was “excessive reprojection”, which meant that somehow the pose prediction feedback loop was wrong (which we attempted to thoroughly tune prior to release).
  3. Ah, yup, gotta deal with that. Let me see what I can do.
  4. I believe that Meta changed something, either with Quest 3 or with an overall firmware OS version update, which changes the way the device sleeps. Previously it was doing something fairly graceful and entering a low-power mode; now I believe it’s doing something less graceful but more battery-friendly. If I am correct about that, then we’ll need to debug the entire sleep-resume process again.

So I can replicate, can you tell me how you captured the videos? Is this with on-board video capture, or scrcpy/SideQuest, or MQDH?

3

AndreusNvidus,

Thank you so much for looking into the issues. It seems that you will eventually be able to devise a solution to the problems. However, I am somewhat concerned about the video quality. Your advice and explanations are sound, and it seems they should improve the situation to a certain degree. Yet, we have experimented with the maxResFactor. It definitely altered the video resolution quality, but we were unable to enhance it further. We set the “maxResFactor” to 1.5 in the cxrUnityConfig.json, which allowed us to adjust the resolution on SteamVR on the server, increasing it from the default of 1640x1780 (though I’m not certain those are the exact figures) to something around 2000x2000, but this made no noticeable difference in visual quality. Notably, reducing the resolution to something around 900x900 did degrade the visual quality, suggesting that we are unable to surpass the default setting. It seems there may be a ceiling effect at play, where the resolution increase is either not being effectively utilized by the headset’s display, or another bottleneck (potentially in the compositor or the video stream transmission) is preventing the higher resolution settings from resulting in improved visual quality. You mentioned that an ultimate fix involves ensuring that Unity is sending higher-resolution textures to the compositor and that the CloudXR layer is configured to utilize them. I need to explore how to adjust the CloudXR settings to implement these updates. If you have any preliminary advice on this, such as what to check or change first, that would be greatly appreciated. We are also planning to experiment with the Unity settings to see if we can achieve any quality improvements and will inform you of our findings. Thank you again for your support.

Jacek

P.S. We have used recording directly on the Quest 3 headset and copied the files from there.

4

You’ve got it. There’s two factors at play:
(1) The streaming resolution
(2) The Unity resolution

The former you have some ways to manipulate. The latter is a mess. My best hope is this setting:

1 Like
5

Viewport offset. Your videos make the issue extremely clear, and it requires investigation. I am unsure how this misalignment occurs, but occur it does, and I am almost relieved, because the only explanation I had to offer prior to your videos and description was “excessive reprojection”, which meant that somehow the pose prediction feedback loop was wrong (which we attempted to thoroughly tune prior to release).
I think it is caused by the protective boundary and center positioning. I used the Pico 4 headset for testing. If the protective boundary and positioning are successfully drawn and the CloudXR application is opened, the position and direction of the center point of SteamVR are normal. If you change the position or reset the position, black edges will appear. If you reopen the protection boundary and then open the CloudXR application, the positioning and orientation are based on the new position and orientation.
I recorded a video

  1. Open the protection boundary of the Pico headset and set the protection boundary.
  2. Open the CloudXR application and connect after opening. After the connection is successful, you can see that the center point and direction are the same as when drawing the protection boundary.
  3. Turn to the direction and reset the direction. At this time, a black border will appear.
  4. Exit the application, turn to another direction, open Pico’s protection boundary, and set the protection boundary.
  5. Open the CloudXR application and connect after opening. After the connection is successful, you can see that the center point and direction are the same as those used to draw the protection boundary.
    ScreenRecording_2024.03.11-19.31.27.zip (54.3 MB)
6

Thanks @houstun! This will help with the repro and fix.

In unrelated news, how was it getting the Unity client working on the Pico 4? I myself haven’t had a chance yet, but hope to (quite possibly this week!).

7

Hello AndreusNvidius,

I wanted to provide a small update on our additional tests and investigations into the Unity CXR client. Unfortunately, we have not been able to resolve the resolution issue. Typically, the game starts at 4K resolution, but when the headset is connected to SteamVR, it starts with a degraded resolution. We hope this information might offer some insight into how to address this problem.

Regarding the other issues, the only new information I can share is related to behavior based on latency. To give you some context, we run CloudXR on our platform by hosting servers on cloud instances. Our observations indicate that latency significantly impacts the Unity Client. When the Unity Client runs on the LAN, aside from the resolution issue, it seems to perform well. However, when we connect to anything outside the LAN, with a latency of 35 ms (which is about the best we can achieve in North America), the client connects cleanly but displays all the artifacts previously mentioned. When latency reaches 50 ms or more, connecting to the service becomes unreliable, and the user experience significantly degrades, with user interaction delays measurable in seconds. At latencies of 65 ms and above, the client fails to connect. To provide some context, the Android client, based on the CXR 4.0 SDK, performs well in these conditions as long as the latency remains within the 50 ms range.

We hope this information assists you in identifying some of the issues’ sources and figuring out solutions. Please let us know if there is any way we can support your work and investigation.

Jacek

8

just chiming in quickly.

When the CXR connection first starts, it purposefully begins at a lower bitrate, and then slowly tries to push up to maximum. If you aren’t on a good network, it may be capping earlier. I assume you have compared local LAN to cloud WAN? You should also ensure you have foveation enabled, that is critical for higher resolutions. I’d generally start with -f 50, and if that is doing great but you want to regain some edge quality, then try 60 or 70. Between foveation, a quality network, and a few seconds for the system to establish bandwidth limits, you should get up to the roughly 2kx2k rendered (which at 50% foveation will be 1kx1k per eye transmitted).

When discussing latency, it impacts a lot of things. But the underlying cloudxr client library is the same whether native or unity. So it is possible that unity buffering is introducing additional issues here. However, 50ms shouldn’t really cause it to not function – but we’ll need Andreus to investigate that further.

9

@houstun @chris.jarabek @jacek.wielebnowski
I both do and don’t hope the problem is this simple.

Can you try the steps here and report back?