CloudXR 3.2.2 has worse video quality

Gaming and Visualization Technologies XR – VR/AR/MR CloudXR (VR and AR Streaming)
25

We fixed an image quality issue that was our fault (using the recommended width/height for the swap chain but the display res for the CXR video stream, which resulted in a blurry final image due to downscaling).

But, enabling this flag still yields an unstable app, the video stream hangs after a few minutes. It’s 100% repro, and unreliable for us. I would love the extra performance (or rather, battery life savings, as it runs fine at 120 Hz regardless), but not at the expense of stability. Also, at certain resolutions, the image flips upside down using that flag but not when it’s disabled. I am pretty sure there are bugs in it.

This is using OpenXR on Quest, not OVR / VR Api, so the code is completely different than your code sample but we tried to keep all the relevant CXR parameters the same (sometimes making mistakes along the way, but they’re mostly solved). I’m just curious, do you still have plans to make your own OpenXR-based client? or is that still a ways off. It would be nice to compare implementations.

26

so really quick response:

  1. glad you found quality issue that was downscaling, It’s never going to be perfect match, as steam plays with resolution whenever it wants. :)
  2. we’ve not had any experience with imagereader decoder on quest2 hanging after a few minutes. I would sit here and play alyx, superhot, etc, for 20, 30, 40+ minutes in a session easily. And I don’t know about the image flipping, never seen that. I’ll note we also don’t run 120hz, so that as well is a variable I cannot account for atm.
  3. if you are using openxr, you’re outside our realm of experience at the moment. We do have plans to do an XR client, but my guess is it’s still a ways off.

Most issues we encountered in the past were corner cases, timing related, buffer availability related, missing (or locked) mutex, etc.

27

I’m pretty sure the hanging / instability issue also happens even in your 3.2 Vr Api Quest sample, when Quest 2 is running at 90 Hz which enables the advanced image decoder. I will double check. It could be a side effect of a recent update to SteamVR, who knows. We use the OVR 3.2 sample as well, a lot, but it’s still locked to 72 hz which doesn’t enable the flag. I’ll double check this using your sample with nothing else changed except forcing it to use 90 Hz instead of 72 on Quest 2, or even just keep it at 72 Hz but force the flag enabled regardless of the refresh rate. TBH I’m not sure why the current sample only enables it above 72 Hz, as if it’s stable and more efficient, shouldn’t it be always-on? I’m not sure what the thinking is behind this.

28

image

If I force the flag being used at all refresh rates, on Quest 1 the image is flipped upside down. This is definitely a bug.

Also, on Quest 1 and 2, and Quest 2 at certain resolutions (regardless of the above flag), there are wavey vertical lines when you enable foveation (-f 50). Disabling foveation looks overall worse / blurry, but there are no distracting wavey lines. This is 100% repro, and in your sample code, not related to our implementation at all. (though we experience the same issues). We have to fine-tune the resolution to avoid the wavey lines but the defaults are no good on Quest 1 (1440 x 1600 per eye res, what’s used for CXR stream)

Can you repro this and get back to me?

I get the image flipped upside down on Quest 2 as well, at certain resolutions, when that flag is enabled (> 72 Hz). It seems clear to me that this is an experimental flag that isn’t ready for prime time yet.

29

Hello All
It’s been more than 6 month, and the issue is still open. The main reason is that few users see this happening, and nobody at Nvidia has ever been able to produce it.
Also some other compression issues might look like the same issue.
I’ll summarize what we know:
*Happens on both Quest II & Focus 3, never reported on Windows (internally we have not tested)
*Only happens with CloudXR 3.2, not older version
*Switching to AIImageReaderDecoder makes the issue disappear, but it cannot be a long term solution, as it makes the Focus 3 fan run at full throttle and produces instabilities on Quest II
*And the strangest thing is that when recording the decoded stream on files, the issue is never visible, even if the person who did the recording saw the issue while recording !!!

The general line of thought is that the issue is related to video decompression issues, as it happens on squares and video compression/decompression is assumed to work on tiles. And as CloudXR is what it is, a video compression solution, in case of bad bandwidth, tiles will appear of course.

I’d like to orient the analysis on another direction, a GPU synchronization issue.

When looking more into the error, it looks like the screen is divided in squares, and 50% of the squares display the current frame and 50% the previous frame.

Therefore, and hypothesis would be that the texture copying between the thread that does the decompression and the thread that copy it to the framebuffer, or to the screen has an synchronization issue and start to do the copying while the source has not been yet ready.

I’d like to add two more things we can see, that I do not know if they are related.
(1) Android logcat constantly shows the OpenGL error: “[SurfaceTexture-16-20287-1] bindTextureImage: clearing GL error: 0x502”
(2) A basic OpenGL ES tracing (with Graphics API debugger) of a complete session, including the CloudXR work, shows an interesting error:

2836: Draw 9
├── 2862: eglDupNativeFenceFDANDROID(dpy:12970367409584359472, sync:480609952560)->434
└── 2863: eglDestroySyncKHR(dpy:12970367409584359472, sync:480609952560)->472446402561
2836: Draw 9
└── 2864: glGetError()->1282

But the tracing was done while the issue was not visible and the FPS very bad, so maybe it has nothing to do with the issue.
We’ll try next week to have a trace while the issue is happening see if it can tell more.
A issue with the GPU driver might be another direction.

I hope it rings a bell to anybody and help resolve the issue.

Best regards,
Alexis Vartanian

2 Likes
30

Yes, we’ve been unable to root cause any of the described issues, as I don’t believe they’ve been seen in-house. The new decoder is pretty solid, but admittedly is new and using some probably not-well-exercised interfaces. I believe we’re disabling the AImageReader decoder in a subsequent release, until engineering has enough cycles to dedicate toward it. So long as 90hz is working stable with the original decoder, we should be fine.

But, to pre-answer the question I know will come next, yes, we may very well NEED AImageReader for 120hz stable support (it’s not a bandwidth issue, as it will use same bw for 90 and 120, 120 will just use higher encoding, but every decode session on android takes time, and passing the data around takes time, and at some point you exceed the time alotted to hit 120hz…).

I don’t know that I’ve ever seen an upside-down frame. I do seem to recall we may do a Y inversion at some point, since the frames are coming from DX, and we’re rendering through GLES, and they have different spatial systems. But we either do it everywhere, or I’m wrong and not at all. It shouldn’t at all be random.

Finally, yes, often the issues are related to driver, app, DX, server config, etc. Which may again be why different people see different results. We’d need a system config dump from each user, along with maybe client and server logs from a problematic run, noting which issues they are encountering (in the provided run), to look to see if any common factors.

31

You may need to care about this in next version( tested on Quest 2)

  • At 90 Hz, resolution 1.5x, Foveated 50% : quality is best( similar to 3.1)
  • Various configuration have “wave” issue such as 72Hz, resolution 1.0x, Foveated 100% or 30% or 40%,…
32

The PIco OpenXR / CloudXR sample app enables that flag all the time, and it results in an upside-down image with zero code changes whatsoever on this Pico 3 Neo Pro Eye here. I think flipping the image upside-down depends on the selected resolution, which can vary from headset to headset. They are using supersampling * base recommended res, so on Pico 3 that happens to give a value that results in upside-down images. Looking at the SteamVR mirror window, the actual SteamVR session is right-side up, and I verified all the FOV values were correct and not, say, negated or flipped which could also result in the projection matrix having an upside-down result. The bug is definitely on the client-side library and all other parameters are 100% correct. Disabling that flag makes my Pico 3 render right-side-up 100% of the time.

33

I will say that, 120 Hz mode works absolutely fine on both Quest 2 and Pico 3, without the flag. If it’s less efficient, then so be it, the headset isn’t doing much other work anyway and it’s definitely worth enabling it providing the current network conditions / bandwidth (and GPU performance on the PC side) is sufficient.

I’d like to also say that, the wavey line issue on Quest 2 is only gone at 1832 x 1920 per eye res. Any other CXR res and I see some waveyness as I turn my head. I wonder if the FOV values I’m passing to the local XR compositor for the CXR layer need to be reduced slightly to compensate for CXR. Does CXR add some extra margin of FOV or padding on the images to account for lag? Because that would explain a LOT. I don’t remember seeing anything about this in the docs but padding the FOV (and res) and then clipping it on the client side would make a ton of sense (although you can’t simply consume the frames in that case, as rapidly turning your head will result in seeing the black edges of the limits of the decoded frames).

34

Do you get really 120Hz at client side?
I only get 72Hz from 90Hz on server side when running on cloud. Running in local network is no issue.

35

Yes, I can switch from 72-120 Hz on Pico 3 just like Quest 2, its openxr runtime supports the FB / Meta refresh rate extension. You can see it in the Pico XR CloudXR sample (there they just query the current refresh rate, but getting a list of available ones and then picking the one you want from that list is trivial).

You have to enable the experimental 120 Hz mode in the settings on the headset. But it works 100%, with full 120 FPS streaming remotely (SteamVR side) at 120 Hz client side if you initialize CXR with the current refresh rate instead of something hardcoded. Frankly it’s weird that the OpenXR folks didn’t make changing the refresh rate a mandatory feature. Every VR headset should have this option.

Ex 90 Hz isn’t ideal to watch 24 FPS content on it, it’s better to use 72 Hz in that case (or 120 Hz if your headset supports it, for 5:5 pulldown). 120 Hz is also great for 60 FPS content including most games and 360 stereo VR videos are also recorded at 60p mostly.

Be careful not to use that AI decoder flag at any Hz, it’s unstable on Pico as well (flips the image upside down too, depending on the selected res).

36

Oh wait, sorry, yes you are right, it is challenging to stream 120 FPS from a poor network connection, but the bitrate requirement of a video stream doesn’t rise linearly with the framerate. I read somewhere (a long time ago, not sure if it’s accurate), that 120 FPS HEVC is only 25% more bandwidth vs 60 FPS at similar per-frame quality levels.

Besides, bandwidth for the videostream can be tuned using the max bitrate setting. It may well be the case that during congested network settings 90 FPS would look better than 120 FPS, and maybe even 72 FPS would look better than 90 FPS. These are all good possibilities. But I know that 120 FPS works great and looks great to my local PC over Wifi, and 120 FPS does work with remote cloud connections although it will look more blurry in general. I have a 3gbps up/down fiber optic connection so for me the main problem is the Quest / Pico’s limit of 150 mbps max bitrate on its XR2 video decoding chipset, and of course the cloud server.

We may be better off with higher FPS and lower res, and doing local upscaling via the XR2’s Upscaling software (by Qualcomm), but it’s not available publicly yet. Virtual Desktop already has it implemented and apparently it looks great.