OK Cloud Streamer -- a fully open source CloudXR / OpenXR app

Hi everyone, I wanted to share some news.

At long last, Facebook released their IGL graphics library with an OpenXR sample that runs on Quest. I was waiting for this to happen before delving in on a quick side project that’s useful for my company and my PC VR game using Path Tracing and Nvidia RTX GPUs.

It took me about a week and a half, but I’ve written from scratch a fully open source CloudXR client for Quest and other OpenXR standalones.

It’s called “OK Cloud Streamer”. It uses the MIT license so you are free to use it for whatever purposes you want (just give me credit, please).

Here’s the repo,

Currently the IP address of the server is hardcoded and the the button events are still being patched in, but I’m fixing those tonight. After I get that done, it should be easy to modify the JSON config file to specify your cloud IP address and then connect to it automatically. I will eventually add an IMGUI-based UI panel in VR so you can specify your server IP address, refresh rate, resolution, and other settings prior to connecting to your server.

I sorted out most of the graphical glitches I had before, like slowdowns, micro-stuttering, wavey lines. All that’s sorted out and the result looks quite good, even if I do say so myself (I’m very picky and wouldn’t announce this project, or even use it myself, if it didn’t look at least roughly in the same ballpark as AirLink / SteamLink / Virtual Desktop). I just played SkyrimVR, my “go to” and it’s buttery smooth and looks pretty good! I have Link Sharpening implemented and refresh rate XR extension done too.

I will post a pre-built APK sometime in the next day or so, but feel free to sync it and built it yourself if you want to try it sooner (using your own IP address set in defines.h for now, until I finish the JSON config file parser).

I am open to modifying it to add custom features, reach out to me here or on Github or Twitter

1 Like

Congrats on the launch! Very cool that you’re targeting this to work on the Quest and other OpenXR devices. Have you had a chance to try any of them and see how it works?

I also noticed you had another question that went away after some time. If you’re running into issues let us know. Thanks!

Hi William, I deleted the post cause I figured no one was interested. Glad to see I was wrong!

I fixed the issue finally, it was something incredibly basic but I couldn’t understand at all why it was even working without it. I hadn’t added

image

I got most of the CXR issues sorted, but what I’d really appreciate some help on is a working sample for the direct server-integration (to get rid of the superfluous overhead of running an extra OpenXR compositor on the PC side in my engine), and an update about your Monado-related efforts for removing SteamVR as a dependency, which apparently uses the same underlying server code per your public blog posts about it.

Is there an update coming soon? What about RGBA in stereo? I could use that as well, but it fails with a “max 2 streams are supported error”, which implies the video pipeline is encoding the alpha channel as a separate dedicated video stream. I actually need depth transferred to the quest more than alpha, as depth helps inform the compositor to improve the local reprojection, as I’m sure you know. MVs as well would be good, so you don’t have to recompute those locally on device. MVs are a natural byproduct of video decoding process and it should be possible theoretically to expose it as part of cxrBlitFrame

Another thing I could really use is Vulkan support, i.e. a way to not be dependent on GL ES for cxrBlitFrame, which implicitly copies over the decoded video frames directly into the current context’s currently bound framebuffer. Vulkan support is really important on the mobile side for me, as the IGL graphics library I’m using has many optimizations for it. Also I have no idea how it would even be possible to call the GL ES cxrBlitFrame for single-pass stereo with a 2-layer swap chain, instead of blitting the eye/right eyes sequentially. So, basically, I have to disable single pass stereo in the entire renderer which doubles the number of draw calls and that can add significant overhead, especially on mobile which is heavily draw-call limited. Once I start implementing a full local UI, single-pass stereo of GLTF scenes will become increasingly more important. I could always blit the frames out to separate framebuffers first, then redraw those in a single-pass friendly way using a pixel shader, but I’m wondering if there’s a simpler way.

I have to give the usual answer: we don’t have a publicly announced next release.

I know OpenXR continues forward, but again nothing announced. Unclear we’d point you toward an alternative. Is there a reason you aren’t running against steamvr in its openxr mode?

I would hope we’ll have RGB+A+D support in any next release. No MV as far as I know.

Definitely no schedule on vulkan support in the android client library – but I can circle with Will next week and see if there is anything we can do so you could bypass/replace cxrBlitFrame (no promises, no timeline, but I’ll ask!).

Thanks for the responses.

I am mostly using CloudXR through the SteamVR plugin and SteamVR as OpenXR runtime, but using the non-controller data API requires some additional code, either OpenXR API layers or other SteamVR plugins on the PC side, and that’s extra implementation and debugging overhead for me. I need this for things like eye-tracking and full body tracking to be forwarded to the PC. (I am using both in my PC VR game and cannot rely on AirLink due to FBT/ET requiring a Meta developer account which is a non-starter for end users / gamers)

So for now I’m just implementing those things through the server API and will eventually migrate the non-controller data channel through such plugins / interfaces. I am even considering whether writing my own SteamVR plugin for CloudXR instead of using Nvidia’s, to help me get a “one click installer” to make life easier for end-users. Friction is key. One of the great things my OK streamer does that neither SteamLink, AirLink, or Virtual Provide is clicking on the icon on my Quest and IMMEDIATELY (within 1-2 seconds) I’m in VR. No superfluous clicking on stuff. Hey, I started this app, make it happen now. I don’t understand why so many big companies’ UI experts don’t understand this. The best number of clicks to do something is 0. The next best is 1. Anything more than 1 click to do something I need to do often (like reconnect my headset to my PC VR game session) is a waste of my time. CloudXR also leaving the session active through disconnections is a huge benefit for devs. It reduces so much friction. This is the one “killer feature” that CXR has over all the others: it just gets out of the way and lets you work.

I mean, instead of having to install and configure extra OpenXR API layers. I’d rather the SteamVR plugin is simply a one-stop shop for eye-tracking and FBT (from Quest’s IOBT).

By the way, I posted a new build that seems very stable and reasonably error-free. Switching from AI decoder to normal decoder seems to have changed the gamma (those horizontal grey lines on SteamVR Home are quite dark), I have to look at that.

One way that CloudXR is definitely, objectively inferior to Virtual Desktop and AirLink, is the fact that the video transport layer is using sRGB 888, and doesn’t support HEVC 10-bit nor AV1 10-bit (on supported PC GPUs).

Virtual Desktop does an amazing job with AV1 10-bit from my 4090 to Quest 3, end to end, getting around 35ms latency without any foveation in GODMODE resolution which is quite high. I’d really love to see more configurability and flexibility about video formats, 10-bit HEVC and AV1 would be at the top of my list. 10-bit not only compresses better than 8-bit (as I’m sure you already know), due to less quantization error and less dithering / noise (which is inherently hard to compress), but also would open up HDR10 transport and wide colour gamut. Passing HDR10 through an 888 stream is going to be banding city.

First, thank you for this open source project !

Next, as a newcomer to CloudXR (application to CloudXR program not even accepted yet, so cannot download the SDK), what are the advantages of your client over the ones provided with the SDK ?

Finally, you mentioned Virtual Desktop.
In which cases would it be better to use CloudXR over Virtual Desktop ?

Thank you.

CXR is better for cloud, especially over 5G/L4S, while VRD is better for local wifi in terms of image quality currently.

I don’t see that changing any time soon until Nvidia supports 10-bit HEVC and AV1, both on the client and server sides. AV1 codec from my RTX 4090 in 10-bit GODLIKE res has 5ms less latency (15ms → 10ms) as the same res, 10-bit over HEVC codec, on Quest 3. CloudXR only support HEVC in 8-bit sRGB, which is fine for most VR games, but HDR VR headsets are coming soon and this will become a serious image quality bottleneck sooner than later.

My app is strictly better than Nvidia’s OVR sample, which is just a sample in all fairness, as it runs on OpenXR instead of the now-ancient and deprecated OVR API.

In an of itself OpenXR doesn’t add much on Quest, but what it does provide is access to Full Body Tracking, Hand Tracking, Eye-Tracking, Face-Tracking, Passthrough, simultaneous hand and controller tracking.

None of those features are forwarded to the PC side, yet, as I’m still waiting for Nvidia to release their OpenXR PC runtime based on Monado using CloudXR 4.0’s direct integration feature. This should enable all those features to work on PC, eventually. And also hopefully dynamic foveated transport.

My app is also fully configurable via JSON, and should work (in theory) on non-Meta headsets (once a few issues are sorted out). I currently don’t own a Pico or HTC standalone to test, but when/if I do, I will make sure it’s running well. I did add the controller bindings for Pico 3/4 and HTC Focus 3 and XR Elite recently. But a user is reporting Pico 3 shows a blank screen currently. I think the OpenXR runtime is incorrectly configured or not loading properly.

My app is also based on Facebook’s IGL graphics library, which should make it (in theory) easy to render GLTF files for things like controllers or local UI or home environments, which would be helpful to pick, say, the IP address of your PC without having it in the config file. But I also want to try and make it auto-discover the PC running SteamVR.

My app also should support direct engine integration. I know that it works with Nvidia’s server sample, but you have to open the firewall in Windows for the app for it to connect. The benefit of direct engine integration is possibly reduced latency compared to running an OpenXR runtime. Even the simplest OpenXR runtime possible would add some synchronization overhead or possibly some buffering and add several ms of latency.

1 Like

Thank you very much for the detailed answer, sir.
Much appreciated.

1 Like

A fun feature that I added recently is waist-oriented locomotion at a platform wide level, using Quest’s body tracking feature running locally on device.

It doesn’t require any PC side compositor integration whatsoever, or even game code support, it “just works”.

Here’s a video I made of it working in SkyrimVR:

https://twitter.com/BattleAxeVR/status/1785553082224406859

The idea is that you walk in the direction your body is facing, rather than your head, so you don’t get VR sim sickness since there’s no artificial turning used (unless you deliberately wanted to, say, using the right thumbstick).

It works great even on Quest 1 (which I had to make a separate build for, cause you can’t have simultaneous Quest 1 and 3 supported in the same app manifest).

The latest builds are all here:

1 Like

Yes, I saw it while browsing your GitHub repository.
I was wondering if that was really what I thought, and you just confirmed it.

I was playing Asgard’s Wrath and was moving forward with the right stick, turning my head left and right, and was unhappy to see my path deviating from my goal.

This feature of yours could come very handy.
Somebody asked the Virtual Desktop developers on their Discord to implement it in their software, with a link to your Tweet :)

Great, I hope anyone/everyone copies it, it’s really trivial to implement and a great feature, great reason to exploit FBT even if the game doesn’t support FBT at all.

It’s not my idea at all, I copied it from DecaMove (which I have), but that required a lot of manual recalibration, a SteamVr plugin, and didn’t work in many games. It was annoying to use, TBH. I just took it to the next level and put two and two together once I realized Quest’s FBT waist pose could be used to do the same thing but with 100% reliability and no user calibration needed, ever. That’s pure win. Software solutions that work are better than hardware. And there are 25M Quest users who can use waist-loco now.

There are numerous other tweaks I’ll be adding like custom X axis analog stick curves to make it easier to walk dead ahead without accidentally sliding to the sides, but without making it impossible to side strafe at max speed (or use 2D UI menus effectively).

Try it and let me know what you think! Waist loco is enabled by default in the JSON config.

I have a built-in way to swap analog thumbsticks, but it may or may not work properly w/ waist loco stick value rewriting, it needs some testing / validation feedback. I’m left-handed but prefer normal left=move, right=turn settings, but many VR games (like SkyrimVR) when you select left-handed mode in-game for guns or weapons, swap the thumbsticks too, so I have to “undo” this to play how I want to. Otherwise I’m stuck playing dual wielding characters which is fun but something I don’t always want in my character build. (for sword and board especially, I absolutely need the sword in the left hand and shield in the right, for instance).

Actually, they won’t do it.

Here is their answer:

the reason we would not want to do it natively like BattleAxe is doing is because if you forget to turn it off it can actually screw up some games. it is completely remapping input (takes the joystick angle - headposition from body pose = new joystick angle) which is not something I personally believe a streamer should be doing (BAVR can do whatever he wants in his own streamer of course, he’s building it for his game’s needs first and then making that available to whoever wants it)

Personally, I’ll try as soon as my application is accepted: currently I still don’t have access to the SDK, so I cannot setup a CloudXR server.

I think being able to input the server IP address from within the Client would be a nice option, as it allows the final customer to change it without needing all the Dev knowledge.

Edit: Actually, being able to turn on/off Waist locomotion and other things like that would also be useful, as it would allow to use different apps needing different settings smoothly.

In practice you don’t even need a toggle to turn it on/off, even to use 2D UI menus using the left thumbstick. All you have to do is keep your head relatively pointing in the same direction as your body and the thumbstick values are mostly unchanged. I implemented an easy on/off toggle in PlutoSphere when I first implemented this, using the left stick’s down button (which is rarely used in games), or using an in-VR control panel, but I found I didn’t need it at all. Just look forward when using the menu and it’s fine.

Virtual Desktop is free to do whatever they want, but this is my problem with closed source software: if they don’t want to enable something, even if many people want it, it doesn’t happen. People have been asking, begging, Virtual Desktop, for years, to support hand, eye- and body tracking, and those have only recently come online, thanks to mbucchia’s OpenXR API layers. So I give 90% of the credit to mbucchia for PC VR users finally being able to use Quest features without a developer account (AirLink). Once SteamLink supports FBT I see no real reason to use Virtual Desktop ever again, personally.

It’s also a bit absurd to complain about input remapping which you can do any number of ways, including in SteamVR’s own UI itself. Changing the world scale of games, for instance, allows you to effectively become Ant Man and step over the entire world faster, or squeeze into tiny spots potentially, depending on how the game’s physics behave. I did try implementing full thumbstick movement override for all VR games, i.e. all the HMD + controller poses locally get translated by thumbstick locomotion, and sent to the game that way, as if you had simply walked in that direction physically, but this allows you to basically fly off slopes (or into them) and at least in SkyrimVR, is game-breaking. But overriding the local stage using the right thumbstick to implement smooth turning has no such problems. For instance, many games, even if they do offer smooth turning, their turn rate is too low, which ironically is what causes VR sim sickness in some people.

Faster smooth turning rate to stop in your new desired direction quicker is closer to snap turning, effectively.

Anyway, Guy has the same mentality as Meta: we know what’s best for players, better than they do, and won’t let you customize your game inputs.

I plan on making a SteamVR plugin for waist-loco, like DecaMove, but fully open source one day, which will make his objections moot anyway and work with Virtual Desktop. However, long-term, I expect that Steam Link will completely overtake Virtual Desktop in terms of usage. For one, it’s free, for another, it’s far less clicks to get into VR. Virtual Desktop takes too much time / clicks to get in-game, its main namesake purpose is to virtualize your monitor in a blurry floating texture, which will never be as sharp as a real monitor due to Nyquist sampling theorem (there are infinite frequencies in single-pixel wide text and borders which no amount of filtering or resolution will ever match compared to looking at a real physical monitor 1:1 without any resampling or reprojection at all).

Anyway, I do plan on exposing all these features in a control panel GUI using IMGUI on a quad layer so it’s sharp, but I’m not making any money off this software so people can just use the JSON file for now.

Swapping the left/right analog thumbsticks mid-game or mid-session is far more useful and necessary than toggling waist-loco on/off IMO, because some games I need that swap and others I don’t. But with CloudXR, you can quit the streamer app on the Quest, change a setting in the JSON config, then reconnect and the session stays alive the whole time. The only things you can’t change after SteamVR has a CloudXR connection initially activated is the res and refresh rate. But with direct engine integration I could implement those things as well, simply forcing a reconnection on the client side with new parameters.

Another thing I should note: PlutoSphere was the first OpenXR-based PC VR streaming app, and it shipped in September of 2022. By October, when I bought Quest Pro, it took me literally one hour to implement local dimming, and we shipped it the next week, about six months before either AirLink or Virtual Desktop supported LD for PC VR gaming.

Eye-tracking for gaze cursor control of the local UI also took me only a few hours and I had something similar to Vision Pro, except holding down one Touch controller button to move the cursor, then press another button to click things. Which is far better in my opinion than constantly tracking your eye-sight and forcing the user to actively think about where they’re looking all the time. Many AVP reviewers found this off-putting and I agree, it was obvious to me how irritating constant eye-tracking for UI cursor control is. It would be like crazy gluing your hand to a mouse and then not being able to lift it off the table.