Updates on future plans for CloudXR

Gaming and Visualization Technologies XR – VR/AR/MR CloudXR (VR and AR Streaming)
1

Hi, I was very surpriced there was not a new release of CloudXR fro the September GDX this year.
With the amazing spectrum of impressive technology developments presented during Jensens keynote, it was a bummer to see basically nothing on CloudXR improvements.

I start to wonder if CloudXR been put on pause?

What I am hoping for but not seing:

  • Improvements on the streaming
  • Better ways to send custom data from the client
  • More client examples
    • please update the Android and iOS native examples with more comments in the code and at least for the iOS split the code up a little bit better instead of that massive codefile that does almost everything
      – Please create client projects with Unity and Unreal
  • More server side examples
    • Unity, Epic Games (Unreal Engine), Vulcan based
      – NVIDIA teams up with Unity, Unreal engine and Steam (SteamVR) to make sure end to end system stability and feature availability (F.eks. I don’t think Unity will work with DirectX 12 + CloudXR yet). Unreal Engine 5 would be great with CloudXR, but it doesn’t seem to work properly yet . Why not sync with Epic Games to make sure your two platforms work seemlessly together?
  • More Omniverse integration examples - this is your stuff, you should make it easy for the industry to pick up a few pieces of excample lego blocks and build on it for CloudXR projects, making the Omniverse ecosystem a lot more relevant for high end “real-world-metaverse” usecases.
  • Geospatial pose - not just local: It should be possible to stream poses to the server app that are not just local origin x,y,z. Today there are many “real-world-metaverse” positionings services (typically Visual Positioning Services) from the likes of Google, Niantic , Apple, Snap, Augmented City that allows AR apps to get position and orientation in the real world. If such a position is obtained it should be possible to stream such poses to the CloudXR server. I suggest using the GeoPose JSON encoding for this ( http://geopose.org/ ). It is being developed by the Open Geospatial Consortium and being considered for wide ecosystem support by participants of the Metaverse Standards Forum.
  • Try to make it possible to create a browser client that can connect to a CloudXR server. Browsers have supported WebAssembly for many years. Android has supported proper AR sessions in the chrome browser for some time. Eventually Apple should do the same. If lots of really great 5G edgecloud powered CloudXR experiences can only be had on standards compliant android devices it might inspire Apple to finally implement proper support.
    • nothing would be a more seamless “customer journey” than for a 5G connected smartphone or AR device user just open a link and instantly be in a CloudXR experience with no app-download. Experience provider would of course have to make sure enough servers are available so that there are always servers ready to stream. That could help drive more future demand for NVIDIA hardware in edge datacentres, wouldn’t it?
2

Hi Jan and thanks for the thoughts! A quick heads up that we’re launching a feature request portal that tags and tracks items like this so we can resource and allocate properly.

Regarding CloudXR, we’ve been feverously working on CloudXR SDK optimizations, architecture, and partnerships like the recent announcement with Lenovo. However, we’re not able to share specific details but do have plans for CloudXR that touch on many of the bullet points you’ve outlined. Once we’re ready to share we’ll communicate these details to you and the community.

1 Like
3

Looking forward to next version.

I really hope it will included fixes for the following, at the minimum:

-Ability to run stably at 90 / 120 Hz on Quest 2, without crashing after a few minutes due to instabilities in the advanced image decoder (this is 100% repro for us, it’s 100% caused by this flag). We are running at 90 Hz stably with the flag disabled but if it helps performance, it would be desirable to enable it, obviously.

-Oculus Touch controller poses don’t match those you see in-game when you use Link / AirLink.

-Oculus Touch face buttons’ (AB/XY) capacitive touch sensors isn’t supported by CloudXR currently

-cxrHmdTrackingFlags_HasIPD and cxrHmdTrackingFlags_HasProjection flags seem to have no effect. This means if you change your IPD after connecting to CloudXR initially, and change it afterward (say, passing the headset to a friend or just adjusting it, even slightly), it will result in Quest 2 having the wrong FOV. As you know, Quest 2’s FOV depends on the IPD, and once you start CloudXR the FOV cannot change. You have to restart SteamVR and reconnect for there to be any effect.

-Same for changing refresh rate and/or resolution and/or foveation settings. If any of those settings are changed after the initial connection to CloudXR, they cannot be changed afterwards without restarting SteamVR and reconnecting. Changing refresh rate normally requires a SteamVR restart on PC wired headsets, but changing the resolution shouldn’t and is basically the main way that users can hit certain framerates. Virtual Desktop allows you to change res while connected (even if the connection is re-established under the hood, it doesn’t require any manual intervention).

-Lastly, an OpenXR-based sample for PC and Android would be great. Or at least a way to forward eye-tracking data, and hand-tracking poses / finger poses. We are using OpenXR with CloudXR and CloudXR is the data bottleneck that’s preventing both hand- and eye-tracking from being useable on remote PCs as a result. If you can just open up extra data channels for these trackers (including a waist tracker, or FBT systems), that would be terrific. Thanks

1 Like
4

Hi all!
It would be great to add eye coordinates to CloudXR for blink detection and foveated rendering. There are already devices on which it can be used: pico 4 enterprise and qest 2 pro.
You can also add the transmission of coordinates of additional trackers, for example, to track legs.

5

I have filled in a couple of proposals, but there is no category for “CloudXR” so I am prefixing all proposals with “CloudXR:” and then choosing “other” as category. Maybe you could consider adding CloudXR as a category?

6

Thanks for the submissions–these are really appreciated! This forum channel is targeted for CloudXR so most feedback is grouped within that root category. The “Other” category is perfect for anything that doesn’t seem to fit within the available options.

I did add a category for the CloudXR Core SDK functions in case developers are looking for API specific changes.

There’s some great updates coming for CloudXR that we’re very excited to share when they’re ready. Until then, we’re listening and keep the ideas coming.