Cloud XR with Pass-Through

[xaver.freiherr-loeffelhol]

Hi,

I know that multiple parties are in contact with Nvidia about the topic but I thought it would be beneficial to have the discussion also here publicly :)

So in general for me the most requested feature by our users is using pass-through with the Quest 3. My naive understanding would be that it should not be that hard actually, because with CloudXR 4.0 I could just say I want 2 RGBA streams and then activate pass-trough in the client. Now I heard that streaming alpha is not yet possible, but isn’t that done when I stream to an ipad using AR kit? And my understanding was that also for AR kit the server still sends 2 streams but the client just uses one of them. So I wonder what would need to be done on the server side or if that is ready to go.

The other problem I see is that the client uses the deprecated OpenVR sdk and to get the pass-through we would have to integrate the OpenXR sdk, where I am not sure what problems would come up if both are used, or how complex it would be to completelly replace the OpenVR sdk with OpenXR. Other solutions I heard about include some kind of chroma keying on the client side, but even then the pass-through would need to be activted on the client side I think.

Also it would be nice to get clearer feeling for what is coming there on the CloudXR roadmap, so that we know if a sample with pass-through is basically around the corner, or we should invest some time in it.

[AndreusNvidius]

Here are the answers I can provide promptly:

Tablet AR vs Stereo Alpha The way the tablet AR sample works does not extend to stereo RGB+A. Because of the OpenVR limitation, server applications can only submit two RGB textures. The current protocol in the sample clients is that for mono RGB+A, RGB is sent through one eye, and A through the other.

OpenVR We know, and we know. Our next generation work is moving to OpenXR, but it simply is not ready for prime time.

Chroma-key on client The clients we provide are sample clients, not meant to be used in their provided form for publication. Chroma keying on the client requires shader level modification to the client, plus as you point out modifying the client to use a pass-through mode, but with the right skill sets, should be fairly straightforward. I have done so with the Unity plugin fairly trivially; the C++ clients will take more work.

So, yes, CloudXR 4 supports RGB+A stereo streaming, but there is not a server that will send stereo RGB+A, neither one published by NVIDIA nor anyone else so far as I am aware.