CXR 3.0: Bidirectional Audio - how to receive sound in app?

Gaming and Visualization Technologies XR – VR/AR/MR CloudXR (VR and AR Streaming)
1

Hi everyone

according to the latest release notes (CloudXR 3), it is possible to have bidirectional audio as in audio from the HMD to the server. Sending audio from the server to the HMD works flawlessly, the other way around not so much.

It is unclear to us, how to access the audio input from the HMD on our app running on the server; we were hoping CloudXR would expose a microphone to the system we could then grab as an input, but this is not the case neither on the AWS AMI provided by Nvidia nor our local servers. We’ve added the -ra/-sa flags, but so far have no clue how to process this data serverside (or where).

Is there anything we are missing?

Greetings

Dominique

2

Hi all -

We will look into this.

Thanks!

Veronica

3

Hi Dominique,

Have you looked at the Windows and Android Sample clients that show how this can be achieved using the CloudXR cxrSendAudio() API and the cxrClientCallbacks.RenderAudio() callback?

4

Hi Dominique,
Thank you for trying out the new back channel audio feature! The CloudXR server installer should be installing a virtual audio device driver on the server machine (see NVIDIA CloudXR Server — NVIDIA CloudXR SDK documentation) These devices can be read to and written from using the Windows Audio SDK of your choice. The examples Greg mentioned above are in our sample client (they use SDL) and can be used as a reference when implementing server side read/write audio as well.

A good way to test if things are working is to open up a teleconferencing app like Slack etc from within SteamVR’s desktop view and try to initiate a call. Apps that can read from a physical microphone on Windows should not need any changes to be able to read from our virtual microphone device. One thing to note is that the audio devices are only created when a client device connects (see NVIDIA CloudXR Server — NVIDIA CloudXR SDK documentation).

Please let us know if you have any further issues or feedback when trying this out.

Thanks!
-Arjun

5

Hi Arjun, hi Greg & Veronica

thank you all for taking the time, we appreciate your replies. Your feedback led us to check every component again and we’ve rebuilt our server images based on the Nvidia CXR 3 image, we’ve re-downloaded the CXR 3.0 Client SDK and remade our client application and now, using -sa/-ra flags, we see an Nvidia Virtual Audio Device on the server! We will do integration tests now, but it looks much better.

We are not sure what was causing our problems, but we’ve noticed, that our Client libraries were built on a CloudXR 3.0 SDK download dated 2021-07-14, whereas the current download dates 2021-08-04. Was there a hotfix in-between?

Greetings & Thank you again

Dominique

6

Update: It works perfectly, Thank you for the hints!