Python API/SDK to inject custom parameters. in omniverse audio2face

Hello, I am currently working with omniverse audio2face and I was wondering if there’s an external API/SDK (python or node or whatever language) that can be used to input the audio file along with the post processing parameters.

Thank you

Hello @ali.ak007! Welcome to the community!

Audio2Face is built of several components that are meant to be modular depends on the need of each app. We are exploring different scenarios on how to get Audio2Face into different usecases in Omniverse, and our development priority currently is to focus on Audio2Face integration throughout Omniverse apps.

You are able to access many parts of Omniverse and Audio2Face through Python and OV python SDK. If you have particular questions, let us know.

You can use python to access almost all of the UI so loading audio and edit post processing parameters is supported.

Let us know what your use cases are and we can help more

Thanks for your reply,

I want to be able to stream an audio file from a python script to audio2face, not uploading it from audio2face directly.
Let’s say for example, I want to speak or record my voice, and input the resulting audio to audio2face and have the avatar repeat what I said or just say new stuff in real time. (I say hello avatar, and it will reply Hi for example), so more like a back and forth conversation with the avatar.

The final product will be like the MISTY bot, so I just want to know how to stream the processed audio to audio2face using external libraries and plugins.

If you can also elaborate more regarding the python SDK, it would be great.
Thank you!

If you like streaming audio, that is a feature we are adding now, specifically to make TTS streaming input easier.

If you just want to use python to load audio, that is doable. But you will need the UI loaded I believe. A2F has some dependencies right now looking for inputs from the UI at the moment. So sounds like from your use case, you may want to wait for the new streaming feature that is coming.


1 Like

Can you provide a link to the python sdk documentation?

Hello @377351842. Here is a link to our Kit Programming Manual with sections on Python: Kit Programming Manual
Also, here is a link to our Audio2Face documentation

Please let me know if this answered your question or if you need more specific guidance.

Hello, I have one more question.
I have added the following two lines to “audio2face.kit”:
“omni.kit.console” = {}
exts. “omni.kit.console”.autoRunREPL = true
Then execute “kit.exe audio2face.kit” to start the program
But before the program finishes initializing, it enters the python environment.
How do I write the configuration file to complete the initialization first and then enter the python environment?


I want to create an avatar animation by input.

Let’s say for example: I want to pass input data to audio2face that if a person is standing in any position and that would be created on audio2face
or like if a person is sitting on chair and that person is now in different pose, so what I want that I should give some input from backend to omniverse audio2face to create some avatar like that.

Basically, we can give expressions to an avatar with audio2face application but what I want that I can pass the expression values and how the person is standing from backend or with some API.

With this I want, to have conversation with avatar like, if I say “How are you” then the avatar replies me “I am fine”… something like that.

So more like a conversation, with avatar created by backend input.

Please if you can help me on this, it would be great.
Thank you

Hi @siyuen

Could you please share those python scripts to access all of the UI, to load audio and edit post processing parameters.

That will be really helpful.

Thank you

Hello @ankitgargit123! I am sharing some links from our scripting manual. Let me know if these help or if you need something more specific or helpful. I admit that I am not an expert with scripting, so just let me know how I can help!

Hi @WendyGram

Thanks for your response.

Can you please help me for below. I just want to know, if that is possible with the current versions of Machinima or any other app.

1)- I want to create a talking avatar with the help of omniverse. And my requirement is that I can give some inputs like “body movements + facial expressions + some audio” with the help of Python from outside. And in response the avatar will do the same and answer my asked question. Can you please tell me if any such thing is possible with any of the app in omniverse. If yes, please tell me how I can achieve that.

2)- Or if the above is not possible. Then can I live talk with avatar? I mean, if I will ask some question to avatar on live camera, is it possible that the avatar will answer my question.
As you know, on the very latest version of Machinima, we got one new feature that is Pose Tracker and with that we can give live body movements to avatar. So using same, can we talk with avatar and ask question to him.

Please help on this with some valuable inputs from your side.

Thank you