Use audio2face for avatar chatbot

Hi, I would like to use audio2face to animate a chatbot I’m creating. I already have the NLP part of the chatbot, as well as the speech2text and text2speech parts. In fact, to interact with the bot locally, what I do is to launch an Audio2Face window, configure the streaming player and send audios to it via Python. However, I’d like to be able to integrate this into a webpage, rendering the animated face in real-time, as I do in the Audio2Face studio, but via web. I’ve seen that this must be possible due to this demonstration video: Intelligent End-to-End AI Chatbot with Audio-Driven Facial Animation - YouTube However, I haven’t been able to guess how to do something like that in the video… I’ve seen you can export the usd cache from audio2face, but that’s not what I need, as I’d need to animate the usd in real-time; that’s useful only for applications where the audios are fixed, not generated by an AI model…

1 Like

Hello @usuario3863! Welcome to the Community! Wow super excited to hear what you are working on! I send your questions over to the development team to help get you some answers!

Hi @usuario3863 , thanks for the ping.

You can use one of the streaming extensions Omniverse provide if you like to stream through a webpage. We did recently find some of those extensions are not exposed in A2F and we have a hot fix patch coming that fixes that.

Upon doing that, just put the a2f app to full screen and stream the output to the web, that should be it. In the Misty demo above, that was just Omniverse running full screen viewport, that’s it.

hope this helps

Would it be possible to create this kind of AI bot or like Misty but for an AR application? What’s the performance requirements to run this?

Hey, I am working on the same project. It would be great if we could connect!

@siyuen has this patch been updated?