Integration of a chat bot with A2F

Hello,
I’ve been working on a virtual assistant project where I built a system in which the user enters text or audio and a lip movement is generated in run-time and fed into ue4 metahuman.
But as the new releases of A2F came it feels like investing in such technology is much better than building a new system.
so the pipeline I use is simple
Input >> TTS >> Visemes generation >> UE4 metahuman
Here is my question, Is there a way to integrate A2F into such a pipeline to replace this visemes generation step?
Also such a system has to be interactive so I can’t be using it through the launcher but from what I’ve seen while searching A2F has a python sdk however, idk if the functions I need are supported yet.
One last thing if it’s not supported yet can you tell me how far can I go with the current state of A2F.
and Thanks a lot.

1 Like

Hi @user15223 , welcome to Audio2Face!

In the current release, it is not possible to build such interactive system. Audio2Face at current state doesn’t get TTS input nor send live output.
Making the pipeline interactive as you suggested can be one of our potential goals, but at the moment we cannot confirm when it will be ready.
We will let users know when we have any progress in that regards, thank you!

Hi, I would like this feature as well. Not as a chatbot, but as an extension to a home automation system.
Pre-recorded responses are fine with a event driven or user command response triggers.
UE metahuman using A2F generated recordings as talking head core functionality.
It would be nice to implement real time character eye and head movement in response to a Jetson Nano camera for face detection and tracking of users.

Ok, I have another question. maybe I asked for too much "^^
Is there a way to load a the voice into A2F through python script?
it doesn’t have to be a stream of voice, just a normal wav file
this is a step, the other one would be to save the output of A2F through python as well?
I can handle the rest of the pipeline if these two steps are available on the python SDK right now, are they?

Zero seat time on UE4 for me. Omniverse, A2F, Metahumans and UE4 are awesome!
Now for using a Jetson Nano face detect AI as a game controller input into UE4 for real time user detection and tracking.

@r356c
can you explain what you are using A2F for?
because I don’t get it but it feels like it can help me to solve my problem as well

Hi, I used a few youtube vids as workflow tutorials.
Kudos to JSFILMZ creator for the final push to get everything working.

@r356c
ye, but this can’t help you create the logic needed for AI right?
as you can only send the recorded animations from A2F but you can’t generate them in runtime
or am i missing something?

These are next generation video game and cinematic authoring tools.
UE4 looks to me to be at least a 6 month learning curve. At least.
Then, creative talent comes into play, of which I have none.
Fortunately, the Omniverse is going to allow engineering types and creative types to at least talk to, if not understand each other. Exciting times to see what comes from engineering and artistic collaborations.

@r356c
Yup that’s totally true

Hello, Im working on a similar idea, could we connect and share ideas and progress?

That would be great

Hi folks, To update you all. We are currently testing a streaming TTS audio player. We intend to release it as part of 2021.3.2 before the end of this year.

2 Likes

@RonanDB
Best News Ever <3

1 Like

Cool, whats the best way to reach out to you?

@user28327 On here works. If I miss something directed at the A2F dev team the board admins will tag it for us to follow up.