MetaHuman lipsync not correct

So I went through several Audio2Face tutorials to get a MetaHuman talking / singing in UE5 and I am very disappointed in the results. The animation is adequate at best inside Audio2Face but after exporting the USD blend shapes to UE5 and applying to my MetaHuman the results are vastly different… the mouth never closes and the animation is very different from how it is in Audio2Face. Inside UE5 it needs tons of cleanup work to even look remotely believable. Im so dissatisfied and disappointed in these results! I was really hoping for so much more! And why doesn’t this work on actually singing voices ???.. and im not talking about rap!
In order to get it to even remotely work on singing vocals I had to actually re-record the voice track and speak what is normally sung just to get it to track and even then the results are not great. Am I missing something?

We are aware of some of the export to Metahuman requires specific tuning. The short version of it is, the blendshape solve and export mapping to the Metahuman rig can be better tuned out of the box. We are planning to improve this in short term so the blendshape solve works better for Metahuman by default.

In the meantime, you can tune the blendshape solve and try a different export to see, we have seen better results from that.

For Singing, it really depends. Since Audio2Face is sound based, it can cover some rapping and singing but is not optimal for any type of song or singing. Some will work better than others. It is more tuned for speech based animation at the moment.

Thanks for the feedback.

When you say “you can tune the blendshape solve” what are you referring to? How do you tune it?

So, Audio2Face will work with singing voices in the future? This will open up a ton of additional use cases if it can do accurate lip sync and facial animation for singing characters.

I wish the transfer to Metahuman characters was more automatic and an exact copy of the animation I see inside Audio2Face. Even better would be to preview the animation on a MetaHuman face inside Audio2Face. I see on your website that you show other character faces imported into Audio2Face.

Is it better / more accurate to export to Maya and then import to Unreal Engine? Or export to iClone and then import to Unreal. Which option provides the best results?

Any tutorials on how to tune the blendshape solve or how to fix it once it’s inside Unreal Engine? I tried using the additive animation layer once inside Unreal Engine… but there is so much to fix that it’s almost not even worth using Audio2Face… but maybe I’m doing something wrong?

Also, what about the facial expressions and the eyes? When is that update going to be released? I really need to get accurate lipsync and facial animation for my Metahuman character and I was counting on this solution to work… so much so that I went out and bought a high-end gaming PC with an RTX3090 specifically for this.

Hello @idancerecords! You can take a look at our Audio2Face documentation for helpful information, tutorials, and videos.

Here is a link to our full documentation where you can learn more about all the features of Omniverse.

Also, Check out the Learn tab on the Omniverse launcher!

Thank you, but I have already reviewed all the Audio2Face documentation and tutorials and none of them addresses the issues that I have mentioned above regarding tuning the blendshapes solve or exporting to MetaHuman. I really want to like this software but at this point I can’t really recommend it to anyone without these glaring issues being resolved which basically makes the software unusable and a huge waste of time. Please fix the software so that it does what you claim it can do and create some documentation and tutorials specifically for “tuning the blendshape solve” for MetaHumans or just make it work without having to “tune” anything. I don’t want it to be kind of correct, I want it to be exactly correct or at least 90% correct out of the box so that the animation is actually usable.