Hi everyone! I am having problems with the facial animations and lip-syncing results I’m getting after importing Audio2Face USD files to Unreal Engine 5. The animations look great while inside Audio2Face, but when I export them and then import them to UE5, my metahumans’ facial animations don’t look good at all. They lack expressions on their faces, their mouths barely move, the eyes barely blink, the mouths are very stiff. Does that have something to do with the values entered for Weight Regularization, Temporal Smoothing, Weight Sparsity and Symmetric Pose? Or maybe with blend shapes set up? Any help would be very appreciated. Thanks a million in advance!
From the description, you’re using “export .usd” from the blendshape conversion and “import that .usd” into UE5 using the Omniverse connector, correct?
Our default setup should give reasonably good facial animation through this process, and we need to know what’s your scene set-up and the resulting animation in UE5 to see what’s going wrong.
Do you also know we can stream the animation data and animate the metahuman in realtime? you can also test that feature for a sanity check.
1 Like
Thank you so much for the quick reply, yseol. Yes, that’s exactly the situation. I record my voice and save it as an audio file, then I load it into the Audio Player in Audio2Face. Once I’m satisfied with the face expressions, I convert it to a blendshape animation and export it as a USD file. Then, I use the Omniverse connector to import it into UE5, as you well said. I have good news for you, yseol: yesterday, I have been doing different tests, changing values to Weight Regularization and Temporal Smoothing, and I was able to get better results. The metahumans now show more face expressions and the lip-syncing has improved too. They are not perfect yet, but I’ll keep on doing more tests. So, I guess it’s only a matter of experimenting with those settings. Answering your question, yes I have heard now it’s possible to stream animation data and animate metahumans in real-time, but I’m not sure what that means. Is that for people who use metahumans during podcasts online? Please, clarify. Thanks a million again for taking the time to reply to my post. Very appreciated!
could the values mentioned in this part of the video when prepping for a Blender mesh play a role as well?
and I think what @yseol mentioned pertains to this training series - https://www.youtube.com/watch?v=1Tuov1gLCgQ and also an older video made by the man himself - https://www.youtube.com/watch?v=jrZ71xtaJ5E
2 Likes
@rominachirre great to hear that you’re getting better result. Yep, please test the different values in the setting. We tried to maintain the quality of the A2F as much as possible, but we might still have some quality loss due to the blendshape conversion process and mapping to MH controls.
Regarding the streaming feature, it can be used for all the live streaming use cases like interactive avatar application, micro-services, previewing the result realtime, etc.
1 Like
Hi, Simplychenable :) Thank you so much for sharing both videos with us! I have tested the values given in the tutorial “Omniverse Audio2Face and Blender | Part 2: Loading AI-Generated Lip Sync Clips”, which is 0.001, in the “input” section of Audio2Face, but, in my particular case, I didn’t see much improvement. The face expressions of my metahuman could barely be detected. Maybe that number is for smaller characters than my metahuman, but, I still need more testing to be sure if this is correct. These are the values that I believe give better results:
Hopefully more people will comment here and let us know if they have tested with other numbers and what results they obtained.
And, wow! I’m glad you shared the tutorial “Overview of the Omniverse Audio2Face to Metahuman Pipeline with Unreal Engine 4 Connector”. It’s something new for me and I find it very useful! Thanks a lot!
Thank you so much for the information about the streaming feature, yseol! I can’t wait to learn more about it. Yes, I will definitely keep testing different values, and hopefully I’ll find out which ones work best. Even if there is some quality loss, transferring animations from Audio2Face to UE, I still appreciate that this amazing tool exists! Thanks for the great tech support as well. Very appreciated!