I was wondering if you can import a MetaHuman face mesh into Audio2Face to achieve better results. I see on the Audio2Face product page there are other character face meshes imported into Audio2Face from what looks like iClone or Character Creator so I was wondering if the same thing would work for MetaHumans? I’m asking because the results I’m getting from exporting the blend solve and then importing into Unreal Engine 5 is not really usable.
Hi @idancerecords, probably you can try this route: export metahuman into maya, then convert to .usd to import into audio2face.
Could you give us more detail which part of blendshape solve + import is not usable for you? We can see if there is room for improvment.
There is definitely room for improvement. The blendshape solve basically destroys all the great AI lipsync that Audio2Face generates. After I exported from Audio2Face and imported into Unreal the entire animation was way off… the mouth never even closes. It’s just a mess of unusable animation. I basically had to redo the entire animation by hand … creating blend shape poses for all the phoneme shapes and then re-animating the entire thing. So yeah, there is definitely room for improvement. You need to make it so that the animation that Audio2Face generates is exactly the same when it’s applied to a custom character… and in this case, to the Epic Games, Metahuman characters. If I had to tweak the animation a tiny bit here and there I would be okay with that, but the results I got were not usable at all and I ended up having to animate the entire thing by hand… which defeats the whole point of using this software.
I actually think the whole approach to this software is off. I don’t want to start off in Audio2face using some generic male avatar head… what if my character is female? It would be way better to be able to import your custom character… whatever that is… Character Creator, iClone, MetaHuman, DazStudio, Poser, custom mesh, etc. and then begin the animation process with the character that will ultimately be using the animation so you don’t waste time tweaking the animation on some random generic proxy character. You would then know exactly what you would be getting. The final resulting facial animation and lipsync should be exactly what you see inside Audio2Face.
Hi @idancerecords ,
We are aware of some of the bad mappings and will be addressing them in future updates. In the meantime, the blendshape solve does have some attributes you can tune to improve this.
To answer your other question about taking MH face to Audio2Face, yes you can definitely do that and you would not have this issue doing a character transfer to that face instead, and also you get better quality compare to blendshape. Depending on your need, this may work also.
Can you point me to some docs or instructions/tutorial on how to take MH face into Audio2Face to do a character transfer to that face?
+1 on this. I used Audio2Face 2021.3.3. export to MetaHumans in Unreal 4 with great results and eagerly awaited the 2022.1 release. But when I tried to export data from the new “Full Face” model to MetaHumans in Unreal 5, the lip sync completely failed.
Compare this screen capture in audio2face 2022.1 to the export of the same motion data to MetaHuman 5 via Bluprint.
After so many pushed back release dates due to quality assurance and the GREAT new emotion features, this is kind of frustrating. Can I kindly ask the awesome NVIDIA engineers on this project to look into this and post a fix?
Rather than waiting for another full update, maybe posting an just updated blueprint for export to MetaHumans/UE5 as a separate file would be faster and therefore much appreciated.
Thank you very much in advance for considering this.
@siyuen : can you please elaborate about the blend shape attribute tuning?
Bump… any news on this?
I see, this is not resolved yet on your side. sorry to hear that.
We actually tested this many times on our side - exporting from 2022.1.2 → UE5, and it worked as we expected, so we thought it’s a user mistake somewhere in the process.
How can we debug this?
- are you using the UE5 connector from Omniverse launcher?
- you can send your audio file and we can run it and verify the retarget on a metahuman in UE5. then send you the .usd file for you to import.
- if it doesn’t solve your issue, you can maybe share your metahuman and we can test.
I’m looking into this as well. I haven’t found any tutorial videos for getting a metahuman into Audio2Face. I see you gave instructions on how to import into Maya and then to Audio2Face. I’m not very familiar with Maya or other 3D editing software. I think Maya requires an Adobe subscription? Is there a way to do it with Blender?
Any chance for a tutorial video on this?
It looks like I was able to import the FBX directly from UE5 instead of needing maya to create the .usd
I’m unsure how to get the textures/materials to render properly on the mesh in Audio2Face. Also the process of doing the mapping of the points on the face to the target skin seems very prone to error and require a lot of tweaking. You can see here for the steps I followed.
I’m hoping instead I can figure out a way to do a realtime pipeline from Audio2Face → UE5 metahumans
1 Like
Bump. I tried to export the face skeletal mesh for a metahuman into omniverse using the connector for UE5.1 - the skin texture did not work despite trying different export settings. Furthermore I did not find a way to preview the animation on my custom mesh. Blindly exporting an animation that worked OK on the default a2f head into metahuman, the result was not really usable - mouth too wide open, and top teeth too exposed throughout. Even adding an additive layer did not produce anything usable. So, it is very important to allow custom metahuman heads into a2f, to preview what the result looks like.
Also when I was transfering the animation to blend shapes (ideally this would not be a necessary operation each time) one side of the face looked different, the animation was not the same, there was significant discrepancy.
Overall it seems to me that manual animation of faces is the only way to get a decent result, it’s definitely amazing that something like this can be automated but to me a2f is a proof of concept/novelty. The absolute weight of the package (3GB+nucleus+UE5 connector), the video memory it eats which makes UE5 crash in the background, the fact that you can’t easily use your own mesh for animating, the hundreds of unknown parameters, Nvidia forcing you to install a server on your PC… it all ends up being an extremely poor experience.
2 Likes