Custom character and blendshapes

I have some questions regarding Audio2Face in different topics:
1: I have a custom mesh and texture of my head and face, I want to add audio to it (to have audio driven 3D model in Omniverse) using Audio2Face and I don’t need to export it to any other graphical software right now. Does my custom mesh need to have blendshapes?
2: If yes, is there any rules for these blendshapes? (count, names)

Hi @ALearner
If you just want the talking face in Audio2Face, you don’t need to provide blendshapes, just use character transfer function in the audio2face. You can find a series of related tutorial here:

1 Like

@yseol Thank you for your reply.
I have tried this tutorial many times but it didn’t work. Attached please see the videos. I have tried one of the NVIDIA’s mesh and a custom mesh which didnt work.

00072.usd (466.9 KB)
maya.rar (31.9 MB)

This is weird. Your model actually works on my audio2face. Do you see any error message when you apply “begin mesh fitting” or “begin post wrap”?
or can you capture the node-graph from your omnigraph editor? (from menu, OmniGraph → Editor…)


1 Like

Hi @yseol
I didnt see any error in “begin mesh fitting” or “begin post wrap” steps. But before those steps, when I added custom mesh on “Target Mesh” It shows me an error in the button of the screen:
“Error: /World/_0071/a_0071_obj/defaultobject/defaultobject.faceVertexCounts UsdRelationship not supported in GetPrimArrayAttr”

For any custom meshes it shows me this error (For NVIDIA’s mesh it didnt raise this error but still didn’t work for that at the end)
I took a video to show you the omni graph (if it should be more clear please let me know)
There are some warning as well which I attached their pictures. (you can see the warning or error messages at the button of the pictures)
I have attached the custom mesh which I used in video
Thank you! Waiting to hear from you!

WomanHead.usd (1.3 MB)

Hi @ALearner, I found that you’re getting warnings and an error that I don’t get for the same process - it might be some hardware issue or installation issue.
We will investigate what can cause those warnings and error message.
In the meantime, maybe you can remove Audio2Face and re-install it and try again, or update your GPU driver to the latest? thanks!

Hi @ALearner ,
Thank you for providing the file and video. (Although looks like I can’t play the video)
Only the first 3s was playing…

I’ll take a look today and try to repro as well.

Hi @ALearner ,

I can’t seem to repro the error.
As I can’t play your video, I can only see your screenshot that A2F is not loaded yet.
When you start the A2F App, are you able to play with the default scene?

On a different note, this character seems to have the teeth and gum as part of the face mesh, which is not ideal.

Hi @yseol
I updated the drivers but still couldn’t run the Audio2Face pipeline. (I tried more than 20 time with different situations just one time the A2F pipeline was working on my custom mesh)
here you can see the version of the drivers,
I have ran some CUDA samples to make sure the drivers are installed, updated and communicate properly.

Hi @esusantolim
To answer you question: “When you start the A2F App, are you able to play with the default scene?”
yes the default scene is working well when I start A2F

I have provided some screenshots of my drivers in the my previous reply to “yseol”

The only time which I get the error is when I set the “target mesh” and the error message is " Error: /World/_0071/a_0071_obj/defaultobject/defaultobject.faceVertexCounts UsdRelationship not supported in GetPrimArrayAttr"

And to reply this part of your comment: “this character seems to have the teeth and gum as part of the face mesh, which is not ideal.” , I tried different characters but none of them works and this time I just need to add the A2F pipeline
to my mesh and it is not important if it doesn’t create the ideal result.
I attached another character which doesn’t have teeth and gum.

I have attached the video again hopefully you can watch it this time.

00071.usd (466.9 KB)

@ALearner this seems like a difficult bug to address, cause we can’t reproduce it here so far, sorry for that.
Did you also un-install and re-install the audio2face app? if that doesn’t work, maybe the entire Omniverse launcher?

I did that. I uninstalled all the Omniverse applications and connectors and uninstall the Omniverse launcher as well. And reinstall them again. For the first run I could add the pipeline to the custom model and it works well. (It didn’t raise the " Error: /World/_0071/a_0071_obj/defaultobject/defaultobject.faceVertexCounts UsdRelationship not supported in GetPrimArrayAttr" error. ) but when I ran it again it showed me the mentioned error for another custom mesh and stopped working. (I tried the same process on a RTX 6000 machine and the result was the same)
It is like after using an inappropriate mesh, something is changed and the pipeline stop working even on the proper mesh.