I have some questions regarding Audio2Face in different topics:
1: I have a custom mesh and texture of my head and face, I want to add audio to it (to have audio driven 3D model in Omniverse) using Audio2Face and I don’t need to export it to any other graphical software right now. Does my custom mesh need to have blendshapes?
2: If yes, is there any rules for these blendshapes? (count, names)
Hi @ALearner
If you just want the talking face in Audio2Face, you don’t need to provide blendshapes, just use character transfer function in the audio2face. You can find a series of related tutorial here:
1 Like
@yseol Thank you for your reply.
I have tried this tutorial many times but it didn’t work. Attached please see the videos. I have tried one of the NVIDIA’s mesh and a custom mesh which didnt work.
00072.usd (466.9 KB)
maya.rar (31.9 MB)
This is weird. Your model actually works on my audio2face. Do you see any error message when you apply “begin mesh fitting” or “begin post wrap”?
or can you capture the node-graph from your omnigraph editor? (from menu, OmniGraph → Editor…)
1 Like
Hi @yseol
I didnt see any error in “begin mesh fitting” or “begin post wrap” steps. But before those steps, when I added custom mesh on “Target Mesh” It shows me an error in the button of the screen:
“Error: /World/_0071/a_0071_obj/defaultobject/defaultobject.faceVertexCounts UsdRelationship not supported in GetPrimArrayAttr”
For any custom meshes it shows me this error (For NVIDIA’s mesh it didnt raise this error but still didn’t work for that at the end)
I took a video to show you the omni graph (if it should be more clear please let me know)
There are some warning as well which I attached their pictures. (you can see the warning or error messages at the button of the pictures)
I have attached the custom mesh which I used in video
Thank you! Waiting to hear from you!
WomanHead.usd (1.3 MB)
Hi @ALearner, I found that you’re getting warnings and an error that I don’t get for the same process - it might be some hardware issue or installation issue.
We will investigate what can cause those warnings and error message.
In the meantime, maybe you can remove Audio2Face and re-install it and try again, or update your GPU driver to the latest? thanks!
Hi @ALearner ,
Thank you for providing the file and video. (Although looks like I can’t play the video)
Only the first 3s was playing…
I’ll take a look today and try to repro as well.
Hi @ALearner ,
I can’t seem to repro the error.
As I can’t play your video, I can only see your screenshot that A2F is not loaded yet.
When you start the A2F App, are you able to play with the default scene?
On a different note, this character seems to have the teeth and gum as part of the face mesh, which is not ideal.
Hi @yseol
I updated the drivers but still couldn’t run the Audio2Face pipeline. (I tried more than 20 time with different situations just one time the A2F pipeline was working on my custom mesh)
here you can see the version of the drivers,
I have ran some CUDA samples to make sure the drivers are installed, updated and communicate properly.
Hi @esusantolim
To answer you question: “When you start the A2F App, are you able to play with the default scene?”
yes the default scene is working well when I start A2F
I have provided some screenshots of my drivers in the my previous reply to “yseol”
The only time which I get the error is when I set the “target mesh” and the error message is " Error: /World/_0071/a_0071_obj/defaultobject/defaultobject.faceVertexCounts UsdRelationship not supported in GetPrimArrayAttr"
And to reply this part of your comment: “this character seems to have the teeth and gum as part of the face mesh, which is not ideal.” , I tried different characters but none of them works and this time I just need to add the A2F pipeline
to my mesh and it is not important if it doesn’t create the ideal result.
I attached another character which doesn’t have teeth and gum.
I have attached the video again hopefully you can watch it this time.
00071.usd (466.9 KB)
@ALearner this seems like a difficult bug to address, cause we can’t reproduce it here so far, sorry for that.
Did you also un-install and re-install the audio2face app? if that doesn’t work, maybe the entire Omniverse launcher?
@yseol
I did that. I uninstalled all the Omniverse applications and connectors and uninstall the Omniverse launcher as well. And reinstall them again. For the first run I could add the pipeline to the custom model and it works well. (It didn’t raise the " Error: /World/_0071/a_0071_obj/defaultobject/defaultobject.faceVertexCounts UsdRelationship not supported in GetPrimArrayAttr" error. ) but when I ran it again it showed me the mentioned error for another custom mesh and stopped working. (I tried the same process on a RTX 6000 machine and the result was the same)
It is like after using an inappropriate mesh, something is changed and the pipeline stop working even on the proper mesh.
Hi, @ALearner , unfortunately, our dev team is unable reproduce the error but we will let you know if we find any new info related to your issue. I am sorry for the frustration.
The good news is that there is a new Audio2Face version coming soon (this or next week), so perhaps the new version will work better for you! Please let us know. :)
Cheers,
Edmar
P.S. We are constantly adding new tutorials and aggregate them all in the “learn” tab on the launcher (also see: https://developer.nvidia.com/nvidia-omniverse-tutorials).
he / him | LinkedIn | Discord ID: Prof E#2041
Omniverse: “A New Era of Collaboration and Simulation” (video)
Omniverse Create’s latest features show & tell: Twitch livestream.
As seen in VentureBeat | Physics Showcase
Omniverse Resources:
Open Beta | Documentation | Tech Requirements | Videos | Roadmap | Twitter | Discord | Twitch
PLEASE NOTE: If you are reporting a bug/issue, please provide OS, GPU, GPU Driver, the version of the app, and full log file (if applicable). For crashes, please zip and provide a link to your logs → C:\Users\ [YOUR NAME] \ .nvidia-omniverse\logs
1 Like
I used the “Omniverse clean up” tools and cleaned all apps and data.
Now, when I want to install Audio2Face I get this error ---->
“Error occurred during installation of Audio2Face: EBUSY: resource busy or locked, open ‘C:\Users\ailab\AppData\Local\ov\pkg\audio2face-2021.3.2\kit\extscore\omni.kit.pip_archive\pip_prebundle\PIL_imaging.cp37-win_amd64.pyd’”
Do you know what is the reason?
Hi @ALearner, I feel sorry for those errors.
FYI, I just ran the clean up tool here and reinstalled the audio2face and it is working for me without error.
We will let you know when we get the similar error and find the solution, at the moment, things are hard to reproduce here. Sorry for the inconvenience.
Sorry For the late reply @ALearner,
Not sure why I didn’t get notification from your previous reply.
This video plays fine this time. I can see that the A2F node itself is working fine for you, but something on the deformers to your target character that is not working.
I have my suspicion, but let me confirm it and get back to you.
Sorry for the inconvenience.
I’m asking the Launcher team as well to see the error regarding the your uninstall error.
hi @ALearner ,
I still can’t repro your initial deformer error.
Here is the video showing what I did using your file directly.
At the end of the video, see if you can confirm on menu OmniGraph>Settings if your setting is the same as mine.
And one other request is, can you open OmniGraph>Editor and do a screen grab of what is on your setup when it’s not working, please?
Here is how mine looks:
And, attached is the setup file I created in the video, try open it and see if the target head is moving or static on your end. Thank you.
ALearner_00071_applied.usd (17.7 MB)
1 Like
Hi @esusantolim
I really appreciate your support to figure out the problem.
I did some changes in omnigraph setting to reduce the workload and tried to follow instruction again. But unfortunately still didn’t work.
The only time which I get the error is when I set the “target mesh” and the error message is " Error: /World/_0071/a_0071_obj/defaultobject/defaultobject.faceVertexCounts UsdRelationship not supported in GetPrimArrayAttr"
Here I recorded a video from the scene. At the end of video you can see the graph structure.
I opened the file you have sent and that USD works well (the lip has motion)
Hi @ALearner ,
Can you try save that scene and reopen it? see if that helps refresh the deformers internal data.
Also, can you share the scene of your result setup, please?
Thank you
Hi @esusantolim
Today I tried again and it worked amazingly.
I think I should have restart my system after doing those changes you mentioned(Omnigraph-> Setting → uncheck extra loads)
It doesn’t show me that error anymore.
I don’t believe it yet :) so I wouldn’t close this post for few days to make sure everything is working fine.
I really appreciate your amazing support.
Thank you all @esusantolim @emendizabal @yseol
1 Like